1 Introduction

2 Project Plan

I will now explain my current plan for the project. Notice that I say current here – this may change throughout the course of the project: I may narrow in on a topic of interest, or branch out to investigate anomalous research findings.

I will be building the end product – the dissertation and software – via a process of iterations, much like an iterative Software Lifecycle. The Literature Survey is ongoing – throughout the whole project from beginning to end – feeding into all parts of the dissertation, and indeed this Proposal, as shown in the Gantt chart (Figure
). The literature I choose is sometimes chosen to support points I wish to make, sometimes acting to guide my next area of research, reinforce findings, compare or contrast with other research, and probably many other things I have not yet thought of. Most importantly, I will be looking at who the paper/article etc. is cited by, preferring sources that are peer-reviewed.

As well as this literature research, I will also have an ongoing Product Literature Survey – looking at existing software out there that is related to my current area of interest.

Central to this idea of iteration is my desired method of performing user studies: I will first do what I have called a “Pilot” – a short and shallow trial User Study that focuses not on the research I’m concerned with, but instead the particular experimental design I would like to use in my actual User Study. By employing a Pilot I can hopefully get an idea of the nature of the experimental design – perhaps discovering any variables I had not previously considered that will require me to increase my sample size or simplify the experiment in order to mitigate their effect on the dependent variable I wish to test for. These are all problems discovered in Yates (2012) – including basic teething problems in getting the experiment to flow smoothly. In an even less detailed aspect, the pilot may allow me to look at what is out there. It may help to not look for anything in particular initially, and see what happens.

At this stage, with the help of discussion with my Project Supervisor, I have some ideas about how to gather data in User Studies and these pilots could prove to be a useful testbed for such tools. I have a hypothesis that the novice developer “thrashing” Lopez et al. (2012) can be observed by shorter pauses between editing and experimentation, and I could measure this by way of measuring the mouse position relative to the IDE, clicks, and key-presses, using tools built-in to Elm and a bit of extension to stream this over the Internet to my storage facilities Czaplicki (2013b).

As you will see in the Gantt chart (Figure ) I have included Testing & Implementation under the same heading as I will be doing Test Driven Development. My experience on Placement at PicoChip, my job as a Software Engineer at Altran and readings have helped me realise that this way of developing is time-saving and improves code quality by enforcing modularity in order to test it Martin (2008) and Hunt & Thomas (1999).

2.1 Required Resources

I will now talk about the resources I require for the completion of this dissertation, including the availability of these resources.

I will require users for my user study. These users must be proficient in at least one programming language (declarative programming languages are niche in and of themselves, never mind the discipline of programming, so some basic knowledge is required in order to see useful patterns in User Interface Design). Suitable candidates are First and Second Year Computer Science students from most Universities in the UK. Their availability is limited – Christmas holidays and coursework deadlines may mean that certain periods of the term are particularly busy for them. At Bath, suitable periods are therefore November, January to Mid February (inclusive), Mid-March to April (inclusive). It will be useful to procure free periods for other nearby Universities to hedge my bets, and to have a decent random assignment of users so I can make equivalent groups in my experiments.

The ACM Digital library, accessible via the Bath portal either from University or from home via Single-sign-on is a valuable resource for research papers, articles and references. The Cited-By feature will allow me to assert the popularity/ranking of each resource. Another valuable resource is the Psychology of Programming Interest Group, a “[group of] people from diverse communities to explore common interests in the psychological aspects of programming and in the computational aspects of psychology”, with peer reviewed papers on particularly relevant topics to my area of research.

I will require regular access to the Internet, Emacs with haskell-mode installed and Elm version 0.10 Czaplicki (2013a). I will also need git for software source control, and bitbucket.org for online, private backups of my work. I require LaTeX to type up my dissertation, and have chosen texlive on Ubuntu 12.04.3 as my development environment of choice. The full development environment is installed at the house I am staying in, in Bath, on my laptop. I am also able to replicate this environment to a satisfactory level at Bath University on any computer with access via Putty/SSH or similar to LCPU, as all the above software can be installed and run on my Bath University account.

I am using Chromium Version 28.0.1500.71 Ubuntu 12.04 (28.0.1500.71-0ubuntu1.12.04.1) to run the Elm IDE, which is an important dependency that may cause problems in getting Users in User Studies to run a functionally equivalent browser. Only recent editions of Chrome, Chromium, Firefox, Opera and Safari (not Internet Explorer) support Elm web programs.

3 Ethical considerations

In conducting User Studies, I will be interacting with people and collecting data from them, so I must be considerate and mindful of those I talk to and the information I handle.

An Ethical Checklist such as the one Bath University uses as it’s template Bath (2013) may assist my research such that I treat each participant with care and respect. I may learn from the discoveries made by others – in my reading, I came across a paper (also mentioned earlier) that highlighted concerns that participants under study had, and the paper detailed ways to mitigate these concerns so as to make the participant feel that are informed and safe Yates (2012).

4 Literature Survey

4.1 Introduction to the problem area

The problem area of user-interface programming, and more generally, the activity of programming in a context such as a software engineering environment, encompasses certain realms of interest. Through my survey of literature, my research has touched upon the above-mentioned terms, and I have discovered some thought-provoking problems that exist in the field of programming. The concept of ‘Programming’ embodies other concepts – art-forms, engineering processes, science, language, and mathematics, among others. To me, programming is a creative endeavour unlike any other – in which the programmer wields materials of no substance – the code – by manipulating symbols on a screen, which represent states in the machine being used. There are so many programming languages, and all languages (all that are Turing-complete) reduce to the same language – that of a Turing Machine. So, why do we have so many programming languages?.

Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy. (Perlis, 1982)

Different languages lend themselves to different ways of thinking about problems. They may place emphasis on one feature, for example list manipulation and hide others such as types. The language or programming environment may make explicit the effect of changes as they are encoded, as opposed to queuing up a block of changes and the programmer having to initiate an update manually.

I would like to draw your attention in particular to the terms Abstraction, Cognitive offloading, Feedback, Loss of information?/Augmented reality?, Thrashing, and “Programming blind”. These, at current, are my topics of interest, and my literature review has up to this point been inextricably and heavily influenced by this.

4.2 What does it mean to be ‘easy to use?’

In the process of surveying relevant (and sometimes irrelevant) literature to this dissertation, recurring conceptual patterns were observed – one particular instance of this is that several authors seem to lay victim to the trap of claiming their creation is “easy to use”, “better”, “simpler than x” without providing any supportive evidence of this.

Perhaps these are incidents of ‘experimenter bias’ – where the evaluator is naturally predisposed to a positive appraisal of their own findings. One way to avoid this is to have one set of people perform the data capture and another set perform the data analysis. Nevertheless, these patterns emerge, and present numerous opportunities for experimentation and subsequent evidence supporting or contradicting these claims. Experiments may see if the same conclusions are reached as the above-mentioned authors, accounting for the ‘evaluator effect’ (Hertzum & Jacobsen, 2001).

Whether this particular route is taken for experimentation hinges on pilot studies that will be conducted concurrently to the Literature Survey, each inextricably shaping the other’s direction of investigation and inquiry.

The catalyst to this whole dissertation was a talk about the concept of a highly reactive development environment – where changes in the code result in instantaneous updates to the runtime, ‘on-the-fly’. This was presented in Bret Victor’s “Inventing on Principle” (Victor, 2012). In his presentation Bret makes several assertions about the ‘traditional’ style of coding, one statement of which is that “most of the developer’s time is spent looking at the code, blindly without an immediate connection to the thing they’re making”. He argues that “so much of creation is discovery, and you can’t discover anything if you can’t see what you’re doing” – alluding to his earlier statement that the compile-run-debug cycle is much like this.

Evan Czaplicki, in his thesis of which Elm is the product (Czaplicki, 2012), makes similar claims – “[Elm] makes it quick and easy to create and combine text, images, and video into rich multimedia displays.” While the evaluation of Elm’s usability is not the focus of the thesis, rather, it is to establish a context for Functional Reactive Programming and describe the implementation details, he makes other usability claims without evidence – “[non-declarative frameworks for graphical user interfaces] mire programmers in the many small, nonessential details of handling user input and modifying the display.”, “FRP makes GUI programming much more manageable”, and in a section entitled The Benefits of Functional GUIs, “In Elm, divisions between data code, display code, and user interaction code arise fairly naturally, helping programmers write robust GUI code”. If these claims are true, there is all the more evidence that Elm should be a language of choice for GUI programmers, but experiments must be done to determine this.

And perhaps this rapid development cycle is not always suitable – in their 2012 paper, Lopez et al. show that novices tend to “thrash” about, trying out many ideas that may or may not be a solution, and executing “poorly directed, ineffective problem solving …failing to realise they are doing it in good time, and fail to break out of it”, whereas experts think much more about the problem at hand before proceeding with a solution (Lopez et al., 2012).

4.3 Running User Studies

Perhaps a further direction of investigation may be running an experiment to spot whether or not Elm’s auto-updating IDE lends to a lack of critical thinking – some operationalization may be pauses reported as ‘thinking’ made during development – where a pause is disambiguated as ‘thinking’ by the experimenter asking the participant why they did not perform any interaction with the computer for more than 10 seconds, and the participant reports that they were planning/designing/other similar activity. Along this line of thinking, a paper studying the relationship between speech pauses and cognitive load (Khawaja et al., 2008) found through studying 48 mixed gender participants that there is statistically significant indicators of cognitive load through analysing pauses in speech. Perhaps this concept of pauses can be applied to the activity of programming. However, the planned method of disambiguating pauses via self-reporting (previously mentioned) would not be suitable according to these authors – “such measures can be either physically or psychologically intrusive and disrupt the normal flow of the interaction”, although a paper cited by (Khawaja et al., 2008) itself claims that “although self-ratings may appear questionable, it has been demonstrated that people are quite capable of giving a numerical indication of their perceived mental burden (Gopher & Braune, 1984)”. Indeed a pilot study by Fraser and Kölling (McKay & Kölling, 2012) structures the self-reporting by getting the users to evaluate an IDE as they use it using a set of subject-specific heuristics that they have designed. They showed that this customised set of heuristics helped guide the user more effectively than Nielsen’s heuristics in evaluating usability, so one could develop a custom set of heuristics for evaluating the usability of Elm.

From the Elm thesis (Czaplicki, 2012), the language syntax and rapid feedback seem simple enough that it is conceivable (or at the very least, possible and of experimental interest) to allow the user to customise the UI layout to their liking. Letting the user shape the UI in concert with a UI programmer is covered the study of the interface development environment “Mobi-D” in millitary and medical applications (Puerta, 1997), with success in those fields. It may be worth speculating how Elm would fit into the development cycle that Puerta’s paper outlines, as this may lend inspiration to potential user interface enhancements to the Elm IDE for A/B testing. It must be noted that there does not seem to be a re-emergence of Mobi-D since the paper was written, however.

My goal is to answer these questions. By way of conducting user studies, leveraging Elm with extensions to do A/B testing to illustrate it’s effectiveness (or ineffectiveness) at enhancing User Interface Design.

Central to this idea of iteration is my desired method of performing user studies: I will first do what I have called a “Pilot” – a short and shallow trial User Study that focuses not on the research I’m concerned with, but instead the particular experimental design I would like to use in my actual User Study. By employing a Pilot I can hopefully get an idea of the nature of the experimental design – perhaps discovering any variables I had not previously considered that will require me to increase my sample size or simplify the experiment in order to mitigate their effect on the dependent variable I wish to test for. These are all problems discovered in (Yates, 2012) – including basic teething problems in getting the experiment to flow smoothly. In an even less detailed aspect, the pilot may allow me to look at what is out there. It may help to not look for anything in particular initially, and see what happens.

At this stage, with the help of discussion with my Project Supervisor, I have some ideas about how to gather data in User Studies and these pilots could prove to be a useful testbed for such tools. I have a hypothesis that the novice developer “thrashing” (Lopez et al., 2012) can be observed by shorter pauses between editing and experimentation, and I could measure this by way of measuring the mouse position relative to the IDE, clicks, and key-presses, using tools built-in to Elm and a bit of extension to stream this over the Internet to my storage facilities (Czaplicki, 2013b).

5 Experimental methodology

5.1 AB Testing of the languages with the same IDE?

The primary direction I mentioned (as echoed in my Proposal) was doing AB testing of Elm vs. another language (e.g. JavaScript) (i.e. the language is the dependent variable) using the same Concurrent FRP IDE (the independent variable).

5.2 Test just the paradigm?

He also suggested a potential experiment to test just the paradigm, eliminating the IDE from the experiment above. Perhaps for a Pilot study.

5.2.1 Experiment process

1. Study question (e.g. Is it easy?)
2. Measurement concept (e.g. “Easy”)
3. Operationalisation – taking a measurement concept and mapping it to something concrete (e.g. if completing a pre-defined task the user must complete takes  < 5 steps, it is ‘easy’ – we can then compare instances of these studies given our definition of easy). This is much like mapping a design to an implementation, and there is a risk of losing information, or ending up with a mismatched concrete instance that does not represent the concept we wish to convey.
4. Do another operationalisation of our measurement concept – this allows us to get a different perspective of the same concept. (e.g. if total length of pauses during a 1 hour experiment is  < 10 minutes, it is ‘easy’). We do this to get ‘coverage’ of the measurement concept. It is a form of cross validation. If we see an overlap in the correlational results after analysis, we can make a stronger assertion that e.g. “language A is easier than language B.”. The idea I am describing here is methodological decision-making.
5. Predict what will be the likely results of our experiments on the operationalised measurements. This is “feed forward validation”.
6. Do the experiement.
7. Analyse the data. See if the data has patterns that correlate with the assertion I wish to make. I will be representing the raw data in some outcome measure – that is turning the raw data into a set of (or a single) value for comparison.
8. Does the data answer the study question I set out to ask? This is now “feed backwards validation”.
9. Write-up including the ‘nitty-gritty’ of the user study, and a statement like “Given our definition of easy, our multiple operationalisations of the concept of easy show that this is in fact objectively true/false”.

5.2.2 Pilots

We also spoke about ideas for pilot studies – asking “What might be surprising insights into declarative programming languages for User Interface Design – the case of Elm?”.

Speak-aloud protocols where you prompt/facilitate the user to say what is on their mind when that e.g. pause for more than 10 seconds – a measurement I set out to look for during an experiment.

I might ask

• I notice you have paused for at least 10 seconds – why did you?

• I thought the code would do X, but it did Y.

• Why did you think it would do X?

• …

I must ask the participant questions designed in a way that they are not leading.

Leon suggested I gather a rich data set, as it’s difficult to take notes AND prompt the user during an experiment. SO difficult. Perhaps record video.

5.2.3 Actions for next meeting

Devise a Pilot study, answering these 3 questions:

1. What might I ask people to do?
2. How will I gather data?
3. How will I analyse the data?

Also see paper Leon will send me on “Thematic analysis & Psychology”

6 Pilot Study 1

Using per-participant questionnaire (See ), I captured video & audio data of participants while the completed the task of extending a mario game to make mario fly

6.1 Hypotheses

• H1. …
• H2. …

6.1.1 Method

Using Thematic analysis (Braun & Clarke, 2006) to code the data…

6.2 Results

6.2.1 Observation 1

• Prompting “What are you thinking about?” etc. seemed to place additional cognitive load on the user as they spent longer resuming than when not prompted. This caused noise in assessing the actual cognitive load incurred during the completion of the task. Were the signs of struggling/undergoing difficulty due to simply not understanding the language, or were they due to the difficulty of the task?

• In particular, the majority of instances where the users paused turned out to be confusion as to the semantics & syntax of the language.

6.2.2 Model Adjustment 1

• Add tooltips that appear as the user places the keyboard cursor to the right of a token in the language.

6.2.3 Observation 2

• Sifting through 1-hour+ of video data capture for incidences of cognitive load is HARD!. Is there some programmatic way of narrowing the video data to points of interest?

6.2.4 Model Adjustment 2

• Track the user mouse and keyboard movements in a 3-tuple: (Time t, (Mouse.x, Mouse.y), Keypress k)

• It doesn’t have to be implemented this way. I could extend Model Adjustment 1 to define blocks of code as tokens in themselves, and capture how long the cursor is static on that particular token.

• Leon suggested a further refinement of this idea in order to further narrow the data (in fact, just capturing mouse & keyboard movements will result in an explosion of the volume of data – countrary to what I intend to achieve). His refinement was to define regions of interest in the code pane, and only when the mouse/key cursor is in the region, do I capture data.

• Use the if cursor in region then log (Time t, (Mouse.x, Mouse.y), Keypress k) functionality as a lens to focus on significant portions of video capture.

6.3 Further discussion

We then discussed some questions that might lead my direction of study in the next steps of my research:

• Is the mouse/cursor position a proxy for someone’s attention as they carry out the task?

• Often when I’m coding I’ll leave the cursor where it is but think about other regions of code. I don’t necessarily move the keyboard/mouse cursor to the section of code I’m thinking about. Instead, I use it as a ‘bookmark’ to track what I’m currently implementing, and may scroll around to other parts.

6.3.1 We also discussed…

The result of the dissertation will be a list of observed cognitive easing/loading that each language produces for users, much like an advantage/disadvantage comparison:

6.4 Actions

1. Design a task in JavaScript to go inside this adjusted model (incorporating Model Adjustment 1 and 2).

   This will require a degree of “implementation juggling” in order to find a balance of code-length/difficulty over the same task in Elm in such a way that is not creating noise in the thing being studied: Cognitive load.

   Keep the reactivity constant, compare the differences in ease between JS and Elm.

2. If time available, run another Pilot study on this task + adjusted model

6.4.1 Modifications to be made to the experimental methodology

Needs to be more objective! Why? What will I modify?

7 New Study! First, implement an IDE that logs input

7.1 Requirements

I will now identify what the requirements are for the project.

7.1.1 Functional Requirements

1. Write software to assist the capture of objective data to inform me of the user’s activities as they use the Elm IDE.

   1. The program must be able to work offline and later transfer collected data to me once a connection is resumed, collecting mouse and keyboard activity
      Priority: High

2. Perform Pilot and User Studies

   1. I must perform Pilot and User Studies in an iterative fashion, each one learning and building upon discoveries made in prior ones, starting vague and getting more and more focused on a particular facet of User Interface Design and/or Declarative programming as an activity.
      Priority: High

   2. I must use these studies to inform experimental and software design to disambiguate and filter data collected in the experiment, and to exercise hypotheses.
      Priority: High

7.1.2 Non-Functional Requirements

1. Source code

   1. The software must be written clearly and simply.
      Priority: High

   2. The software must have suitable, concise comments which explain the programs intent, but only where the code alone is not enough.
      Priority: High

2. Activity recording

   1. The program activity recording feature must not slow down the user’s use of the IDE more than 1ms difference than without it.
      Priority: High

   2. There should be software to visualise the usage data
      Priority: Medium

7.2 Design

This is the chapter in which you review your design decisions at various levels and critique the design process.

More detail on what I will modify. How will I modify?

7.2.1 Experimental Design

Discussed progress made and what hypotheses to form that may usefully model cognitive load.

7.2.1.1 Progress since last meeting

I have implemented full-screen mouse tracking that stores to a database a tuple:

(t, (x, y))

for every mouse move, producing a list in JSON (so it’s more like {{uniq-userid: {125125, (67, 321)}}, {uniq-userid: {125126, (67, 322)}} ...})

I am ready to demo this (See Action 1.)

The only issue worth tweaking is that user activity data is captured separately from the error output, so I will need to collate the data afterwards or find some way to feed it into the same data store.

7.2.1.2 Meeting Discussion

2 Hypotheses

1. Why the regions (see green boxes in figure above) I define in the code (to mouse-track e.g.) are meaningful

2. Frequency of semantically or syntactically incorrect errors made will differ as a function of the language under study

These need narrowing as they are too broad to test. Explode them into multiple, tighter hypotheses.

They are valid because they are well-founded – i.e. I have good reason to believe that # of errors made is an indication of cognitive load. I have good reason to believe that the selected regions will have more mouse activity (or whatever activity I suspect indicates higher cognitive load) as they are harder regions of code OR they pertain to achieving the set task.

7.2.1.3 Actions

1. Refine Mouse logging

   1. DONE Make it so that I can run arbitrary Elm code in the editor via a fileOpen operation
   2. DONE Make an Elm file that logs mouse movements ready to be loaded into the editor (See )
   3. DONE Load it into the editor and test it uploads to Firebase
   4. DONE Modify Generate.hs (See )

   1 2 3

   case (Elm.compile elmSrc) of Left jsSrc -> ... Right _ -> error "blah"

   So that when we get an error, we timestamp and append it to a log file so this can later be collated with the Firebase to determine when errors were made

   I’ll need to insert a layer between compile :: Snap() and serveHtml :: MonadSnap m => H.Html -> m () that performs the logging. It will have type signature TypedHtml -> H.Html

   See the functions compile and serveHtml in Server.hs (See ).

   5. Make it so I can define regions in the mouse tracking – i.e. ONLY within a defined region is the mouse movement tracked e.g. if mouse(x,y) in some2by2Square then Just mouse(x,y) else Nothing

   See https://github.com/spanners/laska/blob/master/Signals.elm

2. DONE Demo to supervisor
   1. Install on VPS (See )
   2. Run these:

   1 2 3

   git clone https://github.com/spanners/elm-lang.org cd elm-lang.org cabal install --bindir=.

3. DONE Design a task in JS and Elm

4. Define regions to select for logging activity. Why? Because:
   • Complex logic in code, OR
   • Relevant to task
   • Captures Thrash (keep on going over the same thing, e.g.). Errors made also captures thrash!

5. DONE Determine what to do with mouse (for example) data.

What makes code difficult to understand and work with?

• Bit twiddling?
• Declaring and defining simultaneously?
• Compound if/then/else statements?

[Programming is] manipulating symbols blindly ~ Bret Victor

Do a 2×2 study, defining regions in the code monitoring mouse clicks. Regions can either be simple/hard in complexity (exhibiting/not-exhibiting one of the above ‘difficult’ properties). Or code can be task-oriented or not, that is the code does/does not need to be changed to achieve the completed task set for the user:

7.2.1.4 Study method

Look at total and/or mean time in each of these areas for comparison.

My study will be between-subjects instead of within-subjects.

That is, I will study different users for different languages. If a user has completed the task in Elm, I can not have them complete the task in JavaScript, and vice-versa.

I will necessarily make a compromise here:

Between-subjects:

• I lose the ability to keep programmer competence as constant, thus it is a confounding variable

• I gain the ability to ignore learned-experience in completing the task – the participant is different every time so will not have done this task before, thus this is not a confounding variable.

Within-subjects is the converse of the above methodological properties

7.2.1.5 Actions

1. DONE Reorder divs so embedded div is on top of editor div.

   This turned out (I am fairly certain) to be due to codemirror.js binding mouse clicks. It was solved by using Elm’s Mouse.isDown. Using Mouse.isDown has the added benefit of tracking mouse selects and drags, because it logs (x,y) when the mouse is down and (x,y) again when it is up.

2. DONE Create a task that features Hard/Simple x Task/Not-task (See )

3. Implement Region filtering functionality so mouse activity is only logged when the clicks occur within defined region(s)

   I have instead defined bounding boxes that pertain to the regions I want to track as a mouse-data filter – that is, I capture all click data for the whole frame, and then filter it by comparing x,y co-ordinates with my bounding boxes. If it’s in the box, keep it, otherwise discard.

4. DONE Integrate JS task into IDE

5. DONE Perform pilot study

6. WIP Visualise mouse data

7.3 Implementation

Describe how I extended the Elm IDE

**This is the chapter in which you review the implementation and testing decisions and issues, and critique these processes. Code can be output inline using some code. For example, this code is inline: public static int example = 0; (I have used the character | as a delimiter, but any non-reserved character not in the code text can be used.) Code snippets can be output using the environment with the code given in the environment. For example, consider listing 5.1, below. Listing 5.1: Example code

Code listings are produced using the package “Listings”. This has many useful options, so have a look at the package documentation for further ideas.**

7.4 Pilot Study 2

Using the Elm IDE

7.4.1 Observations

The task I chose for Pilot Study 1 was too difficult to capture the cognitive load incurred by the language itself for a given task, due to the difficulty of the task itself creating noise. I could improve this by simplifying the task, in a way that is ‘language agnostic’, i.e. that is not idiomatic of Elm or JavaScript (the two languages that I am comparing). Something like the following will never be that easy in JavaScript:

1

main = lift asText Mouse.position

Saw some things in Pilot Study 1, also in the use of the Elm IDE I extended, I saw some things before Pilot Study 2.

7.4.1.1 Hypotheses

1H.

7.4.2 Experiment

7.4.2.1 Method

A 2×2×2 study, that is 2 Languages (Elm and JavaScript), 2 Region difficulties (Hard and Simple) and 2 Region relevances (Relevant and Not relevant) will be done to determine if the number of mouse clicks per region differ across variables.

7.4.3 Results

See Figure for the visualisation of participant 15 completing the Elm version of the task.

Operationalisation of thrash (the concept), i.e. cementing the concept by a metric that models cognitive load (does it? we don’t know – further work after the analysis of this may determine if it is a plausible indicator of cognitive load)

Leon suggested an improvement over this experimental method is to take people who are new, and train them up either in JS or Elm, and then run the same task. That way, their level of ability is comparable. (New as in never having used JS or Elm)

My current method creates quite a bit of noise in the data, because I rely on self-reported level of expertise in JS/Functional languages. I don’t know how to modify the data to account for this. I could group the analyses into categories? I.e those who reported being experts at JS, those who reported never having used it, those who reported being experts in at least one FP language, and those who reported being new.

Talk about “phases” in a programmer’s activities during task-completion:

(Not necessarily distinct and in sequence — more often interleaved)

1. Familiarisation – Where is the bit I need to change?
2. Narrowing in on the task once discovered – Oh I need to change X, but how?
3. Solved task
4. Playing (?)

7.4.3.1 Mention the ways in which the study is flawed:

1. Self-reported expertise
2. Self-reported task completion
3. No way to be sure which error log pertains to which compile
4. Unique participant ID per Surveymonkey
5. Surveymonkey has taken my data hostage
6. window dimensions?!
7. Syntax reference 404
8. I did not capture window resizing

Not capturing window resizing is problematic – participant 15 (See Figure ) very likely had a much shorter window height than I have used here. I suspect this is the case because of the cluster of mouse clicks in the same range of the x axis as the Compile button, but much futher up in the y axis, but I have no way to be sure as I did not log window dimensions.

7.4.3.2 Results

1. Describe data collected
2. How it was analysed (I aggregated regions and looked at number of clicks per region (Hard/Task, Hard/Not-Task, Simple/Task, Simple/Not-Task)*(Elm, JavaScript))
3. Presentation of data (summary means std dev.)
   1. χ² frequency analyses
   2. 2 × 2 × 2 making 8 cells. My expected is an even distribution of clicks in each category, i.e. if I have 80 clicks in total across all groups, I expect to find 10 in each cell if there is no correlation.

Instead of χ², consider just using multiple regression with dummy variables (binary predictors) (See Table )

• The d_n in the top row are the dummy variables, the values are the codes you assign them
• There are n − 1 dummy variabes; one group is coded as all zeros – that’s your reference group
• Why n − 1? If there are 8 dummy codes, there are the same number of variables as conditions. the model’s fully saturated and there are no degrees of freedom
  • Similar to x₁ = 2, x₂ = 4, x₃ = 1, and y = 2. then trying to solve for y

This is the chapter in which you review the outcomes, and critique the outcomes process.

You may include user evaluation here too.

8 Conclusions

This is the chapter in which you review the major achievements in the light of your original objectives, critique the process, critique your own learning and identify possible future work.

Appendices

Appendix A

Design Diagrams

Appendix B

Mouse click visualisation

Paricipant 15, Elm task (Overlay)

Raw results output

firebase-mouseclick-data.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

{
  "js" : {
    "12" : {
      "-JKVwXgnfg4POT9MArjy" : {
        "t" : 1397490853983,
        "y" : 0,
        "x" : 0
      },
      "-JKVwXl8I_bLZNqxP36c" : {
        "t" : 1397490854551,
        "y" : 444,
        "x" : 417
      },
      "-JKVwXj0LD-uRKDsT2Om" : {
        "t" : 1397490854503,
        "y" : 444,
        "x" : 417
      }
    }
  },
  "elm" : {
    "33" : {
      "-JKRgqJIRPH2EG-edZb5" : {
        "t" : 1397419631953,
        "y" : 249.59999084472656,
        "x" : 48.79999923706055
      },
      "-JKRhRQi31pr9AP1p1Rr" : {
        "t" : 1397419787709,
        "y" : 294.3999938964844,
        "x" : 152.8000030517578
      },
      "-JKRffOszNGfgwNdnO_X" : {
        "t" : 1397419324585,
        "y" : 608,
        "x" : 346.3999938964844
      }
    }
  }
}

error_log.json

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

{
   "2014-04-11 21:14:32.141743994+01:00":{
      "Parse error at (line 37, column 44):",
      "unexpected 'a'",
      "expecting \"{-\", \" \" or end of input"
   },
   "2014-04-11 21:35:41.694436974+01:00":{
      "Type error on line 27, column 16 to 77:",
      "        (min (max x (-hHeight)) hHeight,",
      "         (min (max y (-hWidth)),hWidth))",
      "",
      "   Expected Type: Float",
      "     Actual Type: (Float -> Float, Float)"
   },
   "2014-04-12 00:19:14.945129550+01:00":{
      "Parse error at (line 1, column 1):",
      "unexpected \"<\"",
      "expecting reserved word 'module', reserved word 'import' 
	      or at least one datatype or variable definition"
   },
   "2014-04-12 00:19:21.553633974+01:00":{
      "Parse error at (line 1, column 1):",
      "unexpected \"/\"",
      "expecting reserved word 'module', reserved word 'import' 
	      or at least one datatype or variable definition"
   },
   "2014-04-12 00:19:27.053901481+01:00":{
      "Parse error at (line 1, column 1):",
      "unexpected \"/\"",
      "expecting reserved word 'module', reserved word 'import' 
	      or at least one datatype or variable definition"
   },
}

Appendix C

Code

• All code available here: https://github.com/spanners/elm-lang.org.

  • This is a modifed version of Evan Czaplicki's elm-lang.org code, available here: https://github.com/elm-lang/elm-lang.org
• Elm task here: http://mouth.crabdance.com:8000/edit/task/MovingBox.elm

• Javascript task here: http://mouth.crabdance.com:8000/_edit/task/MovingBox.js

LICENSE

Copyright (c) 2012-2013 Evan Czaplicki

All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:

    * Redistributions of source code must retain the above copyright
      notice, this list of conditions and the following disclaimer.

    * Redistributions in binary form must reproduce the above
      copyright notice, this list of conditions and the following
      disclaimer in the documentation and/or other materials provided
      with the distribution.

    * Neither the name of Evan Czaplicki nor the names of other
      contributors may be used to endorse or promote products derived
      from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

install_elm.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/bin/bash

sudo apt-get install libgl1-mesa-dev libglc-dev freeglut3-dev 
sudo apt-get install libedit-dev libglw1-mesa libglw1-mesa-dev
sudo apt-get install ghc
wget http://www.haskell.org/ghc/dist/7.6.3/ghc-7.6.3-src.tar.bz2
tar xjf ghc-7.6.3-src.tar.bz2
cd ghc-7.6.3/mk
cp build.mk.sample build.mk
sed -i 's/^#BuildFlavour = quick/BuildFlavour = quick/' build.mk
cd ..
./configure
make -j 8
sudo make install
cd
platform="2013.2.0.0/haskell-platform-2013.2.0.0.tar.gz"
wget "http://lambda.haskell.org/platform/download/"$platform
tar xzvf haskell-platform-2013.2.0.0.tar.gz
cd haskell-platform-2013.2.0.0
./configure
make
sudo make install
cabal update
cabal install cabal-install
cabal install elm
cabal install elm-server
exit 0

Server.hs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246

{-# OPTIONS_GHC -W #-}
{-# LANGUAGE OverloadedStrings, DeriveDataTypeable #-}
module Main where

import Data.Monoid (mempty)
import qualified Data.ByteString as BS
import qualified Data.ByteString.Char8 as BSC
import qualified Elm.Internal.Utils as Elm
import Control.Applicative
import Control.Monad.Error

import Text.Blaze.Html5 ((!))
import qualified Text.Blaze.Html5 as H
import qualified Text.Blaze.Html5.Attributes as A
import qualified Text.Blaze.Html.Renderer.Utf8 as BlazeBS
import Text.Regex

import Snap.Core
import Snap.Http.Server
import Snap.Util.FileServe
import System.Console.CmdArgs
import System.FilePath as FP
import System.Process
import System.Directory
import GHC.Conc

import qualified Elm.Internal.Paths as Elm
import qualified Generate
import qualified Editor

data Flags = Flags
  { port :: Int
  } deriving (Data,Typeable,Show,Eq)

flags :: Flags
flags = Flags
  { port = 8000 &= help "set the port of the server"
  }

-- | Set up the server.
main :: IO ()
main = do
  setNumCapabilities =<< getNumProcessors
  putStrLn "Initializing Server"
  getRuntimeAndDocs
  setupLogging
  precompile
  cargs <- cmdArgs flags
  httpServe (setPort (port cargs) defaultConfig) $
      ifTop (serveElm "public/Empty.elm")
      <|> route [ ("try", serveHtml Editor.empty)
                , ("edit", edit)
                , ("_edit", jsEdit) 
                , ("code", code)
                , ("_code", jsCode)
                , ("compile", compile)
                , ("_compile", jsCompile)
                , ("hotswap", hotswap)
                ]
      <|> error404

error404 :: Snap ()
error404 =
    do modifyResponse $ setResponseStatus 404 "Not found"
       serveElm "public/build/Error404.elm"

serveElm :: FilePath -> Snap ()
serveElm = serveFileAs "text/html; charset=UTF-8"

logAndServeJS :: MonadSnap m => H.Html -> m ()
logAndServeJS = serveHtml 

logAndServeHtml :: MonadSnap m => (H.Html, Maybe String) -> m ()
logAndServeHtml (html, Nothing)  = serveHtml html
logAndServeHtml (html, Just err) =
    do timeStamp <- liftIO $ readProcess "date" ["--rfc-3339=ns"] ""
       liftIO $ appendFile "error_log.json" $ "{\"" ++ (init timeStamp) 
                                                    ++ "\"," 
                                                    ++ (show (lines err)) 
                                                    ++ "},"
       setContentType "text/html" <$> getResponse
       writeLBS (BlazeBS.renderHtml html)


embedJS :: MonadSnap m => H.Html -> String -> m ()
embedJS js participant =
    do 
       elmSrc <- liftIO $ readFile "EmbedMeJS.elm"
       setContentType "text/html" <$> getResponse
       writeLBS (BlazeBS.renderHtml (embedMe elmSrc js participant))

embedHtml :: MonadSnap m => H.Html -> String -> m ()
embedHtml html participant =
    do elmSrc <- liftIO $ readFile "EmbedMeElm.elm"
       setContentType "text/html" <$> getResponse
       writeLBS (BlazeBS.renderHtml (embedMe elmSrc html participant))

serveHtml :: MonadSnap m => H.Html -> m ()
serveHtml html =
    do setContentType "text/html" <$> getResponse
       writeLBS (BlazeBS.renderHtml html)

hotswap :: Snap ()
hotswap = maybe error404 serve =<< getParam "input"
    where
      serve code =
          do setContentType "application/javascript" <$> getResponse
             writeBS . BSC.pack . Generate.js $ BSC.unpack code

jsCompile :: Snap ()
jsCompile = maybe error404 serve =<< getParam "input"
    where
      serve = logAndServeJS . Generate.logAndJS "Compiled JS" . BSC.unpack

compile :: Snap ()
compile = maybe error404 serve =<< getParam "input"
    where
      serve = logAndServeHtml . Generate.logAndHtml "Compiled Elm" . BSC.unpack

edit :: Snap ()
edit = do
  participant <- BSC.unpack . maybe "" id <$> getParam "p"
  cols <- BSC.unpack . maybe "50%,50%" id <$> getQueryParam "cols"
  withFile (Editor.ide cols participant) 

jsEdit :: Snap ()
jsEdit = do
  participant <- BSC.unpack . maybe "" id <$> getParam "p"
  cols <- BSC.unpack . maybe "50%,50%" id <$> getQueryParam "cols"
  withFile (Editor.jsIde cols participant)

code :: Snap ()
code = do
  participant <- BSC.unpack . maybe "" id <$> getParam "p"
  embedWithFile Editor.editor participant

jsCode :: Snap ()
jsCode = do
  participant <- BSC.unpack . maybe "" id <$> getParam "p"
  jsEmbedWithFile Editor.jsEditor participant

embedee :: String -> String -> H.Html
embedee elmSrc participant =
    H.span $ do
      case Elm.compile elmSrc of
        Right jsSrc -> do
            embed $ H.preEscapedToMarkup (subRegex oldID jsSrc newID)
        Left err ->
            H.span ! A.style "font-family: monospace;" $
            mapM_ addSpaces (lines err)
      script "/fullScreenEmbedMe.js"
  where addSpaces line = H.preEscapedToMarkup (Generate.addSpaces line) >> H.br
        oldID = mkRegex "var user_id = \"1\";"
        newID = ("var user_id = " ++ participant ++ "+'';" :: String)
        jsAttr = H.script ! A.type_ "text/javascript"
        script jsFile = jsAttr ! A.src jsFile $ mempty
        embed jsCode = jsAttr $ jsCode

embedMe :: String -> H.Html -> String -> H.Html
embedMe elmSrc target participant = target >> (embedee elmSrc participant)

jsEmbedWithFile :: (FilePath -> String -> H.Html) -> String -> Snap ()
jsEmbedWithFile handler participant = do
  path <- BSC.unpack . rqPathInfo <$> getRequest
  let file = "public/" ++ path         
  exists <- liftIO (doesFileExist file)
  if not exists then error404 else
      do content <- liftIO $ readFile file
         embedJS (handler path content) participant

embedWithFile :: (FilePath -> String -> H.Html) -> String -> Snap ()
embedWithFile handler participant = do
  path <- BSC.unpack . rqPathInfo <$> getRequest
  let file = "public/" ++ path         
  exists <- liftIO (doesFileExist file)
  if not exists then error404 else
      do content <- liftIO $ readFile file
         embedHtml (handler path content) participant
    
withFile :: (FilePath -> String -> H.Html) -> Snap ()
withFile handler = do
  path <- BSC.unpack . rqPathInfo <$> getRequest
  let file = "public/" ++ path         
  exists <- liftIO (doesFileExist file)
  if not exists then error404 else
      do content <- liftIO $ readFile file
         serveHtml $ handler path content

setupLogging :: IO ()
setupLogging =
    do createDirectoryIfMissing True "log"
       createIfMissing "log/access.log"
       createIfMissing "log/error.log"
    where
      createIfMissing path = do
        exists <- doesFileExist path
        when (not exists) $ BS.writeFile path ""

-- | Compile all of the Elm files in public/, put results in public/build/
precompile :: IO ()
precompile =
  do setCurrentDirectory "public"
     files <- getFiles True ".elm" "."
     forM_ files $ \file -> rawSystem "elm" 
                                ["--make","--runtime=/elm-runtime.js",file]
     htmls <- getFiles False ".html" "build"
     mapM_ adjustHtmlFile htmls
     setCurrentDirectory ".."
  where
    getFiles :: Bool -> String -> FilePath -> IO [FilePath]
    getFiles skip ext directory = 
        if skip && "build" `elem` map FP.dropTrailingPathSeparator 
                                        (FP.splitPath directory)
          then return [] else
          (do contents <- map (directory </>) `fmap` 
                                                getDirectoryContents directory
              let files = filter ((ext==) . FP.takeExtension) contents
                  directories  = filter (not . FP.hasExtension) contents
              filess <- mapM (getFiles skip ext) directories
              return (files ++ concat filess))

getRuntimeAndDocs :: IO ()
getRuntimeAndDocs = do
  writeFile "resources/elm-runtime.js" =<< readFile Elm.runtime
  writeFile "resources/docs.json" =<< readFile Elm.docs

adjustHtmlFile :: FilePath -> IO ()
adjustHtmlFile file =
  do src <- BSC.readFile file
     let (before, after) = BSC.breakSubstring "<title>" src
     BSC.writeFile (FP.replaceExtension file "elm") $
        BSC.concat [before, style, after]
     removeFile file

style :: BSC.ByteString
style = 
    "<style type=\"text/css\">\n\
    \  a:link {text-decoration: none; color: rgb(15,102,230);}\n\
    \  a:visited {text-decoration: none}\n\
    \  a:active {text-decoration: none}\n\
    \  a:hover {text-decoration: underline; color: rgb(234,21,122);}\n\
    \  body { font-family: \"Lucida Grande\",\"Trebuchet MS\",\
    \  \"Bitstream Vera Sans\",Verdana,Helvetica,sans-serif !important; }\n\
    \  p, li { font-size: 14px !important;\n\
    \          line-height: 1.5em !important; }\n\
    \</style>"

Editor.hs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225

{-# LANGUAGE OverloadedStrings #-}
module Editor (editor,jsEditor,jsIde,ide,empty) where

import Data.Monoid (mempty)
import Text.Blaze.Html
import qualified Text.Blaze.Html5 as H
import qualified Text.Blaze.Html5.Attributes as A
import Network.HTTP.Base (urlEncode)
import qualified System.FilePath as FP

import qualified Elm.Internal.Utils as Elm
import Data.Maybe (fromMaybe)

import Generate (addSpaces)


-- | Display an editor and the compiled result side-by-side.
jsIde :: String -> String -> FilePath -> String -> Html
jsIde cols participant fileName code =
    jsIdeBuilder cols
                 participant
                 ("JS Editor: " ++ FP.takeBaseName fileName)
                 fileName
                 ("/_compile?input=" ++ urlEncode code)

-- | Display an editor and the compiled result side-by-side.
ide :: String -> String -> FilePath -> String -> Html
ide cols participant fileName code =
    ideBuilder cols
               participant
               ("Elm Editor: " ++ FP.takeBaseName fileName)
               fileName
               ("/compile?input=" ++ urlEncode code)

-- | Display an editor and the compiled result side-by-side.
empty :: Html
empty = ideBuilder "50%,50%" "1" "Try Elm" "Empty.elm" "/Try.elm"

jsIdeBuilder :: String -> String -> String -> String -> String -> Html
jsIdeBuilder cols participant title input output =
    H.docTypeHtml $ do
      H.head . H.title . toHtml $ title
      preEscapedToMarkup $ 
         concat [ "<frameset cols=\"" ++ cols ++ "\">\n"
          , "  <frame name=\"input\" src=\"/_code/", input, "?p=", 
            participant, "\" />\n"
          , "  <frame name=\"output\" src=\"", output, "\" />\n"
          , "</frameset>" ]


ideBuilder :: String -> String -> String -> String -> String -> Html
ideBuilder cols participant title input output =
    H.docTypeHtml $ do
      H.head . H.title . toHtml $ title
      preEscapedToMarkup $ 
         concat [ "<frameset cols=\"" ++ cols ++ "\">\n"
                , "  <frame name=\"input\" src=\"/code/", input, "?p=", 
                  participant, "\" />\n"
                , "  <frame name=\"output\" src=\"", output, "\" />\n"
                , "</frameset>" ]

-- | list of themes to use with CodeMirror
themes :: [String]
themes = [ "ambiance", "blackboard", "cobalt", "eclipse"
         , "elegant", "erlang-dark", "lesser-dark", "monokai", "neat", "night"
         , "rubyblue", "solarized", "twilight", "vibrant-ink", "xq-dark" ]

jsFiles :: [AttributeValue]
jsFiles = [ "/codemirror-3.x/lib/codemirror.js"
          , "/codemirror-3.x/mode/elm/elm.js"
          , "/misc/showdown.js"
          , "/misc/editor.js?0.11" ]

jsFiles2 :: [AttributeValue]
jsFiles2 = [ "/codemirror-3.x/lib/codemirror.js"
          , "/codemirror-3.x/mode/javascript/javascript.js"
          , "/misc/showdown.js"
          , "/misc/editor.js?0.11" ]

jsEditor :: FilePath -> String -> Html
jsEditor filePath code =
    H.html $ do
      H.head $ do
        H.title . toHtml $ "JS Editor: " ++ FP.takeBaseName filePath
        H.link ! A.rel "stylesheet" 
               ! A.href "/codemirror-3.x/lib/codemirror.css"
        mapM_ themeAttr themes
        H.link ! A.rel "stylesheet" ! A.type_ "text/css" 
                                    ! A.href "/misc/editor.css"
        mapM_ script jsFiles2
        script "/elm-runtime.js?0.11"
        script "http://cdn.firebase.com/v0/firebase.js"
      H.body $ do
        H.form ! A.id "inputForm" 
               ! A.action "/_compile" 
               ! A.method "post" 
               ! A.target "output" $ do
           H.div ! A.id "editor_box" $
             H.textarea ! A.name "input" ! A.id "input" $ toHtml ('\n':code)
           H.div ! A.id "options" $ do
             bar "documentation" docs
             bar "editor_options" editorOptions
             bar "always_on" (buttons >> options)
        embed "initEditor();"
  where themeAttr theme = H.link ! A.rel "stylesheet" 
                                 ! A.href (toValue ("/codemirror-3.x/theme/" 
                                                      ++ theme 
                                                      ++ ".css" :: String))
        jsAttr = H.script ! A.type_ "text/javascript"
        script jsFile = jsAttr ! A.src jsFile $ mempty
        embed jsCode = jsAttr $ jsCode

-- | Create an HTML document that allows you to edit and submit Elm code
--   for compilation.
editor :: FilePath -> String -> Html
editor filePath code =
    H.html $ do
      H.head $ do
        H.title . toHtml $ "Elm Editor: " ++ FP.takeBaseName filePath
        H.link ! A.rel "stylesheet" 
               ! A.href "/codemirror-3.x/lib/codemirror.css"
        mapM_ themeAttr themes
        H.link ! A.rel "stylesheet" ! A.type_ "text/css" 
                                    ! A.href "/misc/editor.css"
        mapM_ script jsFiles
        script "/elm-runtime.js?0.11"
        script "http://cdn.firebase.com/v0/firebase.js"
      H.body $ do
        H.form ! A.id "inputForm" 
               ! A.action "/compile" 
               ! A.method "post" 
               ! A.target "output" $ do
           H.div ! A.id "editor_box" $
             H.textarea ! A.name "input" ! A.id "input" $ toHtml ('\n':code)
           H.div ! A.id "options" $ do
             bar "documentation" docs
             bar "editor_options" editorOptions
             bar "always_on" (buttons >> options)
        embed "initEditor();"
  where themeAttr theme = H.link ! A.rel "stylesheet" 
                                 ! A.href (toValue ("/codemirror-3.x/theme/" 
                                                      ++ theme 
                                                      ++ ".css" :: String))
        jsAttr = H.script ! A.type_ "text/javascript"
        script jsFile = jsAttr ! A.src jsFile $ mempty
        embed jsCode = jsAttr $ jsCode

bar :: AttributeValue -> Html -> Html
bar id' body = H.div ! A.id id' ! A.class_ "option" $ body

buttons :: Html
buttons = H.div ! A.class_ "valign_kids"
                ! A.style "float:right; padding-right: 6px;"
                $ compileButton
      where
        compileButton = 
            H.input
                 ! A.type_ "button"
                 ! A.id "compile_button"
                 ! A.value "Compile"
                 ! A.onclick "compile()"
                 ! A.title "Ctrl-Enter: change program behavior \
                             \but keep the state"

options :: Html
options = H.div ! A.class_ "valign_kids"
                ! A.style "float:left; padding-left:6px; padding-top:2px;"
                $ (docs' >> opts)
    where 
      docs' = 
        H.span  ! A.title "Show documentation and types." $ "Hints:" >>
            H.input ! A.type_ "checkbox"
                    ! A.id "show_type_checkbox"
                    ! A.onchange "showType(this.checked);"

      opts = 
        H.span  ! A.title "Show editor options." 
                ! A.style "padding-left: 12px;" $ "Options:" >>
            H.input ! A.type_ "checkbox"
                    ! A.id "options_checkbox" 
                    ! A.onchange "showOptions(this.checked);"

editorOptions :: Html
editorOptions = theme >> zoom >> lineNumbers
    where
      optionFor :: String -> Html
      optionFor text =
          H.option ! A.value (toValue text) $ toHtml text

      theme =
          H.select ! A.id "editor_theme"
                   ! A.onchange "setTheme(this.value)"
                   $ mapM_ optionFor themes
              
      zoom =
          H.select ! A.id "editor_zoom"
                   ! A.onchange "setZoom(this.options[this.selectedIndex].\
                                  \innerHTML)"
                   $ mapM_ optionFor ["100%", "80%", "150%", "200%"]

      lineNumbers = do
        H.span ! A.style "padding-left: 16px;" $ "Line Numbers:"
        H.input ! A.type_ "checkbox"
                ! A.id "editor_lines"
                ! A.onchange "showLines(this.checked);"

docs :: Html
docs = tipe >> desc
    where
      tipe = H.div ! A.class_ "type" $ message >> more

      message = H.div ! 
                  A.style "position:absolute; left:4px; right:36px;\
                            \overflow:hidden; text-overflow:ellipsis;" $ ""

      more = H.a ! A.id "toggle_link"
                 ! A.style "display:none; float:right;"
                 ! A.href "javascript:toggleVerbose();"
                 ! A.title "Ctrl+H"
                 $ ""

      desc = H.div ! A.class_ "doc"
                   ! A.style "display:none;"
                   $ ""

Generate.hs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118

{-# LANGUAGE OverloadedStrings #-}
module Generate (logAndJS, logAndHtml, html, js, addSpaces) where

import Data.Monoid (mempty)
import Data.Maybe (fromMaybe)
import Text.Blaze (preEscapedToMarkup)
import Text.Blaze.Html5 ((!))
import qualified Text.Blaze.Html5 as H
import qualified Text.Blaze.Html5.Attributes as A

import qualified Elm.Internal.Utils as Elm
import Utils

logAndJS :: String -> String -> H.Html
logAndJS name src = getJSPage name src

logAndHtml :: String -> String -> (H.Html, Maybe String)
logAndHtml name src =
    let elmname = "Elm." ++ fromMaybe "Main" (Elm.moduleName src) 
    in
      case Elm.compile src of
          Right jsSrc -> do
              (getHtmlPage name elmname jsSrc, Nothing)
          Left err -> do
              (getErrPage name err, Just err)

getJSPage :: String -> String -> H.Html
getJSPage name jsSrc =
  H.docTypeHtml $ do
      H.head $ do
        H.meta ! A.charset "UTF-8"
        H.title . H.toHtml $ name
        H.link ! A.rel "stylesheet" ! A.type_ "text/css" 
                                    ! A.href "/misc/js.css"
        script "/pixi.js"
      H.body $ do
        H.div ! A.style "width: 400px; height: 400px; position:\
                        \ absolute; top: 0; left: 0; opacity: 0;" $ mempty
        jsAttr $ preEscapedToMarkup jsSrc
 where jsAttr = H.script ! A.type_ "text/javascript"
       script jsFile = jsAttr ! A.src jsFile $ mempty
       embed jsCode = jsAttr $ jsCode

getHtmlPage :: String -> String -> String -> H.Html
getHtmlPage name elmname jsSrc =
  H.docTypeHtml $ do
      H.head $ do
        H.meta ! A.charset "UTF-8"
        H.title . H.toHtml $ name
        H.style ! A.type_ "text/css" $ preEscapedToMarkup
         ("a:link {text-decoration: none; color: rgb(15,102,230);}\n\
          \a:visited {text-decoration: none}\n\
          \a:active {text-decoration: none}\n\
          \a:hover {text-decoration: underline; \
          \color: rgb(234,21,122);}" :: String)
      H.body $ do
        let js = H.script ! A.type_ "text/javascript"
            runFullscreen = 
                "var runningElmModule = Elm.fullscreen(" ++ elmname 
                                                         ++ ")"
        js ! A.src (H.toValue ("/elm-runtime.js?0.11" :: String)) $ ""
        js $ preEscapedToMarkup jsSrc
        js $ preEscapedToMarkup runFullscreen

getErrPage :: String -> String -> H.Html
getErrPage name err =
  H.docTypeHtml $ do
      H.head $ do
        H.meta ! A.charset "UTF-8"
        H.title . H.toHtml $ name
      H.body $
        H.span ! A.style "font-family: monospace;" $
        mapM_ (\line -> preEscapedToMarkup (addSpaces line) >> H.br) 
                (lines err)
    
            

-- | Using a page title and the full source of an Elm program, compile down to
--   a valid HTML document.
html :: String -> String -> H.Html
html name src =
  H.docTypeHtml $ do
      H.head $ do
        H.meta ! A.charset "UTF-8"
        H.title . H.toHtml $ name
        H.style ! A.type_ "text/css" $ preEscapedToMarkup
         ("a:link {text-decoration: none; color: rgb(15,102,230);}\n\
          \a:visited {text-decoration: none}\n\
          \a:active {text-decoration: none}\n\
          \a:hover {text-decoration: underline;\
          \ color: rgb(234,21,122);}" :: String)
      H.body $ do
        let js = H.script ! A.type_ "text/javascript"
            elmname = "Elm." ++ fromMaybe "Main" (Elm.moduleName src)
            runFullscreen = 
                  "var runningElmModule = Elm.fullscreen(" ++ elmname 
                                                           ++ ")"
        js ! A.src (H.toValue ("/elm-runtime.js?0.11" :: String)) $ ""
        case Elm.compile src of
          Right jsSrc -> do
              js $ preEscapedToMarkup jsSrc
              js $ preEscapedToMarkup runFullscreen
          Left err ->
              H.span ! A.style "font-family: monospace;" $
              mapM_ (\line -> preEscapedToMarkup (addSpaces line) >> H.br) 
                      (lines err)

addSpaces :: String -> String
addSpaces str =
  case str of
    ' ' : ' ' : rest -> "  " ++ addSpaces rest
    c : rest -> c : addSpaces rest
    [] -> []

js :: String -> String
js src = case Elm.compile src of
           Right js -> "{ \"success\" : " ++ show js ++ " }"
           Left err -> "{ \"error\" : " ++ show err ++ " }"

EmbedMeElm.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

module EmbedMe where

import Mouse
import Window
import Keyboard
import JavaScript as JS
import JavaScript.Experimental as JEXP
import Http
import Json

(~>) = flip lift
infixl 4 ~>

clicks : Signal (Time, (Int,Int))
clicks = timestamp (sampleOn Mouse.isDown Mouse.position)

user_id = "1"

firebaseRequest requestType requestData = 
  Http.request requestType 
    ("https://sweltering-fire-9141.firebaseio.com/dissertation/elm/" ++ user_id 
                                                                     ++ ".json") 
    requestData []
 
serialize r = r |> JEXP.fromRecord 
                |> Json.fromJSObject 
                |> Json.toJSString " " 
                |> JS.toString
 
toRequestData (t, (x,y)) = {t = t, x = x, y = y} |> serialize
 
toRequest event = case event of 
  (t, (x,y)) -> firebaseRequest "post" (event |> toRequestData)
 
requests = clicks ~> toRequest
 
sendRequests = Http.send requests

EmbedMeJS.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

module EmbedMe where

import Mouse
import Window
import Keyboard
import JavaScript as JS
import JavaScript.Experimental as JEXP
import Http
import Json

(~>) = flip lift
infixl 4 ~>

clicks : Signal (Time, (Int,Int))
clicks = timestamp (sampleOn Mouse.isDown Mouse.position)

user_id = "1"

firebaseRequest requestType requestData = 
  Http.request requestType 
    ("https://sweltering-fire-9141.firebaseio.com/dissertation/js/" ++ user_id 
                                                                    ++ ".json") 
    requestData []
 
serialize r = r |> JEXP.fromRecord 
                |> Json.fromJSObject 
                |> Json.toJSString " " 
                |> JS.toString
 
toRequestData (t, (x,y)) = {t = t, x = x, y = y} |> serialize
 
toRequest event = case event of 
  (t, (x,y)) -> firebaseRequest "post" (event |> toRequestData)
 
requests = clicks ~> toRequest
 
sendRequests = Http.send requests

fullScreenEmbedMe.js

1
2
3

var firebaseData = new Firebase(
		'https://sweltering-fire-9141.firebaseio.com/dissertation');
var embedMe = Elm.fullscreen(Elm.EmbedMe, {});

editor.js.diff

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

diff --git a/resources/misc/editor.js b/resources/misc/editor.js
index d2bebc8..302663e 100644
--- a/resources/misc/editor.js
+++ b/resources/misc/editor.js
@@ -293,7 +293,7 @@ function showOptions(show) {
 function showType(show) {
     cookie('showtype', show);
     document.getElementById('show_type_checkbox').checked = show;
-    var newMode = (show ? { mode: Mode.TYPES, verbose: false }
+    var newMode = (show ? { mode: Mode.TYPES, verbose: true}
                         : { mode: Mode.NONE });
     if (mode.mode === Mode.OPTIONS) {
         mode.hidden = newMode;
@@ -305,8 +305,8 @@ function showType(show) {
 
 function toggleVerbose() {
     if (!mode.verbose) showType(true);
-    document.getElementById('toggle_link').innerHTML = mode.verbose ? 
-      'more' : 'less';
-    mode.verbose = !mode.verbose;
+    document.getElementById('toggle_link').innerHTML = mode.verbose ? '' : '';
+    mode.verbose = true;
     updateDocumentation();
 }
 
@@ -318,8 +318,8 @@ function showVerbose() {
 function hideStuff() {
     if (mode.hidden) mode = mode.hidden;
     document.getElementById('options_checkbox').checked = false;
-    mode.verbose = false;
-    document.getElementById('toggle_link').innerHTML = 'more';
+    mode.verbose = true;
+    document.getElementById('toggle_link').innerHTML = '';
     updateDocumentation();
 }

MovingBox.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

/* 

Try moving the square around with your keyboard's arrow keys

Click your mouse over there =====> 
Use arrows Up, Down, Left, Right

Whee!

Now modify the code to prevent the square from going outside 
the edge of the grey window.

*/

var WIDTH = 400;
var HEIGHT = 400;
var SQUARE = 40;
var COLORS = [
    "0x000000",
    "0xCCCCCC",
];
var MOVEMENT_SPEED = 5;

var stage = new PIXI.Stage(COLORS[1]);
var renderer = PIXI.autoDetectRenderer(WIDTH, HEIGHT);
document.body.appendChild(renderer.view);

var box = new PIXI.Graphics();
box.lineStyle(1, COLORS[0], 1);
box.beginFill(COLORS[1], 0);
box.drawRect(0, 0, SQUARE, SQUARE);
box.endFill();
stage.addChild(box);

box.x = (WIDTH / 2) - (SQUARE / 2);
box.y = (HEIGHT / 2) - (SQUARE / 2);

var keyState = {};

window.addEventListener('keydown', function(e) {
    keyState[e.keyCode || e.which] = true;
}, true);

window.addEventListener('keyup', function(e) {
    keyState[e.keyCode || e.which] = false;
}, true);

requestAnimFrame(animate);

function animate() {
    if (keyState[37]) {
        box.x -= MOVEMENT_SPEED;
    }

    if (keyState[38]) {
        box.y -= MOVEMENT_SPEED;
    }

    if (keyState[39]) {
        box.x += MOVEMENT_SPEED;
    }

    if (keyState[40]) {
        box.y += MOVEMENT_SPEED;
    }

    renderer.render(stage);
    requestAnimFrame(animate);
}

MovingBox.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52

{- 

Try moving the square around with your keyboard's arrow keys

Click your mouse over there =====> 
Use arrows Up, Down, Left, Right

Whee!

Now modify the code to prevent the square from going outside 
the edge of the grey window.

-}

import Keyboard

areaSize = 400
squareSize = 40

main : Signal Element
main = lift display position

delta : Signal Float
delta = fps 30

input : Signal (Float, (Float,Float))
input =
    let vectors = lift toVector Keyboard.arrows
    in  sampleOn delta (lift2 (,) delta vectors)

toVector : { x:Int, y:Int } -> (Float,Float)
toVector {x,y} =
    if x /= 0 && y /= 0
      then (x / sqrt 2, y / sqrt 2)
      else (x,y)

position : Signal (Float,Float)
position = foldp update (0,0) input

update : (Float, (Float,Float)) -> (Float,Float) -> (Float,Float)
update (dt,(vx,vy)) (x,y) =
    (x + dt * vx / 2, y + dt * vy / 2)

display : (Float,Float) -> Element
display xy =
    collage (round areaSize) (round areaSize)
      [ rect areaSize areaSize
          |> filled grey
      , rect squareSize squareSize
          |> outlined (solid black)
          |> move xy
      ]

DecodeMouseData.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

#!/usr/bin/python2.7

import json
import sys

class DecodeMouseData(object):

    def decode(self, jsonString):
        return json.loads(jsonString)

    def getNumberOfClicks(self, jsonString):
        return len(self.decode(jsonString))

    def getSessionDuration(self, jsonString):
        decoded = self.decode(jsonString)

        finish = max([x["t"] for x in decoded.values()])
        start = min([x["t"] for x in decoded.values()])

        # compute number of minutes in this number of milliseconds
        return (finish - start) / 60000.0

    def getFilesTimeClickDict(self, files):
        timeAndClicks = dict()
        filename = str()
        for jsonFile in files:
            with open(jsonFile) as f:
                jsonString = f.read()
                filename = f.name
                print filename
            timeAndClicks[filename] = (self.getSessionDuration(jsonString),
                    self.getNumberOfClicks(jsonString))
        return timeAndClicks

    def getDictPrettyPrint(self, timeAndClicks):
        output = ""
        tablelines = "-"*11 + " " + "-"*10
        output += tablelines + "\n"
        output += "Time (min)  Clicks\n"
        output += tablelines + "\n"
        length = len(timeAndClicks)
        for i,filename in enumerate(timeAndClicks):
            output += ("%10f %10d" % (timeAndClicks[filename][0],
                    timeAndClicks[filename][1])) + "\n"
        output += tablelines
        return output


if __name__ == "__main__":
    dmd = DecodeMouseData()
    print dmd.getDictPrettyPrint(dmd.getFilesTimeClickDict(sys.argv[1:]))

test_DecodeMouseData.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74

import unittest
import DecodeMouseData as d


class FooTests(unittest.TestCase):

    def setUp(self):
        self.dmd = d.DecodeMouseData()

    def testDecode(self):
        expected = {'1':2, '3':4}
        actual = self.dmd.decode('{"1":2, "3":4}')

        self.assertEquals(actual, expected)

    def testMouseDecode(self):
        expected = {"-JKMBewWrFje3lHT8spD" :
                {"t" : 1397327310399, "y" : 646, "x" : 629}}
        actual = self.dmd.decode(
                '{"-JKMBewWrFje3lHT8spD" : ' +
                '{"t" : 1397327310399, "y" : 646, "x" : 629}}')

        self.assertEquals(actual, expected)

    def testNumClicks(self):
        expected = 1
        actual = self.dmd.getNumberOfClicks(
                '{"-JKMBewWrFje3lHT8spD" : ' +
                '{"t" : 1397327310399, "y" : 646, "x" : 629}}')

        self.assertEquals(actual, expected)

    def testLotsClicks(self):
        expected = 2

        actual = self.dmd.getNumberOfClicks("""{
  "-JKMBewWrFje3lHT8spD" : {
    "t" : 1397327310399,
    "y" : 646,
    "x" : 629
  },
  "-JKMBewawNo6G_Zdfnkk" : {
    "t" : 1397327310465,
    "y" : 646,
    "x" : 629
  }
}""")

        self.assertEquals(actual, expected)

    def testComputeSessionTime(self):
        expected = 0.0011

        actual = self.dmd.getSessionDuration("""{
  "-JKMBewWrFje3lHT8spD" : {
    "t" : 1397327310399,
    "y" : 646,
    "x" : 629
  },
  "-JKMBewawNo6G_Zdfnkk" : {
    "t" : 1397327310465,
    "y" : 646,
    "x" : 629
  }
}""")

        self.assertEquals(actual, expected)

def main():
    unittest.main()

if __name__ == '__main__':
    main()

Appendix D

Pilot Study 1 -- Questionnaire

Consent Form

Study Overview

This study aims to assess how Functional Reactive Programming Languages are used. To do this, we will be asking you to modify a Mario game to get him to fly. The session will take no more than 1 hour.

During the session, you will be introduced to Elm, a functional reactive programming language, as well as being shown what we want you to create. We’ll also present you with a questionnaire to see what experience you’ve had with Functional programming (or similar concepts) before. Finally we’ll give you another questionnaire to ask how you think the session went, and the level of workload in the task.

The session will be recorded on video and then the audio from the session will be transcribed anonymously in order to find any problems that you had during the session. During this process, the data will be stored securely. Important Information

All data collected during this study will be recorded such that your individual results are anonymous and cannot be traced back to you. Your results will not be passed to any third party and are not being collected for commercial reasons. Participation in this study does not involve physical or mental risks outside of those encountered in everyday life. All procedures and information can be taken at face value and no deception is involved. You have the right to withdraw from the study at any time and to have any data about you destroyed. If you do decide to withdraw, please inform the experimenter.

By signing this form you acknowledge that you have read the information given above and understand the terms and conditions of this study.

Signed

• .................

Date

• .................

Experimenter: Simon Buist, Dept. of Computer Science. EMAIL ADDRESS

Pre-questionnaire

Functional Programming languages

1. Have you ever used a Functional programming language before? Examples are: Scheme, Lisp, Haskell, ML, SPARK. Please circle one answer)
   • Yes
   • No
2. If so, please list the Functional programming languages you’ve used before:
   • .............................................................

Design & Software

For the purposes of this questionnaire, we consider a piece of software to be an application for which you have received/conceived of a specification, and coded a solution that meets this solution.

1. Have you designed software before?
   • Yes
   • No
2. On what platforms have you designed software?
   • Desktop
   • Mobile
   • Tablet
   • Web
3. For what purposes have you designed software?
   • Commercial
   • Academic (e.g. Coursework)
   • Personal project
   • Other:
4. Roughly how many pieces of software have you designed?
   • ...
5. What programming languages do you know?
   • .............................................................

General Demographics

1. How old are you?
   • ...
2. What is your sex? (Please circle one answer)
   • Male
   • Female
3. What is the highest degree or level of education you have completed? If currently enrolled please indicate the highest you have attained previously. (Please circle one answer)
   • None
   • GCSEs or equivalent
   • A/AS levels or equivalent
   • BSc/BA or equivalent
   • MSc/MA or equivalent
   • PhD or equivalent
4. In what field was your highest qualification?
   • .........................
5. What is your current employment status? (Please circle one answer)
   • Unemployed
   • Self-employed
   • Employed
   • Student
   • Retired
   • Unable to work

Post-Questionnaire

1. Please detail any comments on the result that you achieved

.............................................................

.............................................................

.............................................................

2. Please detail any comments on how you achieved it

.............................................................

.............................................................

.............................................................

3. Please detail any other comments

.............................................................

.............................................................

.............................................................

If you want to have the study as a whole explained to you, please do so now. However we ask that you refrain from discussing this with potential future participants.