Archives for posts with tag: Novice programmers

In today’s lab meeting we were discussing another paper, this time on the emotions experienced by novice programmers during their very first programming lesson. It was presented by Bosch et al* at the 2013 conference of Artificial Intelligence in Education.  The study participants were undergraduate students from the Psychology Student Pool who had no experience of programming. These students underwent a 3-stage process over a period of 2 hours: first they completed a series of “scaffolded” problems, with hints and help available. Then they completed two “fadeout” exercises, with no help available. Finally the students were asked to review the videos of themselves completing the first two exercises and asked to select what their emotions were at various points from a subset of Pekrun’s taxonomy of academic emotions.

The authors found that four emotions were experienced more frequently than the others: flow/engagement, confusion, frustration and boredom. The proportions of these apparently varied depending on which of the first two stages of the experiment (scaffolding/fadeout) the students were performing. They also correlated the emotions with the success of the students at completing the tasks, and attempted to systematically link them to three student behaviours (running the code, idling, or constructing the code).

As ever, having a paper to start the conversation produced a lively debate! One of our big problems as a group centres on the idea of flow/engagement being the desirable state for learning. As teachers and learners, we have found that periods of frustration are not altogether bad, and the joy of passing through that frustration to understanding is sometimes one of the key highs of learning. Equally, what externally looks like flow/engagement may very well not be. There have been other studies that the group knew of where people looked like they were in this flow/engagement state, but when the researchers examined what they had produced it was rubbish. What was felt might be a better approach would be to try to examine the routes through the different emotional states, and correlate those routes with different outcomes. The challenge then becomes how to identify that someone is on a destructive pathway and, crucially, what (pedagogically) to do about it, rather than reacting instantly to prevent frustration.

The use of the thirteen emotions also raised some issues for the group. By limiting the responses, in a sense the researchers are pre-determining their outcomes. There also didn’t appear to be any agreement with the subjects on what the terms actually meant to them. Jim’s experience in counselling suggests that when dealing with emotions you sometimes have to explain what the word refers to before people understand what they are feeling. Equally some of the group remembered working a project with Madeline Balaam where they negotiated the terms to be used in reporting the emotions with a group of students, rather than predetermining them.The group also thought this would have been quite an interesting project to use the same techniques that Pejman used, where he used biometric data to highlight areas that were worth talking about with a player, rather than asking them for feedback on blocks of around 20 secs at a time.

We also came up with some interesting ideas for follow-up studies. Obviously in this experiment they were relating the emotion to results within the same session. However, we felt it would be valuable to do a slightly more longitudinal study, and relate the emotions in the first lesson to outcomes after an entire term of study. This would help to demonstrate the importance (or maybe unimportance) of the first lesson. Also, we thought it would be interesting to try the same study with a group of computer scientists, or at least with students who had expressed some level of interest in trying to learn to program. The comparison in the emotions experienced in the two groups could be helpful in exploring the role of motivation in the emotions and outcome.

The group were very definitely thoroughly engaged by this paper, which touched on so many areas of expertise for all of us. An entertaining and thoroughly enjoyable discussion.

*Bosch, N., D’Mello, S., & Mills, C. (2013). What Emotions Do Novices Experience During their First Computer Programming Learning Session. In H. C. Lane, K. Yacef, J. Mostow, & P. Pavlik (Eds.), Artificial Intelligence in Education (Vol. 7926, pp. 11–20). Berlin, Heidelberg: Springer Berlin Heidelberg. doi:10.1007/978-3-642-39112-5


This Tuesday Edgar presented an overview of his PhD research, mainly focussing on his current task derived from his last experiment carried out in Mexico. The aim of the project is to explore the usefulness of natural language (NL) descriptions for small algorithms when novices are learning the basic concepts of programming (input/output, assignment and loops).

Programming environments, such as Scratch or Alice, are based on graphical manipulation interface and error-free syntax, to support programmers. However, these environments still missing a way to “translate” or express programs in a language that is familiar to novice programmers, thus users cannot easily verify whether the current solution matches their intention. Edgar’s research explores the usefulness of providing a second representation to support a visual language (in this case a flowchart representation) designed for novice programmers. He aims to assess the benefits of a second representation when comprehension, creation and debugging programming skills are involved.

The main inspiration for exploring natural language as tool to support novice programmers comes from FLIP, a system that combines a visual language based on interlocking blocks with natural language support.

flip screenshot

Flip interface: (a) scripting editor top left, (b) natural language description bottom left, (c) toolbox of scripting instructions on the right side.

This research uses a tool called Origami, which is used to teach basic programming concepts to novices in Universidad Autonoma de Yucatan, Mexico. This tool enables students to create, test and debug programs using a visual language based on flowcharts. The programs produced are syntactically error-free, however, students may input instructions into each flowchart block that are semantically incorrect. Origami logs interaction data during the problem solving process: time taken to complete an exercise, the blocks used, the inner blocks’ instructions and any compilation messages. Each interaction is associated with a timestamp that preserves the chronology of the problem solving process.

The Origami tool was improved using some design ideas from Flip and now combines a visual language based on interlocking blocks with natural language

origami screenshot

Origami + Flip-ish = Origami + Natural Language Descriptions

Last August Edgar headed back to Mexico to carry out his experiment. The experiment considered a comparative study with three conditions: the earlier version of Origami (no secondary representation), a version with a secondary Natural Language description and a version with a secondary pseudo code representation. Some details about the experiment:

  • Participants: 75 first year students of a Computer Science 1st year  course (63 males and 12 females), aged between 17 and 21 years old. Three groups were formed and were taught the same material by two tutors
  • Materials: The materials used in the study were a programming pre-test and post-test, the Origami tool (3 versions), and 3 sets of algorithmic exercises. The pre-test and post-test and each set of algorithmic exercises consisted of 3 type of exercises (completion, debugging and tracing) each with three difficulty levels (simple, medium and complex). The format of the items for the pre- and post-test were based on multiple choice questions (debugging and tracing) and open questions (completion), and were administered via the online course delivery tool.
  • Method: The experiment was carried out over a period of approximately 6 weeks. In addition to the practical work there were a total of 12 sessions (24 hrs), in which the basic concepts of programming (variable, sentences, conditions, loops) were taught. The first week the pre-test was applied. The 3 sets of algorithmic exercises were assessed as coursework during the weeks 2 to 5. Each set of algorithmic exercises was divided into two subsets: (1) simpler exercises and (2) medium & complex exercises. Finally, the post-test was administered during the last week of the experiment.
Experimental design

Experiment: Three conditions, Pre and Post Test, and three sets of exercises.

Finally after some really hot weeks in the south of Mexico, Edgar came back to the UK with a bag full of Mexican sauces and lots of data. Currently he’s working through the sauces, trying to keep warm, and looking at the pre- and post-test analysis of the explanation exercises from the students, where the students were asked to explain the solution for a math-related problem in terms of an algorithm. He’s using more detailed rubric, and at the same time detecting errors, using an omission/commission classification based on a previous work from Judith, Katy and Keiron.* Lots of analysis to do!

The session was full of suggestions about different and interesting kinds of analysis. Ben suggested (and Judith strongly agreed) that the analysis of the pre- and post-tests should be done blind to which groups the individuals were in. Ellie asked if the basic Origami system has ever been compared with the paper version it replaced, which made Edgar realise that the people using the basic Origami were still asking for paper to scribble on, while the other two groups weren’t. There appears to be an early indication that the group who had the natural language representation were better at explaining their solutions in the post-test, which caused the group to suggest that maybe this was equipping the students with the language to talk about what they have done.

We are back in our normal Interact Lab next week, at the normal time. Hopefully we’ll get a full house, as we have a special guest appearance!

*Good, J., Howland, K., & Nicholson, K. (2010). Young People’s Descriptions of Computational Rules in Role-Playing Games: An Empirical Study. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing (pp. 67–74). IEEE.

We had a new attendee this week. We have Trevor Nesbit visiting from Christchurch, New Zealand until the end of December. Hopefully we will find out a little more about his work in a later lab meeting, but for this meeting we let him off with a quick introduction and let him settle in and listen to Judith.

Judith was talking to the group about the results of a course she taught last year – Technology Enhanced Learning Environments (TELE). A core component of the course was a group project, and to help to make this project more interesting and relevant Judith asked contacts from a group called Digital Education Brighton to get involved. Three different schools came forward, and the groups were asked to go through the process of gathering requirements by a range of means and producing a motivating learning experience for secondary school children around programming. Judith had three of the finished projects that she wanted to share with us.

Project 1 was a very nicely produced set of resources for teachers, including YouTube tutorials and a PowerPoint self-guided learning system that lead the students through creating a small game in Scratch. The graphics were fantastic, although Judith felt that the material itself was somewhat behaviouralist and not particularly ground-breaking.

Project 2 was a great little game called Blobs, which made use of a puzzle-solving mechanic not unlike lemmings to explore the differences between classes and objects with properties and methods. A series of different types of blob with a variety of different properties and methods had to be used together to solve levels. Pejman had a little difficulty with the third level, so this is by no means straightforward! Again, high production values and a lot of thought had gone into this, but it would be interesting to see whether this did actually help children understand coding. Ben asked if in later levels they introduced actual code, which would be an obvious next step that hadn’t been completed.

Project 3 was awesome. Rather than teach programming (the school they were working with already did rather well at that) they looked at software design, with a particular emphasis on user-centred processes. It was a flash program (I’m hesitating to call it a game) that students could go through, collecting requirements from a school that needed a particular piece of software building. Different parts of the scenario required different data-gathering methods – e.g. A recorded message from the headmaster, a questionnaire for the parents, focus groups etc. There were extra bits of information available on all the methods in program, which was very polished. In addition to this, they had prepared lesson plans for the teacher, work sheets to go with the lesson plans, presentations for the teacher to go through for each lesson… It was extremely complete, and a very interesting angle to take on the problem.

Judith’s main problem is that she wants to make these resources available (along with others from other years), either for teachers to use as is, or for further development/modification. The obvious way to do this is via the web, and the group had a variety of suggestions for how that could be achieved (many featured undergraduate labour – if you are an undergraduate who fancies building this let us know!). It was good to hear about the course and the projects.

Next week features a change of venue, as the Interact Lab is being used for some teaching. We will therefore be colonising the IDEAs lab from 11-12 next Tuesday instead. We remembered biscuits for this week, so those frozen cakes still remain and will be out of reach next week. There’s a good chance the cakes will make it to Christmas at this rate!

This week we had a paper to review, courtesy of Edgar. The paper was “Using HCI Techniques to Design a More Usable Programming System” by Pane, Myers and Miller*, which was recently jointly awarded the “Most influential paper from approximately one decade ago” at the IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2012. That alone makes it worthy of reviewing, but a number of the group also have an interest in visual languages and designing systems to help novice programmers.

The paper gives quite a high level account of the design and initial testing of the HANDS programming system. The design is aimed at children of grade 5 and above, which Judith kindly translated for those of us not familiar with the US grade system as age 10 and over. The authors begin by undertaking a user study to see how non-programmers approached problems they designed, and what sort of constructs and language were used. The programming language and environment were then designed together, with the visual metaphors aiming to underpin the coding structure. This design was then re-tested.

The group (as ever) had a lively debate about the code structure they hit upon, some people saying they hadn’t simplified enough, others (Ellie!) saying that it looked a little ambiguous in places. Some people questioned the methodology (always). Eventually (via Jim) we hit on an apparently old question that is at the heart of the visual languages research: are you trying to come up with something to teach people how to go on and learn other programming languages? Or are you trying to create a new, complete, usable language?

After we finished with the paper we had a slight extension to the meeting. Katy had asked us to provide a student with some feedback on a paper prototype for a visual programming learning tool he (Chris) is designing. It was a lovely prototype, with some really nice touches. It followed on really well from the paper, and hopefully he got some useful feedback and ideas on what to do next. It will be great to see where he takes it.

Sadly no cake this week, but none of us were quite desperate enough to defrost the cakes in the freezer! Fortunately no one was harmed as a result of the lack of sugar (some slightly elderly mini-rolls were “tidied”). Next week we will have to remember biscuits…


*Pane, J.F., Myers, B. a. & Miller, L.B., 2002. Using HCI techniques to design a more usable programming system. Proceedings IEEE 2002 Symposia on Human Centric Computing Languages and Environments, (Hcc), pp.198–206. Available at: