This Tuesday Edgar presented an overview of his PhD research, mainly focussing on his current task derived from his last experiment carried out in Mexico. The aim of the project is to explore the usefulness of natural language (NL) descriptions for small algorithms when novices are learning the basic concepts of programming (input/output, assignment and loops).

Programming environments, such as Scratch or Alice, are based on graphical manipulation interface and error-free syntax, to support programmers. However, these environments still missing a way to “translate” or express programs in a language that is familiar to novice programmers, thus users cannot easily verify whether the current solution matches their intention. Edgar’s research explores the usefulness of providing a second representation to support a visual language (in this case a flowchart representation) designed for novice programmers. He aims to assess the benefits of a second representation when comprehension, creation and debugging programming skills are involved.

The main inspiration for exploring natural language as tool to support novice programmers comes from FLIP, a system that combines a visual language based on interlocking blocks with natural language support.

flip screenshot

Flip interface: (a) scripting editor top left, (b) natural language description bottom left, (c) toolbox of scripting instructions on the right side.

This research uses a tool called Origami, which is used to teach basic programming concepts to novices in Universidad Autonoma de Yucatan, Mexico. This tool enables students to create, test and debug programs using a visual language based on flowcharts. The programs produced are syntactically error-free, however, students may input instructions into each flowchart block that are semantically incorrect. Origami logs interaction data during the problem solving process: time taken to complete an exercise, the blocks used, the inner blocks’ instructions and any compilation messages. Each interaction is associated with a timestamp that preserves the chronology of the problem solving process.

The Origami tool was improved using some design ideas from Flip and now combines a visual language based on interlocking blocks with natural language

origami screenshot

Origami + Flip-ish = Origami + Natural Language Descriptions

Last August Edgar headed back to Mexico to carry out his experiment. The experiment considered a comparative study with three conditions: the earlier version of Origami (no secondary representation), a version with a secondary Natural Language description and a version with a secondary pseudo code representation. Some details about the experiment:

  • Participants: 75 first year students of a Computer Science 1st year  course (63 males and 12 females), aged between 17 and 21 years old. Three groups were formed and were taught the same material by two tutors
  • Materials: The materials used in the study were a programming pre-test and post-test, the Origami tool (3 versions), and 3 sets of algorithmic exercises. The pre-test and post-test and each set of algorithmic exercises consisted of 3 type of exercises (completion, debugging and tracing) each with three difficulty levels (simple, medium and complex). The format of the items for the pre- and post-test were based on multiple choice questions (debugging and tracing) and open questions (completion), and were administered via the online course delivery tool.
  • Method: The experiment was carried out over a period of approximately 6 weeks. In addition to the practical work there were a total of 12 sessions (24 hrs), in which the basic concepts of programming (variable, sentences, conditions, loops) were taught. The first week the pre-test was applied. The 3 sets of algorithmic exercises were assessed as coursework during the weeks 2 to 5. Each set of algorithmic exercises was divided into two subsets: (1) simpler exercises and (2) medium & complex exercises. Finally, the post-test was administered during the last week of the experiment.
Experimental design

Experiment: Three conditions, Pre and Post Test, and three sets of exercises.

Finally after some really hot weeks in the south of Mexico, Edgar came back to the UK with a bag full of Mexican sauces and lots of data. Currently he’s working through the sauces, trying to keep warm, and looking at the pre- and post-test analysis of the explanation exercises from the students, where the students were asked to explain the solution for a math-related problem in terms of an algorithm. He’s using more detailed rubric, and at the same time detecting errors, using an omission/commission classification based on a previous work from Judith, Katy and Keiron.* Lots of analysis to do!

The session was full of suggestions about different and interesting kinds of analysis. Ben suggested (and Judith strongly agreed) that the analysis of the pre- and post-tests should be done blind to which groups the individuals were in. Ellie asked if the basic Origami system has ever been compared with the paper version it replaced, which made Edgar realise that the people using the basic Origami were still asking for paper to scribble on, while the other two groups weren’t. There appears to be an early indication that the group who had the natural language representation were better at explaining their solutions in the post-test, which caused the group to suggest that maybe this was equipping the students with the language to talk about what they have done.

We are back in our normal Interact Lab next week, at the normal time. Hopefully we’ll get a full house, as we have a special guest appearance!

*Good, J., Howland, K., & Nicholson, K. (2010). Young People’s Descriptions of Computational Rules in Role-Playing Games: An Empirical Study. In 2010 IEEE Symposium on Visual Languages and Human-Centric Computing (pp. 67–74). IEEE.