TERENCE was a 3 year project that aimed to produce an adaptive learning system to help children learn to comprehend what they read. The University of Sussex were involved in the evaluation of both the learning outcomes from the system and the usability of the system itself. NicolaYuill and Eric gave us an update on their findings, with particular emphasis for our group on the usability.

Due to the various constraints on the project, the main methods for evaluating the usability of the system were a series of pre-deployment expert-user focus groups, and after the pedagogical intervention a series of interviews with the users themselves (both students and teachers). The project had essentially 3 GUIs – one for the experts creating the content, one for the educators, and one for the learners. This evaluation was aimed squarely at the learner interface.

TERENCE was deployed across 4 hearing schools and 4 hearing impaired schools in the UK. There were 83 hearing students and 24 hearing impaired. The interviews with the hearing impaired students were carried out with the help of an interpreter. The system was also tested in Italian schools, but the outcome of those tests were analysed by a different team.

Eric said that several themes had come through in the discussions with both the teachers and the learners. There were a few technical difficulties with this ambitious project, resulting in a number of usability problems with logging in and glitches when the learner was reading. There were also some issues with the level of the content – the stories were too difficult for some of the students, and this lead to them getting bored and just trying to click their way through without actually comprehending the content. However, after the end of the study, Nicola said she had a number of phone calls from the schools involved asking how they could continue to use the project, suggesting that the usability issues were not enough to dissuade the learners from wanting to use the software.

The group as a whole were interested to hear about the use of the system logs, which really hadn’t been used particularly much in the analysis due to the content being poorly designed for this. As several of us within the group have written systems that include logging, we all recognised how hard it is to design a system that is able to help answer many unforeseen questions! Also some kinds of information may be difficult to log – e.g. if a login fails due to network error, that request may never have reached the server to be logged, or the time spent on a given task may be difficult to track if the person can open the task, walk away, make coffee, chat to three people, then come back 40 minutes later and complete it.

The TERENCE project is now beyond the 36 months for implementation, and it was really interesting to see how far they got in that time. Many thanks to both Eric and Nicola for coming along to share this with the group.