It is that time of year. The ACM SIGCHI reviews are out, and the lucky recipients have one week and 5000 characters to use rebutting them, trying desperately to get their beautiful papers into CHI 2014 and earn themselves a trip to Toronto. In today’s lab meeting we spent a little time unpicking some of the reviews and how they might be responded to. This post is not about the detail of the papers submitted though (and all lab group names will be left out, just in case!!) – this is more about the underlying process, and how to best go about constructing a sensible, non-sweary rebuttal. (It may take a day or two to get to the point where this is possible.)

The key part of a rebuttal is understanding the reviews. Each paper gets three reviews, and a meta-review that is pulled together by the Associate Chair (“AC”). The meta-review is pretty key. This gives you some kind of insight into whether the AC is “on your side” and is leaning towards including your paper or not. The key is the rating they give you. If they have given you a rating that is lower than the average of the three main reviews, you’re going to have to work pretty hard to convince them to include it.

So, then you need to work out what the key points are that you need to address. The focus should be on clarifications of your position, not major reworking of your arguments. As Hyungyoung Song points out, agreeing to a major rewrite of your key arguments suggest that you agree that there are underlying problems and massive rewriting usually suggests that you may be better off submitting to a later conference. Instead you need to focus on areas where the reviewers have misunderstood, so that you can clarify and help. Although it may not feel like it right now, this may, in the long run, even help your paper.

Part of helping to identify the key points that will need clarifying may well be to understand the reviewer’s form for this conference. There are certain key items that a reviewer is asked to look for, and it may well help to actively sign-post those items. For example, the key contribution of the paper is a good thing to highlight. Two of our lab group are working as ACs this year, so they were able to cast some light on this for people in the group who haven’t done this yet.

The format of the rebuttal was also interesting. The advice from one lab member was to be very specific, and say where you will add sentences to clarify, but also where you will remove something to keep within the page limit. The people who make the decisions are trying to decide whether your rebuttals are strong enough to answer your critics, but also whether they are feasible. This kind of structure makes it much easier for them to see that it is possible, and given the workload on the ACs anything that makes things easier for them is likely to count in your favour.

The best advice by far was not to give up at this point. 5000 characters is worth it. Don’t even think about what else you could do with your paper until after the final rejections have come in. There’s still hope!

Good luck to all who are going through this at the moment…

Advertisements

Starting this week with the news updates, the HCT lab is part of one of four larger sub-groups within the Department of Informatics at Sussex. Until recently this group has been known as Interactive Systems, but Judith reported a proposal to change this name going forward to “Interactive Design and Applications for Society”, or IDeAS. This met with general approval from the lab group members, and we will wait to see if it is adopted by the larger group.

Ben reported that he has been made “President-elect” of the International Society for Artificial Intelligence in Education, with his term of presidency due to start in 2015. Much kudos and congratulations for that, and we’ll have to hope the power doesn’t go to his head.

The lab meeting this week was used as a reading group, with Judith leading the discussion on “Uncomfortable Interactions“, by Benford et al*. This paper discusses the role of uncomfortable interactions within HCI, making the point that traditionally HCI has sought to reduce the pain of interactions whilst in other areas (notably art, films and occasionally education) painful experiences can be extremely powerful. The authors identify different kinds of discomfort – physical yes, but also cultural or control. This is an idea that some of the members of the group working on the African Farmer Game are becoming more familiar with, as the game does tug on some emotional strings of the players, frequently leaving them in a less-than ideal situation. Without this outcome, the power of the gaming experience for the entire group of players would be much reduced.

The discussion covered a many points. The perception of what is discomforting varies wildly from individual to individual, whether via an interface or in a social situation – e.g. the group’s experience of working with children on the autistic spectrum with their needs for calm and predictability may cause something to be an extremely uncomfortable experience for them. This in turn leads to questions about the ethics of the situation, which were covered in the paper but obviously sparked many personal viewpoints within the group.  We also noted a lack of the educational examples – the authors appeared to be firmly embedded in the performance side of uncomfortable interactions, but as mentioned above there is a lot of scope for a controlled level of discomfort in learning situations. And finally, the group felt it would be good to get a clearer view of the type and sense of discomfort that was being designed, and how those aims have influenced the interface design decisions made.

Overall, the group agreed that this was a very well-written paper that began to explore some very interesting points.

*Benford, S., Greenhalgh, C., Giannachi, G., Walker, B., Marshall, J., & Rodden, T. (2012). Uncomfortable interactions. Proceedings of the 2012 ACM annual conference on Human Factors in Computing Systems – CHI  ’12, 2005. doi:10.1145/2207676.2208347

TERENCE was a 3 year project that aimed to produce an adaptive learning system to help children learn to comprehend what they read. The University of Sussex were involved in the evaluation of both the learning outcomes from the system and the usability of the system itself. NicolaYuill and Eric gave us an update on their findings, with particular emphasis for our group on the usability.

Due to the various constraints on the project, the main methods for evaluating the usability of the system were a series of pre-deployment expert-user focus groups, and after the pedagogical intervention a series of interviews with the users themselves (both students and teachers). The project had essentially 3 GUIs – one for the experts creating the content, one for the educators, and one for the learners. This evaluation was aimed squarely at the learner interface.

TERENCE was deployed across 4 hearing schools and 4 hearing impaired schools in the UK. There were 83 hearing students and 24 hearing impaired. The interviews with the hearing impaired students were carried out with the help of an interpreter. The system was also tested in Italian schools, but the outcome of those tests were analysed by a different team.

Eric said that several themes had come through in the discussions with both the teachers and the learners. There were a few technical difficulties with this ambitious project, resulting in a number of usability problems with logging in and glitches when the learner was reading. There were also some issues with the level of the content – the stories were too difficult for some of the students, and this lead to them getting bored and just trying to click their way through without actually comprehending the content. However, after the end of the study, Nicola said she had a number of phone calls from the schools involved asking how they could continue to use the project, suggesting that the usability issues were not enough to dissuade the learners from wanting to use the software.

The group as a whole were interested to hear about the use of the system logs, which really hadn’t been used particularly much in the analysis due to the content being poorly designed for this. As several of us within the group have written systems that include logging, we all recognised how hard it is to design a system that is able to help answer many unforeseen questions! Also some kinds of information may be difficult to log – e.g. if a login fails due to network error, that request may never have reached the server to be logged, or the time spent on a given task may be difficult to track if the person can open the task, walk away, make coffee, chat to three people, then come back 40 minutes later and complete it.

The TERENCE project is now beyond the 36 months for implementation, and it was really interesting to see how far they got in that time. Many thanks to both Eric and Nicola for coming along to share this with the group.

We have been a little bit late starting back with our lab meetings – term actually started back in September – but we’re back! Much has been happening over the summer. Some have left us, some joined us, and those that stayed have been busy, so we started back with a bit of a catch up.

Since the end of the last proper term in April (wow, that long?), four of the group have submitted their PhD theses. Pejman, Gareth and Tom are still waiting for viva, but Liz has successfully made it through hers with minor corrections. So big congratulations to Liz! Three have now moved on from Sussex. Liz has moved to North Wales, Gareth has a job in Shanghai, and Pejman is now with UOIT in Toronto. We’ve also heard that another HCT alumni, Chris Frauenberger, has just been successful in a grant application, so we look forward to hearing more about that from him.

Ben has had multiple holidays (although he only owned up to those in the pub later…) and has been working on a couple of proposals, papers, and is – along with Judith – going to be organising the next PPIG meeting in Brighton next year. Alison is continuing to work on her PhD remotely in Newcastle. Judith has been working on a new house, as well as keeping various students out of mischief (or into it, as required), writing various papers, and trying to clear the decks to have time to write a couple of grant proposals this autumn. Eric has been working with Kate on a new project called TRACE that they are going to be telling us about in a lab meeting later in term, as well as continuing his work with OlderView and doing a bit of work on the TERENCE project. Edgar has had a busy summer sorting out his scholarship application and working on a panel proposal with friends from Paris and Barcelona. He also attended VL/HCC in San Jose, where he “ate tacos like a crazy man”. Ellie has been working on a couple of papers and attended DiGRA in Atlanta, which coincided with Dragon*Con and crazy fantasy mash ups that slightly blew her mind.

Muppet stormtroopers

Over the summer there were two new posts of Lecturer in Interaction Design that fall at least partially into our group. One of those posts was successfully filled by Kate, who has spent the summer kicking off the TRACE project with Eric and others (more to come!), the Face 2 Face project (a collaboration with Rachel Thomson from Education and Social Work and co-investigators at the University of Brighton and the Open University) and polishing her plans for teaching this term. She’s got a lot going on, and it all sounds pretty awesome.

The other post has been taken up by a new-comer to the group, Dr Marianna Obrist. Marianna joins us from the Culture Lab at Newcastle University, and has had a very busy summer trying to wrap up all of her projects up there! She has been looking at user experience methods and theories from the ground up, taking things back to first principles and trying to really define user experience. We’re all very excited to have her here with us, and really looking forward to hearing more about what she has planned over the coming months.

At this week’s meeting we had Kate Howland telling us about what she’s been up to recently. Kate has been working as a lecturer in Informatics since successfully completing her PhD last year, and this was a look at the research work she has been doing.

She initially started out working on TERENCE project, which is a large EU project supporting reading comprehension in children with and without hearing impairments of age 7-11. The project is a web-based game-like tool using an adaptive learning system to match the difficulty level to the skill of the child. Kate was involved with Nicola Yuill in conducting an initial small scale evaluation with users and teachers. Sadly due to the timing of the study (mid-July, just as all the schools go on summer holiday), they were unable to find the numbers of child participants they had been hoping for, so a lot of the feedback they got was from the teachers. This threw up some interesting results in itself. The game is designed to be used by an individual child, and the adaptive learning system models that child and their progress. However, some teachers said that they would be far more likely to use it in a full-class situation, discussing the solution with the class as a whole before responding. This is somewhat counter to the design of the application, and a useful thing to find out.

Another project that Kate was involved with was the Expertise scoping study, alongside Caroline Bassett. This was looking at the concept of expertise in digital technologies, particularly as they intersect with cultures and communities. A series of interviews was carried out with members of a wide variety of different communities to build up a picture of the way different people view their ability with digital technologies and what they perceived as the barriers to them becoming “expert”.

Kate was involved with a local school, and rather than conducting interviews in isolation ran an activity with them to see them actually using their digital skills. She started with a taster session where they came to campus and used the green screen technology to create some short film clips. Then she ran a series of after-school events, leading the children through the process of making their own stop-motion animations. She showed us one of the resulting animations, which was really fun! She felt that this followed on from  her thesis work, and allowed her to examine the way that children constructed a narrative and could be supported in doing so by digital means. She found it generated a lot of interesting data, but needs a bit more time to analyse it properly.

The third project Kate has been involved in this year is the Space Invaders project with the Centre for Innovation and Research in Childhood and Youth. This has two parts: a competition for young people to submit short videos on their experiences of social media and online gaming (as opposed to adults discussing what they think happens with children in these situations), and a live debate as part of the Brighton Fringe where the top 10 videos will also be shown.

As well as all that, Kate is also part of a project that has secured funding and is due to start in September. Face 2 Face (too new to have a web page yet!) is a collaboration with the School of Education and Social Work, the University of Brighton, and the Open University. It will examine the ubiquitous nature of screens in our lives these days, and the effects that is having on children and families.

So combined with teaching, it’s been a spectacularly busy year so far for Kate! There’s some really interesting stuff to follow up on with the work she’s been doing, and we look forward to seeing what happens next.

At today’s lecture, we managed to cajole Eric into telling us what he’s been up to recently. Eric has been sadly missing for most of the recent lab meetings, as it clashes with a Computer drop-in session at Age UK that Eric has been helping out with. This, as it turns out, is all part of Eric’s master plan (no evil chuckle required!).

Eric has been heavily involved with the HCT group for a fairly long time now, learning a lot about participatory design practices amongst other things. Simultaneously (inevitably?) he has also been gaining an increasing awareness of the difficulties faced by the slightly older individual (he cites his first pair of reading glasses as quite a defining moment in this). So he started looking to combine these two interests.

He quickly found that although designing for older people has been reasonably well-explored in academia, it has mostly been done from an external observation point of view. Very little of the work that he has found (in industry or academia) really seeks to engage with this audience, and get their input into the designs. He feels there are a few different reasons for this, such as different language and so on, but also that most designers are younger and find it easier to talk and spend time with their peers rather than their elders. He also feels it is not as easy to get access to the older generation for their views – channels such as twitter, facebook, and other social media routes are not as well used by this particular audience (although clearly there are exceptions).

So, Eric has created a website specifically looking to gather feedback on a variety of technical issues. OlderView.com posts news and reviews about various apps and pieces of technology, and solicits feedback via surveys. Eric then reviews the responses, and posts a follow-up post. He’s trying to keep the posts coming at a reasonable rate – aiming for one a month at the moment – and is using his contact with older people who (by turning up to the drop in centre) have an interest in technology to drive up interest. At the moment, his main aim is to build this site into a useful place for people to visit, but in the longer term he is hoping to link this with industry. He has a couple of ideas for this, from potential access to older, tech-savvy people, or potentially as a way to source panels of more naive users for companies to talk to (clearly all with the participants’ approval, and sharing any income generated!). This is still in the very early stages.

Eric is also talking about trying to reach out to older people who currently don’t use technology at all. This is a hard audience to reach, but could have some really key insights into the barriers they feel exist for them. He has already identified two types of problem that he sees most regularly: the accessibility issue – accounting for the physical and cognitive deterioration that comes with advancing years, and the more difficult problem of technical solutions not matching the outlook of the older population. It is probably this second category of problems that could be most helped by a more participatory approach. Jim also asked if he’d thought of offering a service for re-writing instructions, and Eric agreed that a lot of the time it’s not that they don’t want to use the technology, but they are scared off by the initial setup.

A proposal has also recently been submitted to look more closely at some aspects of this work, so fingers crossed for that!

(Eric also tweets on these issues. Anyone interested can find him there!)

Today’s lab meeting was almost like a reading week! We had a catch up and planning session. Always useful mid-semester.

We did, however, have two exciting bits of news! Judith is co-author on a paper that has been accepted to Interaction Design and Children 2013, and Kate is co-investigator on a project that has secured funding from ESRC.

Kate is hopefully going to be telling us a little more about that and some of the other things she’s been getting up to next week, and hopefully Judith will be sharing the contents of the paper with us at some point too. We’ve also had a pretty poor showing on the edibles so far this term, so maybe next week will change all that? Well, we can hope.

Today the lab group were asked to be lab rats for the first-time-in-the-world-ever playing of a computer game! We also welcomed a couple of guests: Helen Pain from the University of Edinburgh and John Thompson from the Institute for Development Studies

African Farmer is an online, multiplayer game that is being written by Ellie and Jim for the Green Revolution project. It is loosely based on a couple of boardgames used in teaching scenarios for International Development and for policy makers. The game is ideally aimed at groups of between 15 and 25, with players being asked to take care of a family in a rural farming village in Africa.

The game revolves around a series of locations that each provide different information and access to functionality:-

  • Home – gives information about the health and wealth of the family. 
  • Farm – shows the state of the crops and fields that the family own.
  • Village – allows the players to get a feel for how other people in the village are getting on.
  • Market – allows players to buy and sell things that they need, such as crops, fertiliser, food, school vouchers etc.
  • Bank – for loans, and the payment of penalties.
Farm view

The farm view.

The idea is that people have incomplete information about the growing conditions, how much their crop will be worth and so on, and have to make some quite complex decisions. The outcome for the families can be harsh – if they do not secure enough food each year, family members will die. The simulation is a very simplistic view of farming and environment, but the decisions soon mount up and allow for a wide range of strategies to be employed.

This was the first time that the game had been run with multiple people logging in simultaneously, and the lab group were asked not to hold back with their feedback (although Ellie did say if they wanted to be nice about it they could be!). We had a lively session which completely overran our usual 12pm stop, and generated a lot of feedback on the interface. Sadly we didn’t get quite as far through the game as was hoped. Some of the interface difficulties slowed down people getting to grips with the game and being able to find the right information.

Still, as Jim put it this was a pre-alpha version, and the feedback was immensely useful for clearly indicating which areas need more work. The range of backgrounds within the group highlighted different problems – some very quickly got to grips with the communication panel and other elements of the interface, others grasped the flow of the game much more quickly. The game itself actually fared quite well – the group were absorbed and wanted to see what the outcomes of their actions were – so with a few changes to the interface the whole thing is looking extremely promising. Jim and Ellie have been left with a lot of ideas to think over.

Maybe after the next iteration the lab group will agree to play again!

Ostensibly this meeting was to discuss the paper that Ben had forwarded around the group on Grounded Theory. Actually this paper was more of an entry point for discussion. Nick Collins (a colleague from Music Informatics) has recently started some qualitative research, and was looking to discuss some of his reservations about the methods that he was having.

In a way, this made starting from Grounded Theory an interesting choice. Whilst many of the group members have conducted qualitative research, none of the group who were present have actually used Grounded Theory itself. So although we did spend a little time discussing the area the more interesting discussion came when we started talking more specifically about Nick’s current research interest and problem, and how the group’s experience could help.

Nick is doing a series of semi-structured interviews, the questions for which came from the results of a previous online study. Judith suggested that perhaps he should look at Thematic Analysis as an approach, but Katy has had some trouble finding a good reference for how to go about that.

Lesley uses an iterative approach to coding, starting potentially with very low level coding (examples used were “red flower” and “my mum said” – which did make sense in the context, honest!) and use those to then create higher level categories (e.g. “External objects”, “other people”) and recode. She says she tries not to have any preconceived ideas about what she’s looking for, but accepts that on some level she does.

Bias (which is introduced by the preconceptions) came up repeatedly. It seems that different studies have quite different approaches to this problem. Some work around it by having a large number of coders and looking for a high degree of inter-coder reliability. Others use two coders, who only confer when there is a disagreement. Still others use a single coder (and it was a real shame that Gareth wasn’t here at this point) who provides a single clear viewpoint. As always, the problem is finding the approach that is most appropriate for each particular study.

A related question was when to start looking at the data. Nick was very worried about looking back on it too soon, in case it changed the way he conducted later interviews. Lesley had examples where tweaking the questions on the way through had uncovered important information that would otherwise have been missed, although the change had to be documented when writing up the study. She also said that when you carry out your own interviews to code you can’t help but know what’s been said previously. Judith queried about the difference between knowing what had been said and reflecting on it, forming further ideas about it. The group consensus seemed to be that leaving it as long as possible before that happens would be preferable – which may explain why none of us have used Grounded Theory!

All in all it was a good discussion of the practices that the group have used when conducting research in the past. We hope Nick found it useful and not too overwhelming. We also hope that he might find time to come back and tell us about what he finds.

Given that a few of the group are attempting to write up at the moment, this seemed like the perfect time to set up our own little “Shut up and write!” session.

If anyone wants to join us, we’ll be in the Bridge cafe, 2pm on Friday 15th. All welcome (tables permitting), not just those trying to write their thesis. At the moment, Pejman and Ellie are confirmed (links for photo id purposes).