Quiz UI redesign/usability testing of August 2008/Executive summary

Return to: Quiz UI redesign | Quiz Usability portal

For novices, adding single questions to a quiz is now a breeze in terms of the Quiz edit UI, although there are issues with question names and question forms.

What is important to note when reading the issues is that all minor errors of the users are reported. In usability tests, we expect a certain amount of issues that arise from different amounts of experience using computers and also from different circumstances of the moment. As the concepts are not trivial, it is not a catastrophe if a user cannot perform a task at first time – users do realize that using something for the first time takes some amount of learning.

I had one test subject on August 14^th, who had a complete design of an exam with him/her. This was a simple exam with <10 non-random questions of types essay, true/false and multiple choice. With the actual quiz editing form, s/he succeeded spectacularly: The problems that occurred were related to the names of the questions in Finnish (True/False was untranslated) and to the question forms.

Main findings:

The basic ideas help text of Quiz is helpful, if users find it. It is not easy enough to find as the default font for help is smaller than 1em. It needs an introduction to quiz pages.
The Question bank window is not prominent enough, and it is not obvious to a novice what its purpose is. To a degree, this is acceptable since it is advanced functionality and that is why it is hidden by default, but it needs to be easier to identify (its title bar is way too subtle now)
The Question bank window does not sufficiently respond to changes introduced from outside the window.
Further tweak the graphical appearance of the UI closer to the OOo prototype.
make it clear to users in the context of selecting a question type, what each question type means.

Creating random questions can be yet simplified (but is still an advanced operation)

When it comes to the editing UI itself, it works well otherwise but it is still too difficult for novices to create random questions. This is in part the fault of the somewhat complex conceptual system, but the UI could still make it more straightforward. The problem is, as the concepts are complex, making the UI simpler than the concepts would mislead users to false beliefs about the underlying conceptual system. A completely separate wizard/assistant might help alleviate this.

During the tests it seemed though that with the new UI, users do not need to understand the concepts and their relationships thoroughly to successfully create random questions. Still, creating random questions is painful and with many of the six subjects that went through the actual test (todo: verify how many), I had to help them by telling them which on-screen instructions to focus on, even though they had read the “basic instructions of creating quizzes” introductory text.

Nevertheless, all of the subjects eventually created one or two random questions with two essay questions in them successfully.

Issues that were solved from previous iteration

The following issues did not reappear in the final usability tests. Of course in theory it is impossible to prove they might sometime still appear. Issue numbers are from previous testing report.