Development:Quiz UI redesign - prototype testing report

Back to Development:Quiz_UI_redesign

General observations

Those subjects, who reportedly had no experience making moodle quizzes, all successfully added random questions to the quiz, though there were problems and they probably would not have succeeded in this so quickly without having made to read the brief help file first.

After having read the text, one of them also needed me to explain the relationships of the concepts before they understood the concepts (todo: listen to the recording and try to solve what they misunderstood). But after this explaining even they did succeed adding a random question to the quiz.

This report does not include the preliminary testing done with the original prototype, which mainly gave promising results, but was not enough to really confirm much.

For privacy reasons, I am not publishing much details about the test persons.

Test subjects

All test subjects were Finnish, from different cities and 4 different educational institutions.

• Test subjects 1,2,3 (academic teachers): May 29th
• Test subjects 4,5 (it support): May 30th
• Test subject 6 (student): June 24th
• Test subjects 7 (student), 8 (academic non-teacher): June 25th

Issues

Issue

• Users affected: 5
• Issue: The user didn't realize what category he was in and used the default category though he should have added a question in the random question (category) he had just created.
• Attempt to solve 2008-06-18: Added a separate category line to the category window's title bar; moved the category description there, too. Changing the category is separate.

Issue

• Issue: The user didn't understand the meaning of the Magnifying Glass icon.
• Users affected: 1, and others (check from recordings?)
• Severity: 2; the user may never find out that single questions can be previewed, too, unless they happen to try.
• Solution: attach a "preview question" label or tooltip to the icon.
• Attempt to solve before usability tests of 2008-05-30: Added an "Esikatsele" (preview) label on the left side of the icon
• Canceled solution: Adding a label adds too much extra clutter; settling for tooltip in implementation phase. The user can use the exam preview

 * Not confirmed 2008-05-30; test subjects 4,5 were experienced quiz users and assumably knew the meaning of the preview icon

Issue

• Users affected: 4
• Related issues: 5
• Issue: The user mentioned that in a normal situation s/he would want a random question category to be a subcategory of another category; adding a random question by clicking "add random question" in the quiz does not enable this.
• Attempted solution 2008-06-19: When adding a new category with a random question, add an option to add it as a subcategory of another category.

Issue 4

• Issue n:o 4
• Users affected: 3
• Issue: There was a random question in the quiz. The category of the random question, which had one question in it, was open in the question bank contents window. The user explained during trying to carry out a task that s/he didn't know, whether or not the questions in the category were included in the random question (the category of which was the same one). Later s/he explained that now s/he understands, s/he needs to add these questions (from that same category) to the quiz from the category before they are in the quiz.
• Attempted solution 2008-06-19: List the names and the contents of at least some of the questions (which are in the category) in the randon question box, and below that text: Total of N questions in category [in addition, if there are questions of different question types in the category, express that here].

Issue 5

• Task: Add a random question from category
• Severity: 5; the application does not support the natural mental model of the user
• Issue: The user assumed the category already exists. S/he clicked "Lisää satunnaiskysymys" (Add random question) but the dialog didn't enable him/her to select a previously existing category.
• Attempt to solve before usability tests of 2008-05-30: Extended the functionality of the dialog to enable the user to add a random question even from an existing category directly into a quiz.
• Further attempt to fix
• Unconfirmed 2008-05-30: the user confused about what s/he was editing was not helped by the new dialog

Issue

• Task: preview question
• Users affected: 4
• Issue: The user mentioned that if this were for real, she could use the "View as student" (tsekkaa oikea nimi) mode for previewing the quiz. However, this option is not available in the quiz, and intervewees/test subjects have not been aware of it, describing that they have a separate student user ID for previewing different quiz options in "Update this quiz".
• Solution: Add "View as student" as an option somewhere close to the preview button; provide clear indication of this mode change and a clear route back to teacher mode -- before doing this, it needs to be confirmed from Moodle development community as it seems a higher-level decision

Issue

(Related to the small change, adding "add directly to the exam" checkbox, made to the "modify question" screens in the prototype)

• Users affected: 1-3?, 5
• Issue: the 1-3? user asked out loud "I wonder what 'Add question directly to exam' means" - I guess, since they assumed that there is nowhere else to add questions to. User 5 was confused about the (by default unchecked) option when s/he was adding a question to a category s/he had just made for a random question.
• Solution: Revise the checkbox text somehow or find out some other way to express and control whether the question being edited is:

- going to be added into the quiz or just in the question bank
- already in the quiz

Issue

• User: 2
• Issue: The user assumed the question content field is for adding "pictures, or something".
• Related issues: 2
• Solution: See issue 2

Issue

• Users affected: 5
• Issue: the user was unaware of whether the random question indeed contains all of the questions of the category, since s/he saw the option to add questions from the random question to the quiz
• Solution: Explicitly express this in the random question box; after the list added in the solution of issue n:o 4: "in addition, this random question has the rest of the N questions in the category Z as alternative questions."
• Attempted solution 2008-06-19: Random questions now list max. 2 of the category; if there are more questions in the category, a link to show more of them is shown.

Issue

• User: 2
• Issue: The user attempted to use the "Tallenna" (Save) button (that only saves the question points) for saving the changes she had (successfully) just made to a question, commenting along the lines of "you can never be quite sure enough".
• Solution: Change the button label to "Save points" or equivalent
• Attempt to solve before usability tests of 2008-05-30: Due to lack of horizontal space, just added a colored background, grouping the "Pisteet" (points) label, the text field, and the save button together, hopefully making it harder to assume the save button affects the whole question

 * preliminarily confirmed 2008-05-30; test subject 4 mentioned that she would normally add points for the question here and then click this button to save the change of points
* 2008-06-19: Also added the points label; this change may be canceled later on if previewing questions with just the small magnifying glass icon without label turns out to be too hard - the space is too small for both these extra labels.

Issue

• User: 2
• Issue: the user attempted to click the random question box's notify text's part '“kysymyspankin sisältö” >>' even though the “kysymyspankin sisältö” (Question bank contents) window was already open
• Severity: 1; the application did not respond
• Solution: Make the application respond unobtrusively
• Attempt to solve before usability tests of 2008-05-30: Removed the ">>" when question bank is already open

 * Not confirmed 2008-05-30; the test subjects 4,5 did not attempt to open the question bank first but while it was still closed clicked the link in question.

Issue

• User: 2
• Issue: The user didn't realize what on the screen was the exam until she stopped to think about what is on the screen / to study the screen contents more carefully
• Note: this happened fairly early during the test, it might have been caused by the test scenario and because the user may have been so involved in his/her task that s/he did had not stopped to actually study the whole of the UI before this point.

Issue

• Location: Help window/basic help of quiz
• Users affected: 3
• Issue: The help window was closable with just one click. The user didn't know the conventions about where a single click and where a double click is needed. A double click caused the underlying window to react and take the user somewhere the user didn't expect to be taken to. (This was partially an issue of the prototype, but could have similarly occured in a regular)
• Severity: 2; usually recoverable with the browser's "back" button, assuming that all clickable places that take irreversible actions are backed by a confirmation dialog.

Issue

• Location: Help window/basic help of quiz
• Users affected: 3
• Issue: The user misunderstood the help text
• Severity: 4; The user would likely find the correct conceptual model by trial and error and by after trial and error perhaps rereading the help text. Possibly the user would give up using the application if s/he would try to use categories or random questions.
• Solution: Evaluate, how the user understood the help text. Revise the text if needed.

Issue

• Severity 0-2; Not necessarily an issue.
• Users affected: 3
• Issue: The relationship between a random question and a question bank is not well understood. It seems(?) that the fact is hard to understand that a random question is itself a question and it contains other (other kinds of) questions. In task X,
• Solution: Perhaps name "random question" something else than a question, since it is kind of a metaquestion. However, this is problematic since old Moodle users already know the concept of a random question. In this and some earlier usability test, elder teachers have commented that random question or category should be named something entirely different than satunnaiskysymys (Finnish direct translation for 'random question"):

-"random question" -> "book category" -"kysymyskategoria" (question category) -> aihepiiri (topic/topical field) or "kokonaisuus" (a whole); lower hierarchy levels would be "alaotsakkeet" (subheadings) or "osa-alueet" (translation?)

Issue

• Users affected: 3
• Issue: the user didn't understand what "oletusarviointi kysymyksille" means.
• Further issue: I don't, either.

Issue

• Issue: User did not find how to add additional instructions
• Users affected: 3

to the quiz. S/he did not think "-- Select question type --" next to an "Add question" box was a good place to look for something to provide instructions to the student with. (labels translated)

• Note: The user eventually looked in the aforementioned menu and succeeded in adding what was called "Kuvaus/Ohjeteksti" (Description/Instructional text) - s/he wrote the instructional text in the question name field, though.
• Attempted solution (when?): added a separate button for this

Issue

• Task: Add additional instructions that are related to a random question (=to any and all of the questions in the category of that random question)
• Users affected: 2,3
• Issue: The user tried to add the instructions inside the random question or somehow make them actually related to the question

Issue

• Issue: the user did not realize s/he could reorder the questions with the UI
• Solution: make the icons more explicit by adding a label or something

Issue

• Users affected: 3
• Task: Add additional instructions that are related to a random question (=to any and all of the questions in the category of that random question)
• Issue: The user tried to make the instructions actually related to the random question by trying to select the category of the question to be the category of the random question.
• Comment: This problem may have been caused by the task itself; that is, the user was explicitly told to make instructions /related/ to the random question and that may have been the reason s/he did not accept it being put just next to the question.

Issues not related to the quiz creating UI

Issue nr1

• Users affected: 1,2,3,6,7 (=every one of the users with no previous experience of moodle)
• Related issues: 2, 3
• Issue: User assumed that the field with the label "Kysymyksen nimi" (Question's name) meant the question itself and wrote the question in that field, consequently assuming the question field itself is meant for something else (answer box for student or answer template/example answer)
• Severity: 4; unless the user previews the quiz they might end up having something in the finalquiz not intended there
• Notes: this issue did not affect subjects 4,5 who are working as moodle support staff. Subject 7 noted that s/he assumes the actual question text field is for additional, more specific details.
• Solution: rearrange the fields so that the question content itself is first; warn the user if the question content has been left empty (this may already be there); refine the labels of the fields.

Issue nr2

• Location: Modify essay question
• Users affected: 3
• Issue: Luuli kysymyskenttää vastauspohjaksi
• Severity: 5; seuraus että tentissä vastauspohja kysymyksenä
• Solution: Make the field label clearer, group the field closer together with the label

Issue nr3

(not directly related)

• Location: Modify essay question
• Users affected: 3
• Issue: The user didn't use the 'save' button. Perhaps the user didn't notice it or didn't understand its meaning.
• Solution: Add an "onunload" prompt to the page if it is exited by other means than submitting the form IF the page has changed (problem: this requires javascript)

Issue nr4

(Not directly related to Quiz creating UI work)

• Users affected: 3
• Location: edit essay question
• Issue: The user assumed that the question field is the same view that a student sees. The label of the field was far enough from the field itself that the user assumed it is unrelated
• Vakavuus: 3; seuraus epäselvä tai ei mikään
• Attempt to solve before usability tests of 2008-05-30: Moodlen standardiksi: Kirjoita joka paikkaan, onko jokin näkymä opettajan omaa näkymää vai näkyvätkö jotkut tiedot myös opiskelijalle vai onko näkymä opiskelijan näkymää simuloiva esikatselu.

 * Not confirmed 2008-05-30; test subjects 4,5 were experienced quiz users and were aware of the screen being solely for teachers.

Issue nr5

• Issue n:o 3
• Users affected: 4
• Issue: The user assumed that a description type question needs no name
• Confirm this is an issue?

Test subject comments

Comment

• User comment: user didn't realize the "yläpalkki" ("upper bar") had to do with the entire exam (and only the exam?)
• Solution: Add a heading (original quiz probably has this)

Comment

• User comment: user thought it might be a good idea to have a chance to force students to give feedback about any exam question in a separate field (next to the question?)
• Note: The user seemed to understand what s/he was supposed to do, according to what s/he explained. It was assumed in the test that such a task is realistic and relevant. This was not confirmed; the problems may well have arosen(?) from how the

task was written.

Comment

• Users affected: 4,5
• What does "muokkaa kysymyspankissa" (edit in question bank) edit?
• Solution: revise the link or add the rest of the category editing options to the window...?