If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Quiz UI redesign - prototype testing report

From MoodleDocs

Back to Quiz_UI_redesign

General observations

Those subjects, who reportedly had no experience making moodle quizzes, all successfully added random questions to the quiz, though there were problems and they probably would not have succeeded in this so quickly without having made to read the brief help file first.

After having read the text, one of them also needed me to explain the relationships of the concepts before they understood the concepts (todo: listen to the recording and try to solve what they misunderstood). But after this explaining even they did succeed adding a random question to the quiz.

This report does not include the preliminary testing done with the original prototype, which mainly gave promising results, but was not enough to really confirm much.

For privacy reasons, I am not publishing much details about the test persons.

Test subjects

All test subjects were Finnish, from different cities and 4 different educational institutions.

  • Test subjects 1,2,3 (academic teachers): May 29th
  • Test subjects 4,5 (it support): May 30th
  • Test subject 6 (student): June 24th
  • Test subjects 7 (student), 8 (academic non-teacher): June 25th


Issue 1

  • Users affected: 5
  • Issue: the user was unaware of whether the random question indeed contains all of the questions of the category, since s/he saw the option to add questions from the random question to the quiz
  • Solution: Explicitly express this in the random question box; after the list added in the solution of issue n:o 4: "in addition, this random question has the rest of the N questions in the category Z as alternative questions."
  • Attempted solution 2008-06-19: Random questions now list max. 2 of the category; if there are more questions in the category, a link to show more of them is shown.
    • Later tests: This seems to have helped.

Issue 2

  • Users affected: 5
  • Issue: The user didn't realize what category he was in and used the default category though he should have added a question in the random question (category) he had just created.
  • Attempt to solve 2008-06-18: Added a separate category line to the category window's title bar; moved the category description there, too. Changing the category is separate.

Issue 3

  • Issue: User did not find how to add additional instructions
  • Users affected: 3

to the quiz. S/he did not think "-- Select question type --" next to an "Add question" box was a good place to look for something to provide instructions to the student with. (labels translated)

  • Note: The user eventually looked in the aforementioned menu and succeeded in adding what was called "Kuvaus/Ohjeteksti" (Description/Instructional text) - s/he wrote the instructional text in the question name field, though.
  • Attempted solution (when?): added a separate button for this

Issue 4

  • Issue n:o 4
  • Users affected: 3
  • Issue: There was a random question in the quiz. The category of the random question, which had one question in it, was open in the question bank contents window. The user explained during trying to carry out a task that s/he didn't know, whether or not the questions in the category were included in the random question (the category of which was the same one). Later s/he explained that now s/he understands, s/he needs to add these questions (from that same category) to the quiz from the category before they are in the quiz.
  • Attempted solution 2008-06-19: List the names and the contents of at least some of the questions (which are in the category) in the randon question box, and below that text: Total of N questions in category [in addition, if there are questions of different question types in the category, express that here].

Issue 5

  • Task: Add a random question from category
  • Severity: 5; the application does not support the natural mental model of the user
  • Issue: The user assumed the category already exists. S/he clicked "Lisää satunnaiskysymys" (Add random question) but the dialog didn't enable him/her to select a previously existing category.
  • Attempt to solve before usability tests of 2008-05-30: Extended the functionality of the dialog to enable the user to add a random question even from an existing category directly into a quiz.
  • Further attempt to fix
  • Unconfirmed 2008-05-30: the user confused about what s/he was editing was not helped by the new dialog
  • Fix canceled: Overcomplicating the dialog will confuse novice users. Expert users need to find adding random questions from an existing category in the question bank window.

Issue 6

  • Related issues: 7, 8, nr2, nr4
  • Users affected: 7, 8
  • The user did not find preview, but instead tried to first read the basic instructions and then tried to edit the question; assumed that the editing screen *is* the preview. User 8 stated that he thinks the student is supposed to write their answer to the field where the question is, below the question.
  • Solution: It appears that the preview icon indeed needs a label, after all
  • Solution 2: Make it clear in the edit screen that the screen is about *teachers* *modifying* questions. Are the main headers of the page clear about this?

Issue 7

  • Task: preview question
  • Users affected: 4
  • Issue: The user mentioned that if this were for real, she could use the "View as student" (tsekkaa oikea nimi) mode for previewing the quiz. However, this option is not available in the quiz, and intervewees/test subjects have not been aware of it, describing that they have a separate student user ID for previewing different quiz options in "Update this quiz".
  • Solution: Add "View as student" as an option somewhere close to the preview button; provide clear indication of this mode change and a clear route back to teacher mode -- before doing this, it needs to be confirmed from Moodle development community as it seems a higher-level decision

Issue 8

  • Issue: The user didn't understand the meaning of the Magnifying Glass icon.
  • Users affected: 1, and others (check from recordings?)
  • Severity: 2; the user may never find out that single questions can be previewed, too, unless they happen to try.
  • Solution: attach a "preview question" label or tooltip to the icon.
  • Attempt to solve before usability tests of 2008-05-30: Added an "Esikatsele" (preview) label on the left side of the icon
  • Canceled solution: Adding a label adds too much extra clutter; settling for tooltip in implementation phase. The user can use the exam preview
    • Not confirmed 2008-05-30; test subjects 4,5 were experienced quiz users and assumably knew the meaning of the preview icon
  • Comment from Tim: The preview icon already has a tooltip - as do all the icons.
  • Comment from Olli: In the final implementation the preview icon has a label everywhere except the question bank.

Issue 9

(Related to the small change, adding "add directly to the exam" checkbox, made to the "modify question" screens in the prototype)

  • Users affected: 1-3?, 5
  • Issue: the 1-3? user asked out loud "I wonder what 'Add question directly to exam' means" - I guess, since they assumed that there is nowhere else to add questions to. User 5 was confused about the (by default unchecked) option when s/he was adding a question to a category s/he had just made for a random question.
  • Solution: Revise the checkbox text somehow or find out some other way to express and control whether the question being edited is:
    • going to be added into the quiz or just in the question bank
    • already in the quiz

Issue 10

  • User: 2
  • Issue: The user assumed the question content field is for adding "pictures, or something".
  • Related issues: 2
  • Solution: See issue 2

Issue 11

  • User: 2
  • Issue: The user attempted to use the "Tallenna" (Save) button (that only saves the question points) for saving the changes she had (successfully) just made to a question, commenting along the lines of "you can never be quite sure enough".
  • Solution: Change the button label to "Save points" or equivalent
  • Attempt to solve before usability tests of 2008-05-30: Due to lack of horizontal space, just added a colored background, grouping the "Pisteet" (points) label, the text field, and the save button together, hopefully making it harder to assume the save button affects the whole question
    • preliminarily confirmed 2008-05-30; test subject 4 mentioned that she would normally add points for the question here and then click this button to save the change of points
  • 2008-06-19: Also added the points label; this change may be canceled later on if previewing questions with just the small magnifying glass icon without label turns out to be too hard - the space is too small for both these extra labels.

Issue 12

  • User: 2
  • Issue: the user attempted to click the random question box's notify text's part '“kysymyspankin sisältö” >>' even though the “kysymyspankin sisältö” (Question bank contents) window was already open
  • Severity: 1; the application did not respond
  • Solution: Make the application respond unobtrusively
  • Attempt to solve before usability tests of 2008-05-30: Removed the ">>" when question bank is already open
    • Not confirmed 2008-05-30; the test subjects 4,5 did not attempt to open the question bank first but while it was still closed clicked the link in question.

Issue 13

  • User: 2
  • Issue: The user didn't realize what on the screen was the exam until she stopped to think about what is on the screen / to study the screen contents more carefully
  • Note: this happened fairly early during the test, it might have been caused by the test scenario and because the user may have been so involved in his/her task that s/he did had not stopped to actually study the whole of the UI before this point.

Issue 14

  • Location: Help window/basic help of quiz
  • Users affected: 3
  • Issue: The help window was closable with just one click. The user didn't know the conventions about where a single click and where a double click is needed. A double click caused the underlying window to react and take the user somewhere the user didn't expect to be taken to. (This was partially an issue of the prototype, but could have similarly occured in a regular)
  • Severity: 2; usually recoverable with the browser's "back" button, assuming that all clickable places that take irreversible actions are backed by a confirmation dialog.

Issue 15

  • Location: Help window/basic help of quiz
  • Users affected: 3
  • Issue: The user misunderstood the help text
  • Severity: 4; The user would likely find the correct conceptual model by trial and error and by after trial and error perhaps rereading the help text. Possibly the user would give up using the application if s/he would try to use categories or random questions.
  • Solution: Evaluate, how the user understood the help text. Revise the text if needed.

Issue 16

  • Users affected: 4
  • Related issues: 5
  • Issue: The user mentioned that in a normal situation s/he would want a random question category to be a subcategory of another category; adding a random question by clicking "add random question" in the quiz does not enable this.
  • Attempted solution 2008-06-19: When adding a new category with a random question, add an option to add it as a subcategory of another category.

Issue 17

  • Severity 0-2; Not necessarily an issue.
  • Users affected: 3
  • Issue: The relationship between a random question and a question bank is not well understood. It seems(?) that the fact is hard to understand that a random question is itself a question and it contains other (other kinds of) questions. In task X,
  • Solution: Perhaps name "random question" something else than a question, since it is kind of a metaquestion. However, this is problematic since old Moodle users already know the concept of a random question. In this and some earlier usability test, elder teachers have commented that random question or category should be named something entirely different than satunnaiskysymys (Finnish direct translation for 'random question"):
  • "random question" -> "book category"
  • "kysymyskategoria" ("question category") -> aihepiiri ("topic"/"topical field") or "kokonaisuus" ("a whole"); lower hierarchy levels would be "alaotsakkeet" ("subheadings") or "osa-alueet" (translation?)

Issue 18

  • Users affected: 3
  • Issue: the user didn't understand what "oletusarviointi kysymyksille" means.
  • Further issue: I don't, either.

Issue 19

  • Task: Add additional instructions that are related to a random question (=to any and all of the questions in the category of that random question)
  • Users affected: 2,3
  • Issue: The user tried to add the instructions inside the random question or somehow make them actually related to the question

Issue 20

  • Issue: the user did not realize s/he could reorder the questions with the UI
  • Solution: make the icons more explicit by adding a label?
  • Severity: 2

Issue 21

  • Users affected: 3
  • Task: Add additional instructions that are related to a random question (=to any and all of the questions in the category of that random question)
  • Issue: The user tried to make the instructions actually related to the random question by trying to select the category of the question to be the category of the random question.
  • Comment: This problem may have been caused by the task itself; that is, the user was explicitly told to make instructions /related/ to the random question and that may have been the reason s/he did not accept it being put just next to the question.

Issue 22

  • Users affected: 6
  • The user did not realize that the questions shown in the random question were the same ones than in the category
  • Severity: 3

Issue 23

  • Task: 2008-06-24/5
  • Issue: The user tried creating a category instead of directly a random question
  • Users affected: 6
  • Severity: 2; the user chose a workflow that implies more work for him/her
  • Attempt solution before 2008-06-25: Make the workflow more straightforward by adding a link from the question bank window to the question bank. (An option is to add a button to create a new category; that would ease the workflow but this is actually a question bank operation and it is important that the question bank itself is also easily discoverable on its own.)

Issue 24

  • Task: 1
  • Users affected: 6, 7, 8
  • The user did not at first understand the relationship of the select question type and add question button
  • Severity: 1
  • Attempt solution before 2008-06-25: Added a hilight to group the elements together
  • Result: user 7 still tried to first use the create new question -button, but recovered quickly. in the final UI there will be feedback which will allow even quicker recovery from this.

Issue 25

  • Task: 2008-06-24/6
  • Users affected: 6
  • The user attempted to add a new random question when s/he was supposed to add a new question into the random question (category)
  • Severity: 4; the user either did not understand the concept of a random question after reading the instructions, or misunderstood the task

Issue 26

  • Task: 2008-06-24/6
  • Users affected: 6
  • The user attempted to add several questions into the same essay question's field
  • Severity: 2; It should be easy to recover from this error although the prototype was too limited to show whether or not this would have happened

Issue 28

  • Users affected: 6
  • The user thought the arrows left to the Jump to... -menu was the browser Back button
  • Note: this may have been caused by the fact that the prototype test did not feature browser controls

Issue 29

  • Users affected: 6
  • The user wondered why she did not see all of the three questions she had created (only two are displayed and there is "show all" text below them).
  • Severity: 1

Issue 30

  • Users affected: 7
  • After reading the instructions, the user thought they need to create a category first, but then tried the "create a random question" button and succeeded creating the random question and questions in the category.

Issue 31

  • Users affected: 7
  • In the create new random question dialog, the user wondered about which category to create a random question in
  • Solution: add "If you want to create your new category as a subcategory of another category, select the parent category here:"

Issue 32

  • Users affected: 6,8
  • The user did not seem to understand the concept of a random question first. Nevertheless, they eventually created it and added three alternative questions into it without help.

Issues not related to the quiz creating UI

Issue nr1

  • Users affected: 1,2,3,6,7 (=every one of the users with no previous experience of moodle)
  • Related issues: 2, 3
  • Issue: User assumed that the field with the label "Kysymyksen nimi" (Question's name) meant the question itself and wrote the question in that field, consequently assuming the question field itself is meant for something else (answer box for student or answer template/example answer)
  • Severity: 4; unless the user previews the quiz they might end up having something in the finalquiz not intended there
  • Notes: this issue did not affect subjects 4,5 who are working as moodle support staff. Subject 7 noted that s/he assumes the actual question text field is for additional, more specific details.
  • Solution: rearrange the fields so that the question content itself is first; warn the user if the question content has been left empty (this may already be there); refine the labels of the fields.

Issue nr2

  • Location: Modify essay question
  • Users affected: 3
  • Issue: Luuli kysymyskenttää vastauspohjaksi
  • Severity: 5; seuraus että tentissä vastauspohja kysymyksenä
  • Solution: Make the field label clearer, group the field closer together with the label

Issue nr3

(not directly related)

  • Location: Modify essay question
  • Users affected: 3
  • Issue: The user didn't use the 'save' button. Perhaps the user didn't notice it or didn't understand its meaning.
  • Solution: Add an "onunload" prompt to the page if it is exited by other means than submitting the form IF the page has changed (problem: this requires javascript)

Issue nr4

(Not directly related to Quiz creating UI work)

  • Users affected: 3
  • Severity: 4
  • Location: edit essay question
  • Issue: The user assumed that the question field is the same view that a student sees. The label of the field was far enough from the field itself that the user assumed it is unrelated
  • Vakavuus: 3; seuraus epäselvä tai ei mikään
  • Attempt to solve before usability tests of 2008-05-30: Make it a Moodle standard: every page should feature a mention of whether it is teacher's or student's view or are some information also visible to students (Moodlen standardiksi: Kirjoita joka paikkaan, onko jokin näkymä opettajan omaa näkymää vai näkyvätkö jotkut tiedot myös opiskelijalle vai onko näkymä opiskelijan näkymää simuloiva esikatselu.)
    • Not confirmed 2008-05-30; test subjects 4,5 were experienced quiz users and were beforehand aware of the screen being solely for teachers.
    • Not confirmed with test subjects 6-8: the test subjects did not seem to notice the text and still had problems understanding whom which screen was for

Issue nr5

  • Issue n:o 3
  • Users affected: 4
  • Issue: The user assumed that a description type question needs no name
  • Confirm this is an issue?

Issue nr6

  • Users affected: 6
  • The user reported repeatedly being lost. Did not understand breadcrumb.
  • Solution: Add actual course name to breadcrumb?
  • Severity: 4
  • Note: may have been caused by the strange course/quiz names in the earlier prototypes

Issue nr7 (formerly 27)

  • Task: 2008-06-24/6
  • Users affected: 6
  • The user attempted to click the edit tab after opening a modify question screen in order to be capable of editing it; s/he did not understand the tab was already active
  • Severity: 4; the user either did not understand the concept of a random question after reading the instructions, or misunderstood the task
  • Solution: further hilight the tab or the line that communicates the inactivity of the other tabs?

Test subject comments


  • User comment: user didn't realize the "yläpalkki" ("upper bar") had to do with the entire exam (and only the exam?)
  • Solution: Add a heading (original quiz probably has this)


  • User comment: user thought it might be a good idea to have a chance to force students to give feedback about any exam question in a separate field (next to the question?)
  • Note: The user seemed to understand what s/he was supposed to do, according to what s/he explained. It was assumed in the test that such a task is realistic and relevant. This was not confirmed; the problems may well have arosen(?) from how the

task was written.


  • Users: 4,5
  • What does "muokkaa kysymyspankissa" (edit in question bank) edit?
  • Solution: revise the link or add the rest of the category editing options to the window...?


  • Users: 6
  • The user suggested tooltip for buttons