Quiz UI redesign/usability testing of August 2008/Issues

Return to: Quiz UI redesign | Quiz Usability portal

This page should be kept in sync with the solutions listing page.

Confusion about where to add random questions/questions in random questions

These issues are particularly prone to misinterpretation. With many of them, it may be that the user just did not see the correct element onscreen, or it may be that they selected the wrong element because they did not understand the concepts related to random questions.

For complete novices that most of my test subjects were, creating random questions is still not as fluid as I wish it to be. It is an advanced operation and it seems a certain learning curve must be accepted.

The transition from just adding questions straight to quiz, to

• adding a container (called a question too, that is, “random question”),
• which is somehow linked to a category
• in a dimension called “the question bank”

does take actual learning, instead of users just doing what they have gotten used to. The issues are divided in several sections. There is a separate document for just The solutions for Quiz UI Redesign.

What is interesting is that with the OOo prototype there were less difficulties with creating random questions. I am not sure whether this is the reason, but there are more generous margins. Also the graphical distinction between a page and a question was more emphasized due to the more contrasted borders in the OOo prototype.

• Solution: Further tweak the graphical appearance of the UI closer to the OOo prototype.

• No. 1
• User: 1
• Task: 9
• Issue: The user tried to find random question in the -- Valitse kysymystyyppi – menu
• Solution: Add “Random question” to the end of Add question menu, under a separator? Having it double (both as a separate button and in the dropdown) might confuse users though?

• No. 2
• Task: 9
• Issue: Even after being told the location of the random question button, the user tried to find random question in the -- Valitse kysymystyyppi -- menu
• The user said that after selecting essay, s/he expects a page on which s/he can add all of the questions, from which to randomize, at once. So it seems it may be that this issue occurred simply because the user did not understand the conceptual system yet.
• Solution: See issue 1, above.

• No. 3
• User: 1, 4, 8
• Task: 9/10 (Random question task 2)
• Issue: The user added a single essay question to the exam
• Note: When on the page, the user looked at the category menu, from which s/he could have selected the random question category, but did not select anything
• Even if s/he would have, the question would still have also been added to the quiz as a single question
• Solution (not to the issue itself though): Indicate on the question editing page if the question is being added straight to quiz or just to category? A checkbox for this was included in the earlier OOo prototype but that confused users, too so the design of this must be carefully considered. Indicating this in the heading might work:
• “Adding question to”/”Editing question in”
• “Quiz [quizname]”/”Question Bank category [categoryname]”

• No. 4
• User: 1
• Task: 6
• Issue: The user tried to use "Essee" (Essay) question type directly in the quiz
• Solution: Since the Quiz did not already contain any random questions, I'm interpreting this as confusion on conceptual level, i.e. the user was guessing since in the face of the task that was the first thing s/he came to think of.
• Related: 3

• No. 5
• User: 4
• Task: create random question (1st one?)
• Issue: Before having created an actual random question, the user tried to create a random short answer matching question but returned after looking at the form for that for a while.
• Solution: make it clear to users in the context of selecting a question type, what each question type means. See ...

• No. 6
• User: 5
• Issue: While trying to add a question inside a random question, the user searched in the "add question" dropdown and tried to add a new random short answer matching question in the category of the random question.

• No. 7
• User: 1
• No. 8
• Task: 8
• Issue: The user tried to use "Satunnaistetut yhteensop. Lyhytvast."
• Related: 10,15

• No. 9
• User:3
• Task: addrandom1
• Issue: The user said "oisko satunnaiskysymys, kysymystyyppi on edelleen tämä sama [essee]" (“would it be random question, the question type is [supposed to be] still this same one [essay]”). This seems to imply that the user thought on some level that s/he needs to specify the specifics of the questions within the random question already when creating the random question.
• Solution: Somehow imply *in the UI* that random questions are containers for questions, (...representing a single question in the actual quiz).

• No. 10
• User:3
• Issue: The user (falsely) said s/he now understands how random questions work after the fact that the random question window had the current category of the question bank tool preselected. S/he expressed that s/he thought this would be the way to add questions to existing random questions.

Question bank / question adding controls visibility

• No. 11
• User:3
• Issue: The user did not notice that the question bank category changed when s/he selected the random question
• Solution: Make the toolbox react to changes more noticeably. Show a "loading" animation in the question bank window if it changes. This might be done perhaps even with an absolutely positioned div with a partially transparent animated GIF. The problem is to only show the loading animation when the category is changed, not on every page load.

• No. 12
• User: 3
• Issue: The user did not notice the question bank tool at all/assumed it is irrelevant.
• Solution: Make the question bank and its title more prominent at least when the window is open. Emphasize its borders and make the title seem more like an actual window title: invert background/foreground colours. This will need to be ensured across moodle themes.

• No. 13
• User: 5 , 6
• Issue: The user searched for "Kysymyspankin sisältö" (Question bank contents) but could not find it.
• Solution: See issue 12

• No. 14
• User:3
• Task: addrandom1
• Issue: The user seemed to assume that the different controls for adding questions are related to the same operation.
• Solution: clearly separate from the two other buttons the title, dropdown and possible button of creating single questions by grouping them together?

Conceptual confusion: categories/random questions

• No. 15
• User: 4
• Task: "14. Add instructions to the exam to tell the student that especially random question number 1 must be answered according to the writing norms of scientific texts." (in Finnish)
• Issue: The user added the instruction in the question text of _one_ of the questions in the random question category
• Note: this may have been caused by the task wording

• No. 16
• User: 5
• Issue: While trying to add questions to random question, the user made a new random question with a parent category of the previous random question
• Solution: Make the random question dialog select the top-level category by default, not the currently selected category of the question bank window.
• Add clearer path: how to add questions to the category of a random question. One solution would be to add controls directly in the random question, but that may be misleading since that would disable users from undestanding categories. Another option is to create a separate wizard for creating random questions: since it would be a separate process, it is true novices would learn the actual concepts even less this way, but then again, they would also not be mislead to believe the concepts work some other way than they actually do.
• Ask about this in ACM, CHI lists? Asking questions about open source UIs might help otherwise, too.
• Related: 17

• No. 17
• User: 6
• Task: 9/10 (Random question task 2)
• Issue: The user was unsure of what "yläkategoria" meant
• Comment: "En tiedä mikä tää on" ("I do not know what this is")
• Related: 18

• No. 18
• User: 5
• Issue: "en tajunnu tota puurakennetta" "tää ei niinku antanu heti [oikeata yläkategoriaa?]"
• Solution: Make the random question window show the parent category: make the category hierarchy clearer

• No. 19
• User: 4
• Issue: the user was puzzled by the "Ylös" (upwards) category in the random question parent category menu
• Solution: discuss which default categories are really needed, what are their purposes, and how to communicate those purposes to users.

• No. 20
• User: 5
• Issue: The user commented that "hämäävää että tuol on kategoria [osoittaa kysymyspankkityökalua] ja sitten allakin on lisää kysymys" ("[it's] misleading that there's a category [pointing at the question bank contents window] and down there there's add question, too [page controls for adding directly on page]")
• Solution: Make the distinction clearer that the page operations are for adding things on the page, not inside random questions ? It may be though that this error was due to conceptual confusion and as such, this solution would not help.

• No. 21
• User: 6
• Task: 9/10 (Random question task 2)
• Issue: The user tried to move the YUI dialog window in order to make sure what the category was s/he was trying to add a random question into (that is, the user thought s/he needs to add a random question in the existing random question category in order to get questions in there.)
• Solution: Make the YUI dialogs movable. I remember there was an option for this.

Make help easy to find: Basic ideas of quiz editing

• No. 22
• User: 5
• Issue: The user added the instructions about writing norms to one of the random questions' questions text.

• No. 23
• User: 6
• Issue When trying to read basic ideas, the user went to "tentit" (quizzes) page
• (-> Since finding the instructions was not a central task to the test itself, I directed him/her to quiz edit page, where s/he found the instructions)

• No. 24
• User: 6
• Comment: The user noted that s/he is not sure of the meaning of "vain" (just) in "laita vain haluamasi" in the basic ideas (vain=only/just; “just put the questions” vs. “put only the questions, which)

• No. 25
• User:6
• Task: add random question related note
• Comment: The user wanted to add a note straight to the random question (searched for such option in the question bank window)
• Note: this might have been a bias in the task, normally the user might have realized s/he can just add a "description" before the question on the same page.

• No. 26
• User:6
• Comment: While reading the basic ideas of quiz, the user commented it could be an option to delete the question permanently also from the question bank, using the control in quiz.

Others

• No. 27
• User: 6
• Task: 9/10 (Random question task 2)
• Issue: The user was unsure of whether the random question actually randomizes between the questions, since single question preview for random questions was broken
• Solution: fix preview for random questions; this may be related to the quiz having empty questions
• Note: Later the user noticed the die icon of the random question and concluded that it must randomize between the questions after all

• No. 28
• User: 2 (test in English; native English speaker)
• Task: 6
• Issue: The user tried adding a random question from a pre-existing category. Did not understand how to add more questions. After saying s/he is stuck several times, the user noticed and tried (seemingly in desperation) the Add Random Question button. Added the random question successfully, and after reading the instructions inside the random question box, successfully added a question to the random question, after which s/he was confused again about the current state, wondering how/where to add further questions. The user edited the first question s/he had created in the category, tried saving it as a new (identical one).

• No. 29 (not directly related to random questions)
• User: 2 (test in English; native English speaker)
• Spontaneous task: "try making them [the two questions in the random question's category] not identical"
• Issue: The user edited the latter of the identical questions. Saved it as a new question. Realized that s/he now has two identical questions and a non-identical one. Went to search for a delete button on the question edit page. Finally saved the question in order to return to quiz editing, and said something like "let's see if there's a delete button here". Selected the latter identical question, found the delete button, deleted the question.

• No. 30
• User: 5
• Task: random question 2
• Issue: the user seemed to wonder about adding a preexisting category, this was due to misunderstanding the task to be related to task random question 1

• No. 31
• User: 6
• Task: 6/7 (Random question task 1)
• Issue: After trying to add an essay question and a multiple choice question, the user gave up the task.
• -> moved on to next task to read basic ideas

• No. 32
• User: 6
• Comment: The user noted that s/he is not sure of the meaning of "vain" (just) in "laita vain haluamasi" in the basic ideas

Reorder/paging issues

Nothing very consistently wrong here: users had unique issues.

• No. 33
• User:6
• Issue: After being told to go to the reorder tab the user returned to the edit tab when being told to add a new page since s/he had already learned to create new pages there

• No. 35
• User: 6
• Issue: The user did not understand the concept of quiz pages, but wondered if the paging was for making the quiz printable
• Note: The user also commented that s/he was searching for a general help where s/he could look up such a concept of a page
• Solution: Document the paging in “Basic ideas” help dialog

• No. 36
• User: 5
• Issue: The color of the reorder text box was misleading, the user did not identify as text box.
• Solution: change colouring, make a select dropdown?

• No. 37
• User: 5
• Issue: did not select page, clicked "lisää sivu valittujen kysymysten jälkeen". eventually realized that s/he has to select, selected only the last one (as was right)

• No. 38
• User: 5
• Issue: The user did not realize pages are pages.
• Solution: Make pages look more like physical pages? But they are not.

• No. 39
• User: 5
• Task: move all questions but first one away from page 1
• Issue: The user thought s/he would move the first question from the question bank to the quiz. S/he commented later during the task that it did not seem to make sense, after all.

• No. 40
• User: 5
• Issue: The user commented that it would be easier if s/he could move the questions like other elements are moved in moodle course front page
• Solution: Do it, this is important for Moodle's internal consistency

• No. 41
• User: 5
• Issue: The user assumed first that the arrows should move questions between pages only. When s/he clicked the down arrow of a not-last-question-on-the-page and it did not move off the page, s/he commented that the arrows must only be for organizing between questions and not between pages. Eventually, s/he did try to move the last question of the page down and realized that it work.

• No. 42
• User: 5
• Issue: The user sought after help for paging in the "basic ideas of making quizzes"
• Solution: A page in Quiz is one of the basic concepts, so add a short explanation there about pages. In addition, add a more thorough explanation next to the paging button, and perhaps in the reordering tab.

• No. 43
• User: 4
• Issue: on the reorder tab, the user said the entire page would me moved from the up/down arrow icons, since there was only one question per page. S/he went to the edit page to use the same arrow icons.

• No. 44
• User: 4
• Issue: the user did not independently find the reordering tab. Not sure this is an issue since the functionality is advanced.

• No. 45
• User: 4
• Issue: the user did not at first find the arrows to reorder questions.

• No. 46
• User: 4
• Issue: When there were only two pages in quiz, the user said questions on the other page were in another quiz. (Seems s/he had not seen an actual moodle quiz or preview.)

• No. 47
• User: 4
• Issue: After the user saw the "add new pages after selected questions" button s/he selected all the questions, thinking that a page will be added only after the last of them.
• Solution: change the wording. this issue may only be there in finnish.

• No. 48
• User:3
• Task: create page and move all but 1 away from page 1
• Issue: The user commented s/he doesn't know whether the arrors signify moving across pages or just between questions

• No. 49
• User:3
• Issue: The user was searching for "Muokkaa" (edit) tab but saw that s/he was on it already so did not seach further to find the paging/reorder tab.
• Solution: Make reordering a subtab of edit?

• No. 50
• User: 2 (test in English; native English speaker)
• Task: 11
• Issue (test issue): The user created a new page after page 1, but did not understand the task, tried to create a new question on the new empty page 2. After being asked to reread the question, searched for something to move the question. Said that s/he could delete the question (being asked where, s/he pointed at the in the question bank's delete button s/he'd found in the previous task) but did not want to lose his/her question. S/He never found the actual moving or deleting controls.

• No. 51
• User: 1
• Task: 10
• The user clicked the "Lisää uusi sivu valittujen kysymysten jälkeen" button without selecting any questions
• Solution: Either add a notification that you have to select some questions, or add a new page anyway at the end?

• No. 52
• User: 1
• Task: 10
• Situation: Järjestä ja sivuta tab, all 4 questions on page 1, no other pages.
• The user selected all questions except the first one and clicked "järjestä kysymykset", causing no reaction by the application (except reordering questions. s/he said s/he expects the, to be moved to the page 2.
• Solution: If user tries to move questions from a page to the same page, do not alter the order. Separate the functionalities of reordering and moving on different pages.