Usability testing in August 2008/Issues found

Simplify question forms (progressive disclosure)

The question editing screens are in desperate need for progressive disclosure, a.k.a. Not Slapping It All On The User's Face Unless Needed.

• No. 1
• User: 4
• Task: 1
• Issue: The user wondered out loud whether or not s/he can fill in many questions at once since there are so many "laatikkoja" (boxes) in the essay editing form. Like many other users, s/he did not notice the labels for the htmlareas before being so lost that s/he had to actually search for clues about the meanings of the "boxes".

• No. 2
• User:3
• Task: 1
• Issue: The user did not find the save button. The user declared that s/he is stuck after filling out the essay question creating form.
• User: 7 (custom exam, no predefined test tasks)
• Issue: The user though “sotke vastaukset” is unnecessary and did nothing though the option was on by default

• No. 3
• User: 7 (custom exam, no predefined test tasks)
• Location: Edit multiple choice question
• Issue: The user was unsure of what “salli monta vastausta”/”vain yksi vastaus” meant

• No. 4
• User: 7 (custom exam, no predefined test tasks)
• Issue: User wanted more choices to his/her multiple choice, did not understand the untranslated “blanks for 3 more choices”. S/he assumed this meant something else even though s/he knew English
• Solution: Change to “Add 3 more choices to this question” or equivalent (and translate to Finnish, too)

• No. 5
• User: 2 (test in English; native English speaker)
• Task: 2 (??)
• Location: In Multiple Choice creating form
• Issue: the user didn't set a maximum grade to any question
• the user did not at first notice the error message since it was not at the top of the page, was at first searching for clues on what might be wrong.

• No. 6
• User: 6
• Comment: The user commented that the essay form was like "shooting a fly with a cannon"
• Comment: The user commented that the essay could have the save button also higher up the form

• No. 7
• User: 6
• Issue: The user commented "maybe some of the fields in the essay question editing form are to be filled in by the students"
• Related: name/question text, htmlarea labels

• No. 8
• User: 7 (custom exam, no predefined test tasks)
• Issue: The user did not want students to be capable of finding out the correct answers in the quiz. Even though selections for when to show results were unchecked in “update this quiz”, the true/false and multiple choice revealed theircorrect answers as soon as the teacher clicked “palauta” (return) in the quiz.
• Solution: Make all options that directly affect quiz behaviour and display (=quiz modes) have their status easily visible without having to go out of the quiz edit view.

• No. 9
• User: 7 (custom exam, no predefined test tasks)
• Issue: user tried to use multiple choice, and put the multiple choice's answers in the feedback field, since the feedback field had no readily visible label and putting it in the field below the “answer” field would have been consistent with the just-learned fact that with question text, the actual question goes to the “big box” under “question name”.
• Solution: make the fields in the forms consistent and lay out the labels obviously.
• Related to: ...

• No. 10
• User: 5
• Comment: about general feedback box: "tämä loota on arvoitus" ("this box is a mystery")
• Solution: Move the HTMLarea label so it groups clearly together with the HTMLarea

Sort out the different preview modes

Also found in previous testing, users are confused about the different options for previewing/seeing the student's perspective.

• No. 11
• User: 4
• Issue: the user was previously familiar with switching to student role so sought after that but did not find it in the quiz UI. Settled for preview in the end, which did not work.

• No. 12
• User: 1
• Task: 4. See how the question you just created looks like to a student in the quiz.
• Issue: Went to course front page with breadcrumb, switched to student role, tried the quiz successfully. Consequence: the quiz was locked down.
• Severity: 5 (fatal operation, no warning, no undo)
• Solution: Find out what are the use cases here. It seems here that teachers use this role for undo. In this case nothing the teacher does in “student role” should have any permanent effect, just like in Quiz preview. But it may be possible though that teachers use the student role for actually making changes “as students”, which they somehow could not make as teachers. In this case, it must be allowed to make permanent changes in student role, but the UI should make it very clear what the ramifications are (i.e. “NOTE! If you attempt this quiz in student mode, you will not be capable of editing the quiz any more!”)

Grading options: make the consequences of each choice obvious

• No. 13
• User:6
• Issue: when told to set points, searches for the category of the question bank and sees if the grade can be set there

• No. 14
• User: 7 (custom exam, no predefined test tasks)
• Issue: The user was unclear about what the grading (?) meant: “10, onkse niinku ykkösestä kymppiin, näkyiskö se opiskelijalle?” (“10, is that like a grade from 1 to 10, would that be visible to a student?”)
• Solution: a help icon (is it there already?)

• No. 15
• User: 4
• Issue: when told to change the grades of questions, went to the edit page of each question in the category and edited the "default grades" (?) within the single questions, too. S/he did this even after noticing the grade selections on the quiz editing page.

• No. 16
• User: 2 (test in English; native English speaker)
• Task: 2
• Comment:"Grade... None. I don't know so I'll leave it"

Quiz / general navigation: stay with general web standards!

For solutions, see also Where to go next with Quiz UI?

• No. 17
• User: 6
• Task: 6/7 (Random question task 1)
• Issue: The user did not know how to get on "houmiin" (finnish-english for "to home"), a basic "start menu" to start searching for the functionality to create what s/he called "arpakone" ("lottery machine", meaning a random question).
• Solutions: Where to go next with Quiz UI?
  

• No. 18
• User: 6
• Issue: When trying to read basic ideas, the user went to "tentit" (quizzes) page on breadcrumb
• -> I directed him to quiz edit page, where s/he found the instructions

• No. 19
• User: 6
• Task: 6/7 (Random question task 1)
• Issue: After getting lost, the user commented: "I don't know if I can use the backspace, on the bank's site I can not".
• Solution: That time, the back button worked, but make sure all pages receiving method=POST forms in Moodle forward the browser instantly to a page which does not require the request to be resubmitted if the user tries to "go back" to it.

• No. 20
• User: 6
• Issue: The user tried if "Update this quiz" would save his essay question. Found another form there, returned.
• Solution: Change update this quiz to describe what it actually is, Quiz settings/options. See also: Increase use of tabs on all mod activities

• No. 21
• User: 3,4
• The user sought after "basic ideas of quiz" from the "tietoa" (info) tab.
• I decided to tell them where it was since this was not part of the actual test.

• No. 22
• User: 1
• Task: 8
• Issue: The user did not find the link to the help

• No. 23
• User: 1
• Issue: Having went to the front page to proceed, the user was puzzled when the quiz link no longer went to quiz edit page.

• No. 24
• User: 1
• Task: 1
• Issue: Trying to create a quiz, the user (unnecessarily) selected "muokkaustila" before clicking the quiz on the course front page

• No. 25
• User: 1
• Issue: Trying to create a quiz, the user tried "aineisto" though quiz is in "aktiviteetti" menu

• No. 26
• User: 2 (test in English; native English speaker)
• Task: 1; create a quiz
• Issue: The user went to Questions on the course front page. Attempted to create a new question until interrupted.

Make links show in standard colours/underlining

Obviously, this is a moodle-wide decision.

• No. 27
• User: 2 (test in English; native English speaker)
• Task: 5
• Issue: The user did not at first think the non-underlined question title link was the way to edit the question. Tried preview before trying it.
• Solution: Add edit icon

• No. 28
• User: 7 (custom exam, no predefined test tasks)
• Issue: The user assumed that the red hover color on a link signified an error. This was because, when moving from a question's edit page to the quiz edit page, the mouse was situated so that it was on an existing question's title. The user realized that it was not so after moving the mouse away from the link. Nevertheless, this is a good demonstration of the unnecessary confusion that can result from using nonstandard link visuals.
• Solution: Strictly make all links use standard colours and underlining (bright colour for unvisited, darker on visited links where it makes sense). In menus underlining is not necessary.

Make question text compulsory default field, and question name optional

I concentrated only on developing the quiz editing UI at this point. Nonetheless, since creating questions is fundamental to the process of creating quizzes, I needed to test the question creating forms in the usability tests in addition to the actual UI. However, the users only used the most basic features of Essay, unless they specifically wanted to try the other question types.

The main single usability fault in the tests, in addition to the problems in the learning process of question bank/random question was the fact that questions are required to have names and not question text. This problem was faced by most, if not all test subjects both in the earlier prototype tests and in the final usability test.

Also, the question name is more prominent than question text in the form, the meaning of which is not clear at all. The labels of the HTMLareas were mostly ingnored by their user duu to the fact that they are laid out to seem unrelated of the boxes.

The novice users always entered the question to the question name field and most often were puzzled of the purpose of the other boxes below that. Experienced users entered their question in both question name and question text, but did not seem to understand what use would the question name field be. In addition, the answer fields of a multiple choice contradict this logic: the first input field, graphically similar to “question name” field, actually does contain the answer visible to the student. This caused the one user that tried to use multiple choice to try to put the multiple choice's answers in the feedback field.

Also, the preview screens of single questions made this worse, since they led any users that tried the preview after writing the question in the question name only, to believe that the question they entered actually is visible in the quiz to the student. One user commented that the “Preview” word should be separated from the rest of the title so that it would be obvious what the question text was.

• No. 29
• User:3
• Task: 1
• Issue: The user filled in the the question in the question name field.

• No. 30
• User: 5
• Task: 1
• Issue: The user wrote the question in both question name and question text.

• No. 31
• User: 4
• Issue: The user said s/he doesn't understand what the question name field is for. Used it and question text correctly (though the question name was not meaningful) anyway.

• No. 32
• User: 3, 6
• Task: 1
• Issue: The single question preview confirmed the user's false belief that Question name equals question shown to student, by showing the question name as the heading of the preview window.

• No. 33
• User: 1
• Issue: User first looked at question name, then at question text. User tried to view the help and browsed around in the help window. Closed help window and finally wrote in the kysymysteksti field. Kysymysteksti field hs laid out on the page strangely and the button rows are in the way.

Question names

The users also repeatedly went for Random short-answer matching when trying to create random questions.

• No. 34
• User: 7 (custom exam, no predefined test tasks)
• Issue: The user's first guess for a free text question was “Kuvaus” (description). S/he thought “Essay” would be too long, as this question only required a short answer.
• Solution: Rename Description to “Help text/description” or similar, and Essay to “Free text question (Essay)”

• No. 35
• User: 7 (custom exam, no predefined test tasks)
• Issue: The user did not find “True/false” in the question type list (Note: No other issues using true/false)
• Solution: Translate the name in the Finnish version

Miscellaneous

• No. 36
• User: 4
• Issue: To make sure that his/her changes were saved, the user pushed the button that saves an individual question's grade.
• Solution: Lay out the titles of the text boxes so that they are not less prominent than the text boxes themselves, making it clear that the box and button are only for saving grades.

• No. 37
• User: 5
• Issue: The user did not understand the "täytä oikeilla" of the question preview window.

• No. 38
• User: 5
• -Issue: The user commented that s/he does not understand "kaikkien ... hakemisto" (Index of all help files)

• No. 39
• User: 5
• Issue: The user asked to have the "edit icons" back like in the rest of moodle (although when asked, did not seem to remember what the icon looked like.)
• Solution: Add the edit icons back to everywhere, keep consistency

• No. 40
• User:6
• Issue: The user wanted to save his/her changes, commented that s/he is confused about the fact that in some places saving (separately, using a button) is necessary, and in others not.