Usability testing in August 2008/Test setting and tasks / background

From MoodleDocs

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

Return to: Quiz UI redesign | Quiz Usability portal

Test design

In order to see if the”basic ideas of creating quizzes” instructions would help users “too much”, confusing them, or whether the UI would enable them to make random questions without any help, I added a new random question creating task in the new usability test before the task where they were told to read the instructions. Though all users had some level of difficulty understanding the relationship between Quiz, Question Bank and Random Questions, I was surprised to notice that some of the users added random questions with relative ease before reading the instructions, and one of the users actually did better before reading the instructions than after it. On the other hand, some users could create a random question better after having read the instructions (and before that, having tried to create a random question without them)

The first task was included only in the first two tests. It took so long for the test subjects to create the quiz and to get stuck in the quiz creating form that

The preparation talk I gave to the subjects and the basic idea of the testing was similar as in the first iteration OOo prototype testing

Issues related to the test setting

Being told to read instructions is obviously not natural and changes the setting. Essentially, what the random question 2 task measured was the users' capability to learn concepts on request in a situation where they probably had much greater resistance to giving up than in normal situations – having to admit to the test organizer they can not do it.

So one could perhaps say as a conclusion that it is possible to figure out how to use random questions under moderate pressure. It was not confirmed whether or not this would have been possible with the old UI as well, although the new one gives a clearer kick start by allowing to create an empty category along with a new random question for it. N of the users succeeded in creating a random question without the instructions, and one of those users created one it much more fluently before reading the instructions than after reading them, in the latter case wondering out loud “how did I do it the first time?”.

  • User:3
  • (Test Issue: The question bank was open before it was supposed to)
  • In all four tests of the day. On the other hand, since after the initial opening of the question bank it is likely to be rare to hide it again, this does demonstrate the question bank's state in most cases. Nevertheless, if it would have been hidden at first, this might have helped see if the users would actually benefit from having the progressive disclosure.

  • User:6
  • *Test Issue: The name of the random question category might have been the name of an exam itself (vallankumouksen jälkeiset vuodet)

  • User:6
  • *Test Issue: The user said "Niin paljon samoja sanoja ettei voi olla kokeilematta" ("so many of the same words that I can't not try") with the "add page here" button and the test task
  • Note: The task did not measure whether or not the users knew how to use the application but their capability of finding the same words in the ui that were in the task. (check how many)


  1. (Create a new quiz on the course “My Course”.) This task was left out after the first test as irrelevant to Quiz itself, as it took too much time from the start of the Quiz.
  2. Create a new essay question to the quiz
  3. Question: Describe Napleon's journey on his way to autocracy.
  4. See how the question you just created looks like to a student in the quiz.
  5. You notice that the question you just made has a spelling error. Fix the error.
  6. To make sure students can not copy questions to each other, you want a selection of questions to the quiz, from which one is randomly chosen for each time a student takes the quiz. Create such a selection.
  7. The topic for the questions: The Life of Marie Antoinette
  8. Add two questions to the selection you just created.
    -Gaining power: Describe Marie Antoinettes efforts to become powerful
    -Death: Which events led to the death of Marie Antoinette
  9. Read about the Basic ideas of making quizzes. Especially familiarize yourself with the concept of a Random question. If something still seems unclear to you, please tell about this to the test organizer.
  10. To make sure students can not copy questions to each other, you want a set of questions to the quiz, from which one is randomly chosen for each time a student takes the quiz. The subject for the question is The Years After the Revolution.
  11. Add two questions to the set you just created.
    How would you describe Napoleon's strategic strengths?
    Analyze the damage, caused by the war, in the society
  12. Add a new page to the quiz. Ensure that only question number 1 is on the first page.
  13. Move the question, currently third in the quiz, to the same page where the first question is.
  14. Switch the order of the questions, which are on the same page.
  15. Add instructions to the exam to tell the student that especially random question number 1 must be answered according to the writing norms of scientific texts.
  16. Open the tab “Order and paging”.
  17. Add a new page to the end of the exam.
  18. Move all the questions in the quiz to the last page in the quiz.
  19. Set grades for each of the questions, which you think are appropriate.
  20. Remove one question from the quiz.
  21. Notify the student in the exam about the fact that cheating will have grave consequences.

Users' background info

TODO: In order to understand the type of users the UI is being designed for, I gathered some information about their computer experience level etc. on forms. The problem is that to protect their privacy, I have to carefully consider what to publish here.

See also