Quiz UI redesign - what makes a scenarios interview?

Back to Quiz_UI_redesign

The idea and the content of the scenarios interviews I did

A.k.a. Quiz UI scenario gathering work in Summer 2008 - what, why and how?

After I had received the Kesäkoodi [1] funding for the project [2], it quickly started to dawn to me that if I want to take designing the new UI seriously, what it takes is really getting to know the users - teachers, IT support, and even administrators. I used all the contacts I had - namely, my old job at the university of Tampere, the ITPEDA mailing list and Moodle-rinki, a community of Finnish Moodle users.

I studied a bit about what the methods to get learn about the users are. First, I considered doing contextual inquiries, but that would have required a lot more time with each user than the one hour I had managed to agree with some teachers. So, a rather informal interview it was. Next, I am going to introduce here the different parts of what I did, so you can do so too if you want to. Everything here has been freely translated from Finnish so that you can get an idea.

Background material

• Forms
  • Consent for recording audio (1 A4 sheet)
  • Background information form (1 A4 sheet):
    • Frequency of website/browser usage
    • Do you experience with which learning environments?
    • How often do you use a learning environment as a teacher
    • Question types you use: multiple choice; essay; other, what
    • Categorisation basis of question categories
    • Do you use electronic exams, with which software?
    • If you use websites, which kinds of services do you mostly use?
• Question list

Preparatory talk

"We are developing the Moodle Quiz module. The goal is to understand how actual users/teachers do their work, especially related to exams. The setting here, so to speak, is that you are the master and I am an apprentice - that is, I will mostly just ask questions, listen and take notes. I apologize in advance that I might seem very concentrated on my notes at some point, since I am not a very accustomed interviewer. "

Actual interview questions

The idea is not to follow this question list strictly, but to use it as a rough guideline of the questions to find out about. When a teacher's thought wanders, we let it do so and may ask previously unthought of questions related to what a teacher has said, if it seems related.

Background

• Please describe the character of your work.
  • What kinds of responsibilities do you have?
  • What forms of teaching do you use; group work, lectures, online teaching, ...?
• Apart from reality, if you could imagine a tool that would optimally serve your way of making exams/quizzes, what would the experience of using it be like? What tasks related to making quizzes/exams could a machine help you with or do for you?
• What is usually the context of evaluating students' learning?
• Within that context, how do you end up with the conclusion that your course requires an exam/quiz?
  • Curriculum, course?
• What do you think about exams as a way to evaluate students' learning?

Exam making

Process

• What tools or accessories do you use while making exams/quizzes, physical or software?
• Please describe how you proceed from that to making an exam/quiz. What are the different stages?
• If you have made electronic exams, what do you do, how does the process differ from making pen and paper exams/quizzes?
• Chronologically (on a time scale), at which stage of a course do you think of the exam/quiz content for the first time?
  • How long a time is there between making the quiz/exam, actually having the students do the exam, and grading it? How often do students take your quizzes/exams, what determines this?
  • When do you create the actual, exam/quiz which you give to students to do?

Environment/situation, content

• What is the situation like when you create an exam? Where do you do the exam physically? Are there distractions or can you tolerate distractions?
• What do you use as material or tools when creating an exam?
• Do you have to copy some information to many places?
• Do you cooperate with other parties when making exams or otherwise about exams? What are the ways/medias in which you communicate with them?
  • What determines the structure of an exam?
  • Do you use old exams as material?
• Is the order of questions predefined and determined already before you create the actual exam, or do you reorder questions when they are already in the exam?

Tools

• What is problematic about the current tools for making quizzes/exams?
• What is good about current tools for making quizzes/exams?
• When you create an exam, where there are questions that are randomly selected from a larger selection of questions, to each exam attempt, what are the stages to doing this?

If they have used Moodle Quiz:

• Do you use names of questions? How, for what? What could you imagine them being used for?