Quiz UI redesign scenarios - Preparing questions

Preparing questions

Users of various question types: your feedback is especially needed at this point. Which goals, pedagogical or other kinds, make you choose different question types for your exams?

List of material used while creating questions

Literary covering course themes; for example, scientific literature, educational textbooks, belletristic writing, dictionaries, encyclopedias
Electronic sources such as electronic libraries and the internet (especially with IT subjects)

current, topical articles from newspapers

Jack

Questions come from a common question bank

Prepreparation / context for exams

While designing the course itself and its topics, at least for the first time, Jack often does early "sketching" of quiz questions for at least some of the topics. When a new course instance starts to which new themes have been added, he may return to this sketching phase.

Material for questions

When designing the exam and its questions, Jack then takes

Aforementioned question sketches
Literary covering course themes; for example, scientific literature, educational textbooks, belletristic writing, dictionaries, encyclopedias
Question metadata (see below)
The course curriculum which lists course literature and gives a broad structure of the course. The exam is structured according to the curriculum, making sure that every essential subject field has been covered.
Web pages (= does multitasking while having Moodle open in another window, requires Moodle to be able to recover seamlessly)
Current, topical articles from newspapers

He will use some of these also as material students need to read before or in or the exam and refer to in their exam answers.

To determine a reasonable difficulty level for the exam (and to get feedback about his classroom teaching), he also uses assignments during the course. The exam content also varies based on which themes Jack has had time for/covered during a given course instance.

Question metadata

To make decisions about what to actually use in his exam, Jack uses any information about the past usefulness of individual questions.

With old exams, questions, exercises, Jack uses metadata such as

feedback about issues from students (for example, in understanding a question)
issues discovered while reading students' exam answers (for example, the student has misunderstood the question and answered to a different question or from a wrong point of view)
statistical data about how students have done in previous exams replying a certain question

The school Jack teaches in has a common question bank and thus some of the questions he uses may have been written by someone else than him so any background information to understand the motives behind the question and how it has worked is needed.

In addition to written feedback, Jack has also graded students' answers to his own questions from previous semesters numerically, so he can use statistical tools to get an idea about which questions have worked before.

Also, as some exams occur only once per semester or sometimes even more rarely, he tends to forget feedback about the exam questions received from students, currently usually by email or in the classroom. Sometimes students also give feedback about questions in their exam answers.

This information about the questions (metadata) would need to be stored with the questions themselves so that they would be easy to find where they are needed.

Writing questions

When Jack has all the material there, he starts to write questions for each of the course themes/topics based on them. He does this category by category, trying to balance each category appropriately.

Conclusions

For many teachers, question metadata and statistics about past usage and student success in a particular question seems to be important, since it enables the teachers to make decisions about the which questions to use in current and future quizzes.

Mary

Circumstances: Mary considers exam design phase very intensive and solitary work, where she wants to lock herself out from the world around her for one or more days, possibly not even answering the phone.

Material: To create her extensive question banks, Mary uses

existing questions as templates,
- she has written notes (metadata) about the usefulness of those questions alongside the questions
the material she has covered during the lectures, which is a fairly standardized selection of themes since the exam results need to be comparable across semesters.

Mary needs to, however, adjust the selections of questions in different themes, as well as the number of questions from different categories, based on also the statistics she can see about how well students have done with different questions.

Question metadata: Mary also edits her question material also outside Moodle - exporting and importing questions as needed – since the web interface does not provide functionality such as question bank wide text search and replace. It is thus crucial for her that the import/export mechanism works without error, or at least reports to her when some part of her data could not be preserved. It would be important to preserve question metadata even when it is exported and imported.

Even though her questions are selected randomly from categories to exams, she sometimes wants to primarily use questions that have not been seen by students before, to gather data about how well those questions work. For this, she needs to know when a question has been used.

She gets feedback about the exam from students, but since she does not use any question types where students could freely type their answers, he has not gotten feedback about exam questions with the exams.

Writing questions She does this fairly seldomly, once per year or possibly not even that often. For Mary, the central resource is the question bank, which she tries to complete at once and then refine later. However, she finds it hard to find time for this refining and editing the questions based on feedback

For Mary, an essential goal is to be capable to see all of the questions of a given theme at once, in order to evaluate whether the theme has been covered to a sufficient degree. If she decides it hasn't, she can then add new questions from a new angle to her collection.

Conclusions: Teachers could have an option to have a “give feedback about this question” button in the exam with each/some of the questions which would give a separate writing area for the student to write in. This would allow student-entered metadata to be stored with the questions and presented to the teacher in context.

Documentation