Quiz UI redesign scenarios - Preparing questions

Back to Quiz_UI_redesign

Preparing questions

Users of various question types: your feedback is especially needed at this point. Which goals, pedagogical or other kinds, make you choose different question types for your exams?

Mack Marketing

• Questions come from a common question bank

Prepreparation / context for exams

While designing the course itself and its topics, at least for the first time, Mack often does early "sketching" of his marketing quiz questions for at least some of the topics. When a new course instance starts to which new themes have been added, he may return to this sketching phase.

For online courses the sketching is left out, since Mack has to design the whole course beforehand, also the exam.

Material for questions

When designing the exam and its questions, Mack then takes

• Aforementioned question sketches
• Literary covering course themes; for example, scientific literature, educational textbooks, belletristic writing, dictionaries, encyclopedias
• Question metadata (see below)
• The course curriculum which lists course literature and gives a broad structure of the course. The exam is structured according to the curriculum, making sure that every essential subject field has been covered.
• Web pages (= does multitasking while having Moodle open in another window, requires Moodle to be able to recover seamlessly)
• Current, topical articles from newspapers

He will use some of these also as material students need to read before or in or the exam and refer to in their exam answers.

To determine a reasonable difficulty level for the exam (and to get feedback about his classroom teaching), he also uses assignments during the course. The exam content also varies based on which themes Mack has had time for/covered during a given course instance.

Questions from students

Mack has also asked students to create relevant exam questions during the course, and some of the questions he has gotten have proven surprisingly good. He then has taken the best ones and imported them to his question bank (which is a Word document).

Question metadata

To make decisions about what to actually use in his exam, Mack uses any information about the past usefulness of individual questions.

With old exams, questions, exercises, Mack uses metadata such as

• feedback about issues from students (for example, in understanding a question)
• issues discovered while reading students' exam answers (for example, the student has misunderstood the question and answered to a different question or from a wrong point of view)
• statistical data about how students have done in previous exams replying a certain question

While creating a question, Mack notes down another piece of metadata: a model/template answer to use as grading criteria when he gets the exam answers from students.

The school Mack teaches in has a common question bank and thus some of the questions he uses may have been written by someone else than him so any background information to understand the motives behind the question and how it has worked is needed.

In addition to written feedback, Mack has also graded students' answers to his own questions from previous semesters numerically, so he can use statistical tools to get an idea about which questions have worked before.

Also, as some exams occur only once per semester or sometimes even more rarely, he tends to forget feedback about the exam questions received from students, currently usually by email or in the classroom. Sometimes students also give feedback about questions in their exam answers.

This information about the questions (metadata) would need to be stored with the questions themselves so that they would be easy to find where they are needed.

Writing questions

When Mack has all the material there, he starts to write questions for each of the course themes/topics based on them. He does this category by category, trying to balance each category appropriately.

While creating a question, Mack notes down another piece of metadata: a model/template answer to use as grading criteria when he gets the exam answers from students.

Conclusions

For many teachers, question metadata and statistics about past usage and student success in a particular question seems to be important, since it enables the teachers to make decisions about the which questions to use in current and future quizzes, and what about a particular question should be reconsidered.

Ida Informatics

Material

To create her extensive question banks, Ida uses

• existing questions as templates,
  • she has written notes (metadata) about the usefulness of those questions alongside the questions
• the material she has covered during the lectures, which is a fairly standardized selection of themes since the exam results need to be comparable across semesters.

Ida needs to, however, adjust the selections of questions in different themes, as well as the number of questions from different categories, based on also the statistics she can see about how well students have done with different questions.

Question metadata

Ida also edits her question material outside Moodle - exporting and importing questions as needed – since the web interface does not provide functionality such as question bank wide text search and replace. It is thus crucial for her that the import/export mechanism works without error, or at least reports to her when some part of her data could not be preserved. It would be important to preserve question metadata even when it is exported and imported.

Even though her questions are selected randomly from categories to exams, she sometimes wants to primarily use questions that have not been seen by students before, to gather data about how well those questions work. For this, she needs to know when a question has been used.

She gets feedback about the exam from students, but since she does not use any question types where students could freely type their answers, he has not gotten feedback about exam questions with the exams. Also feedback about how a student has understood a question in the first place is useful, since if a student has answered incorrectly but the answer makes sense in relation to how the student has understood the question, it might still be just to give him/her a good grade for that question.

Writing questions

Ida considers exam and question design very intensive and solitary work, where she wants to lock herself out from the world around her for one or more days, possibly not even answering the phone.

As a consequence, does this fairly seldom, once per year or possibly not even that often, and wants to get a fairly finalized set of questions as a result.

She takes her early ideas about the course themes and possible multiple choice answer options she may have come to think of while designing the course content. She then tries to create a sufficient amount of multiple choices for each theme, one theme at a time, for the different difficulty levels. The most important considerations are to make sure the questions are not obvious or ambiguous and that they are valid, actually test the understanding they are designed to test. She also needs to check for any typos or wrong/imprecise terms, since they can affect how well students understand questions.

For Ida, the central resource is the question bank, which she tries to complete at once and then refine later. However, since the work demands full attention, she finds it hard to find time, in the middle of everyday work, to orient herself to this refining and editing the questions based on feedback. Here, the tools she uses could perhaps support her memory so that there would be less of a barrier to get the refining done.

For Ida, an essential goal is to be capable to perceive all of the questions of a given theme at once, in order to evaluate whether the theme has been covered to a sufficient degree. If she decides it hasn't, she can then add new questions from a new angle to her collection.

Conclusions

Teachers could have an option to have a “give feedback about this question” button in the exam with each/some of the questions which would give a separate writing area for the student to write in. This would allow student-entered metadata to be stored with the questions and presented to the teacher in context.

Harvey Historian

Harvey has a history exam that every student needs to take at some point of their studies. The exam covers five books, which are all in the status of a classic in Harvey's subject field. There are only so many questions that can be made from each of these books, and as Harvey has taught the course for two decades, he has managed to gather pretty much all variations of them.

Fred

As Fred's course is an advanced one with other courses as prerequisites, he has a test at the beginning of a course to see what is the general level of a given year's students. In this test, he uses some of the more difficult questions of the exams of the prerequisite courses.

He also uses the prerequisite courses' exams for reference, since those exams demonstrate quite well which subjects have been taught on those courses - he is painfully aware that the official curriculum often does not correspond tightly with the actual course content.