Quiz UI redesign scenarios - Preparing questions

Back to Quiz_UI_redesign

Preparing questions

Users of various question types: your feedback is especially needed at this point. Which goals, pedagogical or other kinds, make you choose different question types for your exams?

List of material used while creating questions

• Literary covering course themes; for example, scientific literature, educational textbooks, belletristic writing, dictionaries, encyclopedias
• Electronic sources such as electronic libraries and the internet (especially with IT subjects)
• Old exams, questions, exercises; metadata such as feedback about an old exam from students or statistical data about how students have done in previous exams replying a certain question
• current, topical articles from newspapers

Jack

• Questions come from a common question bank

Before getting to the actual exam, while designing the course itself and its topics, Jack often sketches quiz questions he is going to use.

When designing the exam and its questions, he takes the course curriculum, which lists course literature and gives a broad structure of the course. The exam is structured according to the curriculum, making sure that every essential subject field has been covered. Also, he uses

• web pages
• current, topical articles from newspapers

which he will use as material students need to read and refer to in their exam answers. He also uses assignments during the course to get feedback about his classroom teaching and to determine a reasonable difficulty level for the exam. The exam content also varies based on which themes Jack has covered during a given course instance.

Question metadata: The school Jack teaches in has a common question bank and thus some of the questions he uses may have been written by someone else than him. To make decisions about what to actually use in his exam, statistical and any other information about the usefulness about individual questions is needed. Also, as some exams occur only once per semester or sometimes even more rarely, he tends to forget feedback about the exam questions received from students, currently usually by email or in the classroom. Sometimes students also give feedback about questions in their exam answers. In addition to written feedback, Jack has also graded students' answers to his own questions from previous semesters numerically, so he can use statistical tools to get an idea about which questions have worked before. This information about the questions (metadata) would need to be stored with the questions themselves so that they would be easy to find where they are needed.

Conclusions: For many teachers, question metadata and statistics about past usage and student success in a particular question seems to be important, since it enables the teachers to make decisions about the which questions to use in current and future quizzes.

Mary

Circumstances: Mary considers exam design phase deeply solitary work, where she wants to lock herself out from the world around her for one or more days, possibly not even answering the phone.

Material: To create her extensive question banks, Mary uses existing questions as templates. She also needs to have the course curriculum and the material she has covered during the lectures, which is a fairly standardized selection of themes since the exam results need to be comparable across semesters. She needs to however adjust the selections of questions in different themes, as well as the number of questions from different categories, based on also the statistics she can see about how well students have done with different questions.

Overview: For Mary, an essential goal is to be capable to see all of the questions of a given theme at once, in order to evaluate whether the theme has been covered to a sufficient degree. If she decides it hasn't, she can then add new questions from a new angle to her collection.

Question metadata: Mary also edits her question material also outside Moodle - exporting and importing questions as needed – since the web interface does not provide functionality such as question bank wide text search and replace. It is thus crucial for her that the import/export mechanism works without error, or at least reports to her when some part of her data could not be preserved.

Conclusions: Teachers could have an option to have a “give feedback about this question” button in the exam with each/some of the questions which would give a separate writing area for the student to write in. This would allow student-entered metadata to be stored with the questions and presented to the teacher in context.

The content for an exam comes from different sources, depending on its type:

In general, the content is defined by the curriculum of the course; however, a teacher may choose to adjust the selection and breadth of the themes of exam questions, based on

Also, metadata such as when a question was last used in an exam may be used by teachers to determine whether it should be included in a certain exam. Even if the questions were selected randomly from categories to a previous exam, it may be possible that a teacher wants to use questions that have not been seen by students before, for example to gather data about how well those questions work.