Development:Quiz UI redesign scenarios - Preparing questions

Back to Development:Quiz_UI_redesign

Preparing questions

Users of various question types: your feedback is especially needed at this point. Which goals, pedagogical or other kinds, make you choose different question types for your exams?

Jack

The school Jack teaches in has a common question bank and thus some of the questions he uses may have been written by someone else than him. To make decisions about what to actually use in his exam, statistical and any other information about the usefulness about individual questions is needed. Also, as some exams occur only once per semester or sometimes even more rarely, he tends to forget feedback about the exam questions received from students, currently usually by email or in the classroom. Sometimes students also give feedback about questions in their exam answers. In addition to written feedback, Jack has also graded students' answers to his own questions from previous semesters numerically, so he can use statistical tools to get an idea about which questions have worked before. This information about the questions (metadata) would need to be stored with the questions themselves so that they would be easy to find where they are needed.

When designing an exam and its questions, he takes the course curriculum and course literature, which contains course books, web pages and current, topical articles from newspapers, which he will use as material students need to read and refer to in their exam answers. He also uses assignments during the course to get feedback about his classroom teaching and to determine a reasonable difficulty level for the exam.

Conclusions: For many teachers, question metadata and statistics about past usage and student success in a particular question seems to be important, since it enables the teachers to make decisions about the which questions to use in current and future quizzes.

Mary

Mary considers exam design phase deeply solitary work, where she wants to lock herself out from the world around her for one or more days, possibly not even answering the phone. She needs to have the course curricula and the material she has covered during the lectures, which is a fairly standardized selection of themes since the exam results need to be comparable across semesters. She needs to however adjust the selections of questions in different themes, as well as the number of questions from different categories, based on also the statistics she can see about how well students have done with different questions.

Mary also edits her question material outside Moodle - exporting and importing questions as needed – since the web interface does not provide functionality such as question bank wide text search and replace. It is thus crucial for her that the import/export mechanism works without error, or at least reports to her when some part of her data could not be preserved. It would be important to preserve question metadata even when it is exported and imported. Even though her questions are selected randomly from categories exams, she sometimes wants to use questions that have not been seen by students before, to gather data about how well those questions work. She gets feedback about the exam from students, but since she does not use any question types where students could freely type their answers, she has not gotten feedback about exam questions with the exams.

Conclusions: Teachers could have an option to have a “give feedback about this question” button in the exam with each/some of the questions which would give a separate writing area for the student to write in. This would allow student-entered metadata to be stored with the questions and presented to the teacher in context.

The content for an exam comes from different sources, depending on its type:

• Literary covering course themes; for example, scientific literature, educational textbooks, belletristic writing, dictionaries, encyclopedias
• Electronic sources such as electronic libraries and the internet (especially with IT subjects)
• Old exams, questions, exercises; metadata such as feedback about an old exam from students or statistical data about how students have done in previous exams replying a certain question

In general, the content is defined by the curriculum of the course; however, a teacher may choose to adjust the selection and breadth of the themes of exam questions, based on

• which themes have been covered on the course and
• previous evaluations made by the teacher during the course about the students' level of learning

Also, metadata such as when a question was last used in an exam may be used by teachers to determine whether it should be included in a certain exam. Even if the questions were selected randomly from categories to a previous exam, it may be possible that a teacher wants to use questions that have not been seen by students before, for example to gather data about how well those questions work.