Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Quiz UI redesign scenarios - Preparing questions

From MoodleDocs
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

Back to Scenarios index

Introductory video

Preparing questions

Users of various question types: your feedback is especially needed at this point. Which goals, pedagogical or other kinds, make you choose different question types for your exams?


Mack Marketing

  • Questions come from a common question bank

Prepreparation: sketching questions

Sketching: While designing the course itself and its topics, at least for the first time, Mack often does early "sketching" of his marketing exam questions for at least some of the topics. When a new course instance starts to which new themes have been added, he may return to this sketching phase.

Material for questions

The exam is structured according to the curriculum, making sure that every essential subject field has been covered.

To determine a reasonable difficulty level for the exam (and to get feedback about his classroom teaching), he also uses assignments during the course. The exam content also varies based on which themes Mack has had time for/covered during a given course instance.

When designing the exam and its questions, Mack then takes

  • Literary on course themes;
    • scientific articles, textbooks etc.
    • Web pages (does multitasking while having the exam making application, such as Word, open in another window)
    • Current, topical articles from newspapers
  • Question sketches (see above)
  • The course curriculum which
    • lists course literature
    • gives a broad structure of the course.
  • Question metadata (see below)

He will use some of these also as material students need to read before or during the exam and refer to in their exam answers.

Questions from students

Mack has also asked students to create relevant exam questions during the course, and some of the questions he has gotten have proven surprisingly good. He then has taken the best ones and imported them to his question bank (which is a Word document).

Question metadata

To make decisions about what to actually use in his exam, Mack uses any information about the past usefulness of individual questions.

The school Mack teaches in has a common question bank and thus some of the questions he uses may have been written by someone else than him so any background information to understand the motives behind the question and how it has worked is needed.

With old/existing exams, questions, exercises, Mack uses the following kinds of metadata (information about questions)

  • Feedback about issues from students (for example, in understanding a question)
    • As some exams occur only once per semester or sometimes even more rarely, he tends to forget to take into account student feedback, which he reseives currently usually by email or in the classroom. Sometimes students also give feedback about questions in their exam answers.
  • issues discovered while reading students' exam answers (for example, the student has misunderstood the question and answered to a different question or from a wrong point of view)
  • statistical data about how students have done in previous exams replying a certain question
    • In addition to written feedback, Mack has also graded students' answers to his own questions from previous semesters numerically, so he can use statistical tools to get an idea about which questions have worked before.

While creating a question, Mack notes down two more pieces of metadata:

  • a model/template answer to use as grading criteria when he gets the exam answers from students.
  • feedback templates, which he can easily customize in order to give qualitative feedback to students about their exam answers

This information about the questions (metadata) would need to be stored with the questions themselves so that they would be easy to find where they are needed.

Writing questions

When Mack has all the material there, he starts to write questions for each of the course themes/topics based on them. He does this category by category, trying to balance each category appropriately. Mack's questions are often based on real-world cases, which the student needs to apply his knowledge to.

Based on how the course has gone that time, he looks at what has seemed relevant that time around - the courses themselves can be very dynamic. He then makes the selection of some of the questions based on that.

While creating a question, Mack notes down another piece of metadata: a model/template answer to use as grading criteria when he gets the exam answers from students.

Ida Informatics

Material

To create her extensive question banks, Ida uses

  • existing questions as templates,
    • she has written notes (metadata) about the usefulness of those questions alongside the questions
  • the material she has covered during the lectures, which is a fairly standardized selection of themes since the exam results need to be comparable across semesters.

Ida needs to, however, adjust the selections of questions in different themes, as well as the number of questions from different categories, based on also the statistics she can see about how well students have done with different questions.

Question metadata

Ida also edits her question material outside Moodle - exporting and importing questions as needed – since the web interface does not provide functionality such as question bank wide text search and replace. It is thus crucial for her that the import/export mechanism works without error, or at least reports to her when some part of her data could not be preserved. It would be important to preserve question metadata even when it is exported and imported.

Even though her questions are selected randomly from categories to exams, she sometimes wants to primarily use questions that have not been seen by students before, to gather data about how well those questions work. For this, she needs to know when a question has been used.

She gets feedback about the exam from students, but since she does not use any question types where students could freely type their answers, he has not gotten feedback about exam questions with the exams. Also feedback about how a student has understood a question in the first place is useful, since if a student has answered incorrectly but the answer makes sense in relation to how the student has understood the question, it might still be just to give him/her a good grade for that question.

Writing questions

Ida considers exam and question design very intensive and solitary work, where she wants to lock herself out from the world around her for one or more days, possibly not even answering the phone.

As a consequence, does this fairly seldom, once per year or possibly not even that often, and wants to get a fairly finalized set of questions as a result.

She takes her early ideas about the course themes and possible multiple choice answer options she may have come to think of while designing the course content. She then tries to create a sufficient amount of multiple choices for each theme, one theme at a time, for the different difficulty levels. The most important considerations are to make sure the questions are not obvious or ambiguous and that they are valid, actually test the understanding they are designed to test. She also needs to check for any typos or wrong/imprecise terms, since they can affect how well students understand questions.

For Ida, the central resource is the question bank, which she tries to complete at once and then refine later. However, since the work demands full attention, she finds it hard to find time, in the middle of everyday work, to orient herself to this refining and editing the questions based on feedback. Here, the tools she uses could perhaps support her memory so that there would be less of a barrier to get the refining done.

For Ida, an essential goal is to be capable to perceive all of the questions of a given theme at once, in order to evaluate whether the theme has been covered to a sufficient degree. If she decides it hasn't, she can then add new questions from a new angle to her collection.

Conclusions

Teachers could have an option to have a “give feedback about this question” button in the exam with each/some of the questions which would give a separate writing area for the student to write in. This would allow student-entered metadata to be stored with the questions and presented to the teacher in context.

Harvey Historian

Harvey has a history exam that every student needs to take at some point of their studies. The exam covers five books, which are all in the status of a classic in Harvey's subject field. There are only so many questions that can be made from each of these books, and as Harvey has taught the course for two decades, he has managed to gather pretty much all variations of them.

Harvey has also given some of his courses in other languages than his native one, while doing teacher exchange. For those exams, he has translated and adapted his questions to fit the local cultural context. However, the information about the questions (metadata) is shared between different exams, and also the grading is made comparable between different-language exams. The students choose when they come to the exam, which language they want to take the language in.

Fred

As Fred's course is an advanced one with other courses as prerequisites, he has a test at the beginning of a course to see what is the general level of a given year's students. In this test, he uses some of the more difficult questions of the exams of the prerequisite courses.

He also uses the prerequisite courses' exams for reference, since those exams demonstrate quite well which subjects have been taught on those courses - he is painfully aware that the official curriculum often does not correspond tightly with the actual course content.

Grace Grader

Susan Support

Paul Pedagogue

National Exam grading cop-out

If Paul were to use his own high goals, too many students would get low grades compared to if they went to other teachers, but he doesn't want to just give arbitrary "nice grades". Since Paul finds grading to be "not his thing", he chooses to use the National Central Exams as the basis for his grading. They have a well-defined grading scheme and the questions are often quite varied and interesting (to him, though students tend to prefer the usual "cook book problems"). The exams are given on paper, in a controlled room at the end of the course and he grades them according to the official scheme and uses that grade for the whole course unless some other interaction with the student has given him reason to adjust that grade. He saves notes on various student answers so that he can give tutorial feedback when these tests are not longer secret and can be used for practice. He always offers to students to have an oral session after the exam if the student wishes it. (Some students can get exam black-outs, be better at talking than writing etc.) He hopes that there will be acceptably secure on line testing alternatives in the future, since he finds correcting work uninspiring.

Practice past National Exams

Since both he and the students want them to perform well on the National exams, he starts by making practice questions based on the previous exams. He tries to provide tutorial process feedback for as many answers as possible. Multiple choice questions are easy. More open questions can be more challenging to make into automatic correction questions. Sometimes they have to be broken up into a suite of questions checking whether the student can do the various steps in the process. Though he presents the whole (previous) exams as a package to accustom the students to how the exams work, he would like to be able to tag the questions to be easily found and used in different parts of the course.

Preparing for laboratory

His distance students have to do a large number of laboratory exercises during a single day at the school. To make sure that they can get as much as possible out of the economically limited time, there are some things they must do before coming to a lab day. They must read descriptions about safety routines, how to make and record measurements. There are also some laboratory videos and Java and FLASH simulations they are to do some exercises on. To make sure they have understood the reading and the simulations they have to do a quiz. It will even include screenshots from the videos/simulations in the follow-up questions. There can be some theoretical readings and exercises that should best be mastered before the labs also. The questions following up these things would typically have feedback with links back to the materials/simulations to be mastered. Students would be encouraged to contact the teacher if the material isn't helping. (The teacher has a channel through the quiz statistics to see where the weaknesses are otherwise.)

General Follow-ups

Especially for distance students it is important to give many checkpoints to see if the steps toward the goal are being met. Paul will try to have a couple questions on each Internet page that the student can use to check that. They will usually just have a popup answer. It would be really nice to be able to pull in a couple of Moodle quiz questions into any page. Then you could have a rich feedback opportunity and be able to collect student answers so you can improve the material.


Retrieved from "https://docs.moodle.org/dev/index.php?title=Quiz_UI_redesign_scenarios_-_Preparing_questions&oldid=9451"
Powered by MediaWiki