Quiz UI redesign scenarios - Background
Back to Scenarios index
Background/Personas
An exam can be used at the beginning of a course to evaluate the initial level of the students' understanding, during the course to provide checkpoints for students, or at the end of the course. In any case, quizzes/exams often have a dual role: they give the teacher information about the student's skills, but more importantly they are a pedagogical tool: while answering, a student generates knowledge and learns in the process.
If it seems to you that your idea of using exam is very different from any of the current personas, you may add a new one or tell us about this in the forum thread dedicated for persona/scenario discussion
Key questions for this part of the exam making:
- What are the pedagogigal goals for the teacher?
- Does the exam use essay questions or automatically evaluated (or what)?
- What initiates the lifecycle of an exam?
Mack Marketing
- Mack's persona
- Mack: Preparing questions
- Mack: Organizing questions and adding them into an exam
- Forum discussion to comment on scenarios
Mack is a marketing teacher. He likes golf, but not on rainy days.
- He has quite a bit of experience using the internet, mainly for webmail
He also teaches online a bit, but mostly he has normal classes. In addition to regular, time-limited exams, he also uses take-home exams (with deadlines for the questions/case assignments) and group exams (a group works together on a a case and the work is evaluated).
Exams: why and what
While defining the goals of his course, he ends up with the conclusion that an exam with essay questions is needed at the end of the course for evaluating students' learning. He does not think exams are a very good way of evaluating learning, since they mostly evaluate what students happen to remember. And even though he considers grading essay answers an extra workload, he still sees this as the only way to effectively evaluate students' conceptual understanding, especially on his basic courses where there are more students.
He uses quite similar exams on different courses, our focus is on a Introduction to Marketing exam, which is about:
- Evaluating conceptual learning; can apply what they have learned to real-life cases
- Essay questions
- Qualitative feedback; using templates as basis
Hypothesis: Since his exams are so specialized, Mack is afraid that if another teacher takes up his courses, that teacher will not understand all the nuances he has taken into account while designing his exams. That is why he writes notes about each exam whole and archives these notes that explain the choices he has made in the exam design.
Feedback
He plans to give feedback about students' answers: for him, this enables the exam to work as a pedagogical tool and he does not require the exam results to be absolutely comparable with the exam results of courses of previous semesters. He uses some templates for feedback, and based on that he then writes feedback for each student personally.
Ida Informatics
- Ida's persona
- Ida: Preparing questions
- Ida: Organizing questions and adding them into an exam
- Forum discussion to comment on scenarios
Moodle community contributors to Ida: Gary Anderson
Ida is a teacher in her fifties, still likes teaching but is more-or-less secretly already anticipating her retirement. She teaches informatics. Her teaching responsibilities include lecturing a couple of basic courses with dozens or even some hundreds of students at a time.
Exams: why and what
A lot of her time is spent planning a course of basic information retrieval to 200 first year students each semester. The exam is about:
- Evaluating learning of basic factual knowledge of her field
- Multiple choice questions
- for testing students' basic factual knowledge of her field automatically graded questions, such as multiple questions, suffice well enough (versus Mack's essay questions for testing conceptual understanding).
- Automatically graded questions are especially beneficial when students immediately get feedback on their answers and have a chance to give it a second or third try. It also significantly reduces the teacher load while giving better feedback on the progress of individuals and the whole class on topics covered.
Ida sees exams beneficial since they provide measurable grounds on which to compare and grade learning on a numerical (quantitative) scale and for quick feedback to students. She wants to have her exam as automated as possible. She does not think exams have much pedagogical purpose, and would actually rather not even grade her students, but since the educational system demands comparable grades, she does it this way.
(In an ideal world, she would instantly start using a tool that would analyze her course material automatically, generate essay questions and then even grade them using linguistic analysis or some other method.)
Harvey Historian
- Harvey's persona
- Harvey: Preparing questions
- Harvey: Organizing questions and adding them into an exam
- Forum discussion to comment on scenarios
Harvey is a history professor in a university. He mostly concentrates on his research, but also does some teaching of basic courses he knows by heart, since he has been doing it for a couple of decades.
Exams: why and what
He uses exams to test the students' understanding of specific eras from different points of view, and students' ability to see the connections between different events in history.
Sometimes Harvey also takes a year off, and the exam bank is an excellent resource for a less experienced teacher to refer to for finding out what is essential to teach about a course.
Feedback
Harvey gives brief feedback to students about what would have made a better grade. Since he has seen most things in the past already, he does not use any feedbacks for template since he has all of it pretty much in the back of his head.
Paul Pedagogue
Moodle community contributors to Paul: Jeff Forssell
Paul is almost 60 years old and is interested in all kinds of fields: science, gardening, languages, politics and music. He spends a lot of time on Internet and is delighted that it is now possible for everyone to contribute. He isn't a trained programmer, but is not afraid to dabble in code to fine tune things to the way he wants to work. Though he loves discussing directly with students, he feels that in a dynamic Internet distance course he can have very personal contact with the students. Good interactive questions can give him freedom from repetitious correcting while giving good guidance to students. He has many distance students, but still has some classroom students.
Grading hinders learning
Paul uses interactive questions, in "quizzes", primarily as a learning tool. He actually would prefer to have "official evaluation" (grading) done by some other instance, like the higher school or workplace that the student is aspiring to. This would free the teacher from the schizophrenic Helper/Judge dichotomy that prevents students and teachers from really working together in trust toward the common goal: lasting knowledge and methods for the student that will help him/er reach his/er goals. Grading in a class can also lead to diminished student interest in helping one's classmates, though this is one of the best ways to really learn something.
Questions are seeds for dialogue
Questions for learning have been the prime tool for many teachers from Socrates to Martin D! Questions usually don't have one RIGHT answer, so it is very important to allow for many alternative answers and this can make pattern matching (like "regular expressions") a high priority. Since the question is used for teaching it is important to make it provide as much dialog as possible. He will generally not like a simplified interface that doesn't have possibilities for rich feedback for varied answers. If possible he will try to give some kind of encouragement/understanding, for common mistakes as well a some clue to help the student in his processing of the problem (preferably without giving the "right answer"). His quizzes give students ways they can assess their own learning and, in a problem-based learning style, build up their understanding. If he has distance pupils, he will create many "interactive question exercise follow-ups" (using quizzes) as soon as he notices things that are common pitfalls. For an answer that isn't the "right one" his feedback will tend to be cautious: "We didn't recognize your answer as being one that we expected. Could you describe how you reached this answer?"
"Organic growth" of interactive questions
Since he sees the questions as part of a developing dialog he values any characteristics that increase the dynamics. Since he's not usually interested in "objective grading", he doesn't like the case that a "quiz" can not have new questions added once someone has "done the test". On the other hand possibilities of increasing the feedback/input of the students are exciting. Students should always be able to [comment on the questions] and ideally be able to suggest other answer alternatives (with their motivations) that could be, ideally, automatically, on teacher review, added to the questions. As a help in emphasizing that his interactive questions are a "work in progress", the student interface should be able to turn off or play down points (example: preliminary points 0.6 of 1.0).
Even if his interest in grades is minimal, a CBG, "certainty based grading", system is an exciting way of getting certainty feedback about the things that students are unsure of (in spite of giving what they think/hope is the right answer) as well as pointing out the seriousness of being very sure of something wrong. This kind of information is important for developing the teaching (including new interactive questions).
Fred
- Fred's persona
- Fred: Preparing questions
- Fred: Organizing questions and adding them into an exam
- Forum discussion to comment on scenarios
We are still missing quite a few details about Fred. If you are or know a Fred, please tell us about him.
Fred is an teacher of an advanced computer sciences course. Several other courses are prerequisites for his course.
Exams: why and what
Fred's exams are:
- At the beginning of a course,
- for evaluating the starting level of the students on his course;
- formative assessment of understanding of the meaning of basic concepts
- Multiple choice
- given periodically to assess the class's ability to grasp new concepts
- multiple choice
- free response (code response)
Fred uses an exam to evaluate the starting level for his students. Some students are given a grade for the whole course just for the exam if they do well enough.
Susan Support
- Susan's persona
- Susan: Preparing questions
- Susan: Organizing questions and adding them into an exam
- Forum discussion to comment on scenarios
Susan works in the IT support of her school.
She does introductory lectures about Moodle and other learning environments. She also tutors teachers hand-by-hand, and the drill is roughly as follows:
- Explain basic principles of Quiz
- Go through the help material
- Teach the question bank separately at first, before looking at the Quiz, it is easier to digest this way (Since Moodle 1.6)
- Make an sample exam
Teachers also send her their exams so that she would make the exam for them.
Grace Grader
- Grace's persona
- Grace: Preparing questions
- Grace: Organizing questions and adding them into an exam
- Forum discussion to comment on scenarios
Is a trainee and only does grading according to instructions.
