Goals of an online assessment system
- How it currently works
- New system overview
- Detailed design
- Question Engine 2 Developer docs:
- Implementation plan
This page tries to provide a high-level overview of what we want from the quiz module; where we are now; and where we might need to go in the future. This is mostly a personal refection by Tim Hunt, maintainer of the Moodle quiz and question bank.
While doing quiz/question development it is worth keeping the following issues in mind. Irrespective of whatever else we do, we want Moodle to have these properties. If you like, you can think of them as meta-requirements or non-functional requirements.
That list is in order of priority, most important first. Let me explain these in more detail, and summarise how I thing Moodle currently performs.
This says that, assuming nothing untoward happens, given certain inputs (question definitions, student responses, ...); and an understanding (specification) of what is supposed to happen; then the quiz module will always produce the right outputs (feedback, grades, ...). This is most important, because without this, you can't rely on anything.
I think Moodle does quite well in this regard, mainly due to many committed users who will report bugs as they are encountered, and smaller group of developers who then fix those bugs.
The previous section started, "assuming nothing untoward happens." Robustness is about what happens if something does go wrong. What if the server crashes in the middle of a test? What if the connection to the database is lost in the middle of processing one student's submission? What if the PHP process runs out of memory in the middle of a re-grade? What if an impatient user clicks submit repeatedly or closes their browser window? And so on.
In some of these cases, it is inevitable that some data will be lost. But robustness says that no matter what happens, the student's quiz attempt must be left in a consistent state (presumably either just before, or just after the last submission) and no more than the last submission's worth of data is lost.
I think that this is currently a weakness for Moodle. If the server goes down at the end of a timed test, bad things happen. See the list of bugs below.
This is about supporting a rich range of educational experiences. One of Martin's 'five laws' is that we learn when creating something for another person to see. An online assessment system can sometimes take the place of the other person. The question is, how effective a substitute can it be in how many situations?
Moodle does quite well here, with adaptive mode, plug-in-able question types and so on. However, we could do better. At the moment, writing a new question type is harder than it need be. Also, the behaviour like adaptive or non-adaptive mode is hard coded into the question engine which makes it harder to support new behaviours like certainty based marking.
Given all the above, how many simultaneous users can be supported with a particular amount of hardware?
Moodle is neither flagrantly inefficient, nor as efficient as it could be. We currently lack easy-to-perform performance measurement. We should establish some, so we can estimate current hardware requirements, and observe the effect of other developments on efficiency.
The components of an online assessment system
I think an online assessment system can be broken down into the following major components:
This is a database that stores question definitions organised into categories and contexts. With facilities to browse, select, edit, import and export the definitions.
This is the subsystem that manages the execution of the question definitions when someone attempts them. That is, it turns the static definition of the question stored in the question bank into an interactive experience.
Quiz, and other activities
This is a particular activity, that uses particular questions selected from the question bank, as executed by the question engine, to try to teach and assess students in a particular way.
Summary of where we are now
This is not the place for a full timeline of the Moodle quiz. I just make some brief observations here:
Situation as at Moodle 1.9
It is worth remembering that, as of Moodle 1.9, and indeed since several releases before that, the Moodle quiz does most of the things that most teachers require most of the time to creating quizzes that help many students learn.
Of course, that is no reason to be complacent. But we must remember that it what remains we are just talking about taking something that already works pretty well, and making it better.
Improvements in Moodle 2.0
- Greatly improved interface for students taking quizzes. It should be much easier to attempt a multi-page quiz. No more confusion about multiple submit buttons at the bottom of the page.
- Greatly improved interface for teachers creating quizzes (thanks to Olli Savolainen).
Improvements in Moodle 2.1
- The new question engine. This makes what happens when a student attempts a quiz much more robust.
- It also introduces more flexibility. There are now question behaviour plugins, that are complementary to question type plugins. That is, a question may be either a short-answer or multiple-choice question, and it may be presented to a student in either an interactive, or deferred-feebdack way.
- The new question engine also makes it much easier to write new activities that use question from the question bank, although no-one has done this yet.
- The quiz and question engine are now fully converted to use renderers, which means that themes have a lot of control over what students see when they attempt a quiz.
- The number of third-party question types that can be plugged into Moodle is gradually expanding.
A roadmap for computer-marked assessment in Moodle
Although I said above that the Moodle quiz is already pretty good, the following are, I think, the most pressing major area for further improvement. This list is in no particular order. (For example it is ordered neither by priority nor difficulty.)
Of course, irrespective of when these major items are tackled, one would expect a constant stream of minor improvements and new features.
Better user interface for creating and editing questions
Moodle 2.0 introduced a better interface for adding questions to quizzes. However, the forms for editing a question of a particular type are still very long and complex and not terribly easy to use. It would be nice to do something about this. Colin Chambers at the OU is working on this.
Colin Chambers at the OU has been doing some work on this. Here are the changes we are proposing.
Add question versioning and better sharing to the question bank
Moodle 1.9 introduced more flexibility about how teachers could share questions (and cleaned up a lot of mess in the question bank code. Thanks Jamie.) However, subsequent discussions in the quiz forum have shown that both the new system is currently to hard for most teachers and administrators to understand, but also, still not flexible enough for everything that people want to do.
I think that several of the issues about sharing questions become easier if we introduce question versioning at the same time. For example, if you take another teacher's questions, you want to be able to edit them for your own use. At the same time, if you are sharing your questions with another teacher, you don't want the other teacher's changes to mess up your quizzes, but you do want the option to get their changes back, if they are improvements. Tracking different versions of a question allow us to offer teachers control around this.
Also, when it comes to sharing, it helps to have clear licensing. Also, computers should track boring details to make things simple for humans. What the means is that the Moodle question bank should track the metadata like who owns each question and what licence it is shared under. Presumably trying to encourage use of creative commons, while not mandating it.
There are several changes in Moodle 2.0 that we should pay attention to when thinking about these changes. Community hubs allow for sharing of courses and building communities of practice. We should make sure sharing of questions fits naturally into this framework in future.
Moodle 2.0 introduced new concepts for managing files (attachments and images). If users find these concepts and user-interfaces easy to understand, then we should ensure that the concepts and interfaces in the question bank are similar.
Moodle 2.0 has a new web-service layer. If the question bank exposes the right web services, then it would be possible for people to make rich client applications for editing questions and quizzes. Since there are complex things, a rich client might provide a better interface than a web application.
Please have any requirements for this part of Moodle to this page: Future_question_bank/sharing/versioning_requirements.
Convert the Lesson module to use the standard question bank and engine
This has been on the to-do list for the lesson since Moodle 1.6 time, if not before. Now that the new question engine work is complete, I hope this will happen.
More activities using the question bank and engine
For example, a module that lets students do self-testing with questions drawn from a question bank. For example the teacher might provide a whole bank of maths questions sorted into categories, and then the student can go in at any time and ask Moodle "Please give me 20 questions that are a mixture easy addition and subtraction problems"; or "keep giving me hard multiplication problems until I tell you to stop." Moodle would present the questions one at a time. It would track the student's performance throughout the course.
There is a similar but different concept, Computerised Adaptive Testing, where Moodle presents questions one-at-a-time, varying the difficulty, in an attempt to assess the student's grade as quickly as possible. See this brief forum thread, or this interesting discussion of the strengths and weaknesses of CAT.
A richer model for which questions are in a quiz
At the moment, a quiz is just a list of questions. Perhaps the order is shuffled, and perhaps some of the questions are randomly shuffled. It is possible to imagine richer models. For example shuffle the first five questions and the last five questions, but do not mix the two halves. Also one could imagine a single quiz returning results against several different outcomes, rather than just a single score.
Forum thread about this: http://moodle.org/mod/forum/discuss.php?d=155552
Rough work breakdown for implementing this:
What needs to be done?
1. Work out what the new data-model for the quiz should be. Of course, we need to support everything Moodle currently does, and the popular feature requests. Also, I would suggest reading the T part of the IMS QTI specification to ensure that anything QTI can do will be supported by Moodle.
2. Given this new data-model, work the database structure required to store it, and how to upgrade from the current structure.
3. Work out how the quiz editing interface has to change so teachers can use all the new functionality, but, at the same time, the interface should not become any more complicated for a teacher sitting down to create their first ever Moodle quiz. That is, we will need careful use of progressive disclosure.
4. Work out how the quiz navigation, review page and quiz reports have to change to support the new, more complex data-structure.
5. Write the code and test it.
More interesting and varied question types
The list of question types available in the Moodle plugins database continues to grow. Most of the common types are now available, and some time we should have a discussion about whether some of the third-party question types should be added to the standard Moodle distribution.
Most of the interactions that people might want (drag-and-drop, markers, etc.) are now available. The place where Moodle is still quite weak is with hybrid questions that combine different types of interaction in arbitrary ways. The Cloze question type makes a start in this direction, but authoring is very hard. STACK takes another approach which is also interesting, but specialised to maths.
One thing we can say for certain with a system that is as rich as the Moodle quiz, which so many teachers are interested in pushing to its limits, is that no matter how good we make it, there will always be more ideas about how to make it even better. Looks like the developers won't ever be out of a job ;-)
- Please discuss the ideas here in the Quiz forum.