Goals of an online assessment system

Goals
Rationale
How it currently works
New system overview
Detailed design
Question Engine 2 Developer docs:
- Question engine overview
- Question Behaviour
- Question type development
  - Developing a New Question Type plug in
  - Converting a Moodle 2.0 question type
- Using QE in a module
Implementation plan
Testing

This page tries to provide a high-level overview of what we want from the quiz module; where we are now; and where we might need to go in the future. This is mostly a personal refection by Tim Hunt, maintainer of the Moodle quiz and question bank.

The goals

While doing quiz/question development it is worth keeping the following issues in mind. Irrespective of whatever else we do, we want Moodle to have these properties. If you like, you can think of them as meta-requirements or non-functional requirements.

That list is in order of priority, most important first. Let me explain these in more detail, and summarise how I thing Moodle currently performs.

Correctness

This says that, assuming nothing untoward happens, given certain inputs (question definitions, student responses, ...); and an understanding (specification) of what is supposed to happen; then the quiz module will always produce the right outputs (feedback, grades, ...). This is most important, because without this, you can't rely on anything.

I think Moodle does quite well in this regard, mainly due to many committed users who will report bugs as they are encountered, and smaller group of developers who then fix those bugs.

Robustness

The previous section started, "assuming nothing untoward happens." Robustness is about what happens if something does go wrong. What if the server crashes in the middle of a test? What if the connection to the database is lost in the middle of processing one student's submission? What if the PHP process runs out of memory in the middle of a re-grade? What if an impatient user clicks submit repeatedly or closes their browser window? And so on.

In some of these cases, it is inevitable that some data will be lost. But robustness says that no matter what happens, the student's quiz attempt must be left in a consistent state (presumably either just before, or just after the last submission) and no more than the last submission's worth of data is lost.

I think that this is currently a weakness for Moodle. If the server goes down at the end of a timed test, bad things happen. See the list of bugs below.

Richness

This is about supporting a rich range of educational experiences. One of Martin's 'five laws' is that we learn when creating something for another person to see. An online assessment system can sometimes take the place of the other person. The question is, how effective a substitute can it be in how many situations?

Moodle does quite well here, with adaptive mode, plug-in-able question types and so on. However, we could do better. At the moment, writing a new question type is harder than it need be. Also, the behaviour like adaptive or non-adaptive mode is hard coded into the question engine which makes it harder to support new behaviours like certainty based marking.

Efficiency

Given all the above, how many simultaneous users can be supported with a particular amount of hardware?

Moodle is neither flagrantly inefficient, nor as efficient as it could be. We currently lack easy-to-perform performance measurement. We should establish some, so we can estimate current hardware requirements, and observe the effect of other developments on efficiency.

The components of an online assessment system

I think an online assessment system can be broken down into the following major components:

Question bank

This is a database that stores question definitions organised into categories and contexts. With facilities to browse, select, edit, import and export the definitions.

Question engine

This is the subsystem that manages the execution of the question definitions when someone attempts them. That is, it turns the static definition of the question stored in the question bank into an interactive experience.

Quiz, and other activities

This is a particular activity, that uses particular questions selected from the question bank, as executed by the question engine, to try to teach and assess students in a particular way.

Summary of where we are now

This is not the place for a full timeline of the Moodle quiz. I just make some brief observations here:

Situation as at Moodle 1.9

It is worth remembering that, as of Moodle 1.9, and indeed since several releases before that, the Moodle quiz does most of the things that most teachers require most of the time to creating quizzes that help many students learn.

Of course, that is no reason to be complacent. But we must remember that it what remains we are just talking about taking something that already works pretty well, and making it better.

Improvements in Moodle 2.0

Greatly improved interface for students taking quizzes. It should be much easier to attempt a multi-page quiz. No more confusion about multiple submit buttons at the bottom of the page.
Greatly improved interface for teachers creating quizzes (thanks to Olli Savolainen).

Improvements in Moodle 2.1

The new question engine. This makes what happens when a student attempts a quiz much more robust.
- It also introduces more flexibility. There are now question behaviour plugins, that are complementary to question type plugins. That is, a question may be either a short-answer or multiple-choice question, and it may be presented to a student in either an interactive, or deferred-feebdack way.
- The new question engine also makes it much easier to write new activities that use question from the question bank, although no-one has done this yet.
The quiz and question engine are now fully converted to use renderers, which means that themes have a lot of control over what students see when they attempt a quiz.
The number of third-party question types that can be plugged into Moodle is gradually expanding.

A roadmap for computer-marked assessment in Moodle

Although I said above that the Moodle quiz is already pretty good, the following are, I think, the most pressing major area for further improvement. This list is in no particular order. (For example it is ordered neither by priority nor difficulty.)

Of course, irrespective of when these major items are tackled, one would expect a constant stream of minor improvements and new features.

Better user interface for creating and editing questions

Moodle 2.0 introduced a better interface for adding questions to quizzes. However, the forms for editing a question of a particular type are still very long and complex and not terribly easy to use. It would be nice to do something about this. Colin Chambers at the OU is working on this.

More sophisticated handling of late quiz attempts

There is a tricky issue to do with what happens exactly when time runs on on a quiz. On the one hand, we want to let students use every second of time. On the other hand, the end of a quiz is often a time of high server load, and so Moodle is likely to responding slowly. Therefore, we have to prevent students from cheating on the time-limit while still processing their last-minute submission even if the server is heavily loaded.

There is also the problem of what happens if the student just logs out and does not submit at all. Related tracker issues MDL-3030 and MDL-4309.

Here is a proposal: Better handling of overdue quiz attempts.

Add question versioning and better sharing to the question bank

Moodle 1.9 introduced more flexibility about how teachers could share questions (and cleaned up a lot of mess in the question bank code. Thanks Jamie.) However, subsequent discussions in the quiz forum have shown that both the new system is currently to hard for most teachers and administrators to understand, but also, still not flexible enough for everything that people want to do.

I think that several of the issues about sharing questions become easier if we introduce question versioning at the same time. For example, if you take another teacher's questions, you want to be able to edit them for your own use. At the same time, if you are sharing your questions with another teacher, you don't want the other teacher's changes to mess up your quizzes, but you do want the option to get their changes back, if they are improvements. Tracking different versions of a question allow us to offer teachers control around this.

Also, when it comes to sharing, it helps to have clear licensing. Also, computers should track boring details to make things simple for humans. What the means is that the Moodle question bank should track the metadata like who owns each question and what licence it is shared under. Presumably trying to encourage use of creative commons, while not mandating it.

There are several changes in Moodle 2.0 that we should pay attention to when thinking about these changes. Community hubs allow for sharing of courses and building communities of practice. We should make sure sharing of questions fits naturally into this framework in future.

Moodle 2.0 introduced new concepts for managing files (attachments and images). If users find these concepts and user-interfaces easy to understand, then we should ensure that the concepts and interfaces in the question bank are similar.

Moodle 2.0 has a new web-service layer. If the question bank exposes the right web services, then it would be possible for people to make rich client applications for editing questions and quizzes. Since there are complex things, a rich client might provide a better interface than a web application.

Please have any requirements for this part of Moodle to this page: Future_question_bank/sharing/versioning_requirements.

Convert the Lesson module to use the standard question bank and engine

This has been on the to-do list for the lesson since Moodle 1.6 time, if not before. Now that the new question engine work is complete, I hope this will happen.

A richer model for which questions are in a quiz

At the moment, a quiz is just a list of questions. Perhaps the order is shuffled, and perhaps some of the questions are randomly shuffled. It is possible to imagine richer models. For example shuffle the first five questions and the last five questions, but do not mix the two halves. Also one could imagine a single quiz returning results against several different outcomes, rather than just a single score.

Forum thread about this: http://moodle.org/mod/forum/discuss.php?d=155552

Rough work breakdown for implementing this:

What needs to be done?

1. Work out what the new data-model for the quiz should be. Of course, we need to support everything Moodle currently does, and the popular feature requests. Also, I would suggest reading the T part of the IMS QTI specification to ensure that anything QTI can do will be supported by Moodle.

2. Given this new data-model, work the database structure required to store it, and how to upgrade from the current structure.

3. Work out how the quiz editing interface has to change so teachers can use all the new functionality, but, at the same time, the interface should not become any more complicated for a teacher sitting down to create their first ever Moodle quiz. That is, we will need careful use of progressive disclosure.

4. Work out how the quiz navigation, review page and quiz reports have to change to support the new, more complex data-structure.

5. Write the code and test it.

More

One thing we can say for certain with a system that is as rich as the Moodle quiz, which so many teachers are interested in pushing to its limits, is that no matter how good we make it, there will always be more ideas about how to make it even better. Looks like the developers won't ever be out of a job ;-)

Documentation