Goals of an online assessment system

Question Engine 2

• Goals
• Rationale
• How it currently works
• New system overview
• Detailed design
• Question Engine 2 Developer docs:
  • Question engine overview
  • Question Behaviour
  • Question type development
    • Developing a New Question Type plug in
    • Converting a Moodle 2.0 question type
  • Using QE in a module
• Implementation plan
• Testing

Quiz

• Design goals
• UI overview
• Database structure
• State diagrams
• Access rules
• Quiz reports
  • Statistics report
• PHP docs
• Code in git
• X-ref-ed Code

This page tries to provide a high-level overview of what we want from the quiz module; where we are now; and where we might need to go in the future. This is mostly a personal refection by Tim Hunt, maintainer of the Moodle quiz and question bank.

The goals

While doing quiz/question development it is worth keeping the following issues in mind. Irrespective of whatever else we do, we want Moodle to have these properties. If you like, you can think of them as meta-requirements or non-functional requirements.

That list is in order of priority, most important first. Let me explain these in more detail, and summarise how I think Moodle currently performs.

Correctness

This says that, assuming nothing untoward happens, given certain inputs (question definitions, student responses, ...); and an understanding (specification) of what is supposed to happen; then the quiz module will always produce the right outputs (feedback, grades, ...). This is most important, because without this, you can't rely on anything.

I think Moodle does quite well in this regard, mainly due to many committed users who will report bugs as they are encountered, and smaller group of developers who then fix those bugs.

Robustness

The previous section started, "assuming nothing untoward happens." Robustness is about what happens if something does go wrong. What if the server crashes in the middle of a test? What if the connection to the database is lost in the middle of processing one student's submission? What if the PHP process runs out of memory in the middle of a re-grade? What if an impatient user clicks submit repeatedly or closes their browser window? And so on.

In some of these cases, it is inevitable that some data will be lost. But robustness says that no matter what happens, the student's quiz attempt must be left in a consistent state (presumably either just before, or just after the last submission) and no more than the last submission's worth of data is lost.

This has got a lot better in the last couple of years. Now all database updates are done in transactions. We have auto-save avaialble. We have ways to handle if the student forgets to submit. And so on.

Richness

This is about supporting a rich range of educational experiences. One of Martin's 'five laws' is that we learn when creating something for another person to see. An online assessment system can sometimes take the place of the other person. The question is, how effective a substitute can it be in how many situations?

Moodle does well here, particularly when you look at the avialbe add-ons like STACK or Pattern-match. The underlying structure that allows all this richness is the way the question engine combines question types and question behaviours to produce the experience the student gets when attempting a question.

Efficiency

Given all the above, how many simultaneous users can be supported with a particular amount of hardware?

Moodle is neither flagrantly inefficient, nor as efficient as it could be, but it is now pretty good. It is feasible to run tests with a few hundred users simultaneously, but this requires careful planning.

The components of an online assessment system

I think an online assessment system can be broken down into the following major components:

Question bank

This is a database that stores question definitions organised into categories and contexts. With facilities to browse, select, edit, import and export the definitions.

Question engine

This is the subsystem that manages the execution of the question definitions when someone attempts them. That is, it turns the static definition of the question stored in the question bank into an interactive experience.

Quiz, and other activities

This is a particular activity, that uses particular questions selected from the question bank, as executed by the question engine, to try to teach and assess students in a particular way.

Summary of where we are now

This is not the place for a full timeline of the Moodle quiz. I just make some brief observations here:

Situation as at Moodle 1.9

It is worth remembering that, as of Moodle 1.9, and indeed since several releases before that, the Moodle quiz does most of the things that most teachers require most of the time to creating quizzes that help many students learn.

Of course, that is no reason to be complacent. But we must remember that it what remains we are just talking about taking something that already works pretty well, and making it better.

Improvements in Moodle 2.0

• Greatly improved interface for students taking quizzes. It should be much easier to attempt a multi-page quiz. No more confusion about multiple submit buttons at the bottom of the page.
• Greatly improved interface for teachers creating quizzes (thanks to Olli Savolainen).

Improvements in Moodle 2.1

• The new question engine. This makes what happens when a student attempts a quiz much more robust.
  • It also introduces more flexibility. There are now question behaviour plugins, that are complementary to question type plugins. That is, a question may be either a short-answer or multiple-choice question, and it may be presented to a student in either an interactive, or deferred-feebdack way.
  • The new question engine also makes it much easier to write new activities that use question from the question bank, although no-one has done this yet.
• The quiz and question engine are now fully converted to use renderers, which means that themes have a lot of control over what students see when they attempt a quiz.
• The number of third-party question types that can be plugged into Moodle is gradually expanding.

A roadmap for computer-marked assessment in Moodle

Although I said above that the Moodle quiz is already pretty good, the following are, I think, the most pressing major area for further improvement. This list is in no particular order. (For example it is ordered neither by priority nor difficulty.)

Of course, irrespective of when these major items are tackled, one would expect a constant stream of minor improvements and new features.

Better user interface for creating and editing questions

Moodle 2.0 introduced a better interface for adding questions to quizzes. However, the forms for editing a question of a particular type are still very long and complex and not terribly easy to use. It would be nice to do something about this. Colin Chambers at the OU is working on this.

Colin Chambers at the OU has been doing some work on this. Here are the changes we are proposing.

Add question versioning and better sharing to the question bank

Moodle 1.9 introduced more flexibility about how teachers could share questions (and cleaned up a lot of mess in the question bank code. Thanks Jamie.) However, subsequent discussions in the quiz forum have shown that both the new system is currently to hard for most teachers and administrators to understand, but also, still not flexible enough for everything that people want to do.

I think that several of the issues about sharing questions become easier if we introduce question versioning at the same time. For example, if you take another teacher's questions, you want to be able to edit them for your own use. At the same time, if you are sharing your questions with another teacher, you don't want the other teacher's changes to mess up your quizzes, but you do want the option to get their changes back, if they are improvements. Tracking different versions of a question allow us to offer teachers control around this.

Also, when it comes to sharing, it helps to have clear licensing. Also, computers should track boring details to make things simple for humans. What the means is that the Moodle question bank should track the metadata like who owns each question and what licence it is shared under. Presumably trying to encourage use of creative commons, while not mandating it.

There are several changes in Moodle 2.0 that we should pay attention to when thinking about these changes. Community hubs allow for sharing of courses and building communities of practice. We should make sure sharing of questions fits naturally into this framework in future.

Moodle 2.0 introduced new concepts for managing files (attachments and images). If users find these concepts and user-interfaces easy to understand, then we should ensure that the concepts and interfaces in the question bank are similar.

Moodle 2.0 has a new web-service layer. If the question bank exposes the right web services, then it would be possible for people to make rich client applications for editing questions and quizzes. Since there are complex things, a rich client might provide a better interface than a web application.

Please have any requirements for this part of Moodle to this page: Future_question_bank/sharing/versioning_requirements.

Convert the Lesson module to use the standard question bank and engine

This has been on the to-do list for the lesson since Moodle 1.6 time, if not before. Now that the new question engine work is complete, I hope this will happen.

More activities using the question bank and engine

For example, a module that lets students do self-testing with questions drawn from a question bank. For example the teacher might provide a whole bank of maths questions sorted into categories, and then the student can go in at any time and ask Moodle "Please give me 20 questions that are a mixture easy addition and subtraction problems"; or "keep giving me hard multiplication problems until I tell you to stop." Moodle would present the questions one at a time. It would track the student's performance throughout the course.

There is a similar but different concept, Computerised Adaptive Testing, where Moodle presents questions one-at-a-time, varying the difficulty, in an attempt to assess the student's grade as quickly as possible. See this brief forum thread, or this interesting discussion of the strengths and weaknesses of CAT.

A richer model for which questions are in a quiz

At the moment, a quiz is just a list of questions. Perhaps the order is shuffled, and perhaps some of the questions are randomly shuffled. It is possible to imagine richer models. For example shuffle the first five questions and the last five questions, but do not mix the two halves. Also one could imagine a single quiz returning results against several different outcomes, rather than just a single score.

Forum thread about this: http://moodle.org/mod/forum/discuss.php?d=155552

Rough work breakdown for implementing this:

What needs to be done?

1. Work out what the new data-model for the quiz should be. Of course, we need to support everything Moodle currently does, and the popular feature requests. Also, I would suggest reading the T part of the IMS QTI specification to ensure that anything QTI can do will be supported by Moodle.

2. Given this new data-model, work the database structure required to store it, and how to upgrade from the current structure.

3. Work out how the quiz editing interface has to change so teachers can use all the new functionality, but, at the same time, the interface should not become any more complicated for a teacher sitting down to create their first ever Moodle quiz. That is, we will need careful use of progressive disclosure.

4. Work out how the quiz navigation, review page and quiz reports have to change to support the new, more complex data-structure.

5. Write the code and test it.

More interesting and varied question types

The list of question types available in the Moodle plugins database continues to grow. Most of the common types are now available, and some time we should have a discussion about whether some of the third-party question types should be added to the standard Moodle distribution.

Most of the interactions that people might want (drag-and-drop, markers, etc.) are now available. The place where Moodle is still quite weak is with hybrid questions that combine different types of interaction in arbitrary ways. The Cloze question type makes a start in this direction, but authoring is very hard. STACK takes another approach which is also interesting, but specialised to maths.

More

One thing we can say for certain with a system that is as rich as the Moodle quiz, which so many teachers are interested in pushing to its limits, is that no matter how good we make it, there will always be more ideas about how to make it even better. Looks like the developers won't ever be out of a job ;-)

See also

• Please discuss the ideas here in the Quiz forum.