Difference between revisions of "Goals of an online assessment system"

Jump to: navigation, search
(Add question versioning and better sharing to the question bank)
(Add question versioning and better sharing to the question bank)
Line 105: Line 105:
Moodle 2.0 has a new web-service layer. If the question bank exposes the right web services, then it would be possible for people to make rich client applications for editing questions and quizzes. Since there are complex things, a rich client might provide a better interface than a web application.
Moodle 2.0 has a new web-service layer. If the question bank exposes the right web services, then it would be possible for people to make rich client applications for editing questions and quizzes. Since there are complex things, a rich client might provide a better interface than a web application.
Please have any requirements for this part of Moodle to this page: [[Future_question_bank/sharing/versioning_requirements]].
===More interesting and varied question types===
===More interesting and varied question types===

Revision as of 11:12, 19 October 2009

This page tries to provide a high-level overview of what we want from the quiz module; where we are now; and where we might need to go in the future. This is mostly a personal refection by Tim Hunt, maintainer of the Moodle quiz and question bank.

The goals

While doing quiz/question development it is worth keeping the following issues in mind. Irrespective of whatever else we do, we want Moodle to have these properties. If you like, you can think of them as meta-requirements or non-functional requirements.

That list is in order of priority, most important first. Let me explain these in more detail, and summarise how I thing Moodle currently performs.


This says that, assuming nothing untoward happens, given certain inputs (question definitions, student responses, ...); and an understanding (specification) of what is supposed to happen; then the quiz module will always produce the right outputs (feedback, grades, ...). This is most important, because without this, you can't rely on anything.

I think Moodle does quite well in this regard, mainly due to many committed users who will report bugs as they are encountered, and smaller group of developers who then fix those bugs.


The previous section started, "assuming nothing untoward happens." Robustness is about what happens if something does go wrong. What if the server crashes in the middle of a test? What if the connection to the database is lost in the middle of processing one student's submission? What if the PHP process runs out of memory in the middle of a re-grade? What if an impatient user clicks submit repeatedly or closes their browser window? And so on.

In some of these cases, it is inevitable that some data will be lost. But robustness says that no matter what happens, the student's quiz attempt must be left in a consistent state (presumably either just before, or just after the last submission) and no more than the last submission's worth of data is lost.

I think that this is currently a weakness for Moodle. If the server goes down at the end of a timed test, bad things happen. See the list of bugs below.


This is about supporting a rich range of educational experiences. One of Martin's 'five laws' is that we learn when creating something for another person to see. An online assessment system can sometimes take the place of the other person. The question is, how effective a substitute can it be in how many situations?

Moodle does quite well here, with adaptive mode, plug-in-able question types and so on. However, we could do better. At the moment, writing a new question type is harder than it need be. Also, the interaction model like adaptive or non-adaptive mode is hard coded into the question engine which makes it harder to support new interactions like certainty based marking.


Given all the above, how many simultaneous users can be supported with a particular amount of hardware?

Moodle is neither flagrantly inefficient, nor as efficient as it could be. We currently lack easy-to-perform performance measurement. We should establish some, so we can estimate current hardware requirements, and observe the effect of other developments on efficiency.

The components of an online assessment system

I think an online assessment system can be broken down into the following major components:

Question bank

This is a database that stores question definitions organised into categories and contexts. With facilities to browse, select, edit, import and export the definitions.

Question engine

This is the subsystem that manages the execution of the question definitions when someone attempts them. That is, it turns the static definition of the question stored in the question bank into an interactive experience.

Quiz, and other activities

This is a particular activity, that uses particular questions selected from the question bank, as executed by the question engine, to try to teach and assess students in a particular way.

Summary of where we are now

This is not the place for a full timeline of the Moodle quiz. I just make some brief observations here:

Situation as at Moodle 1.9

It is worth remembering that, as of Moodle 1.9, and indeed since several releases before that, the Moodle quiz does most of the things that most teachers require most of the time to creating quizzes that help many students learn.

Of course, that is no reason to be complacent. But we must remember that it what remains we are just talking about taking something that already works pretty well, and making it better.

Situation as at Moodle 2.0

The main improvements so far in Moodle 2.0 are:

  • Greatly improved interface for students taking quizzes. It should be much easier to attempt a multi-page quiz. No more confusion about multiple submit buttons at the bottom of the page.
  • Greatly improved interface for teachers creating quizzes (thanks to Olli Savolainen).

A roadmap for computer-marked assessment in Moodle

Although I said above that the Moodle quiz is already pretty good, the following are, I think, the most pressing major area for further improvement. This list is in no particular order. (For example it is ordered neither by priority nor difficulty.)

Of course, irrespective of when these major items are tackled, one would expect a constant stream of minor improvements and new features.

Rewrite of the Moodle question engine

This is this proposal: Question_Engine_2 for cleaning up some long-standing mess at the heart of the question engine; making it easier to create more activities based on the question engine; making it easier to write new question types; and making it easier to implement new interaction models like certainty based marking.

I am working on this now for inclusion into Moodle 2.0 or 2.1.

Better user interface for creating and editing questions

Moodle 2.0 introduced a better interface for adding questions to quizzes. However, the forms for editing a question of a particular type are still very long and complex and not terribly easy to use. It would be nice to do something about this.

Add question versioning and better sharing to the question bank

Moodle 1.9 introduced more flexibility about how teachers could share questions (and cleaned up a lot of mess in the question bank code. Thanks Jamie.) However, subsequent discussions in the quiz forum have shown that both the new system is currently to hard for most teachers and administrators to understand, but also, still not flexible enough for everything that people want to do.

I think that several of the issues about sharing questions become easier if we introduce question versioning at the same time. For example, if you take another teacher's questions, you want to be able to edit them for your own use. At the same time, if you are sharing your questions with another teacher, you don't want the other teacher's changes to mess up your quizzes, but you do want the option to get their changes back, if they are improvements. Tracking different versions of a question allow us to offer teachers control around this.

Also, when it comes to sharing, it helps to have clear licensing. Also, computers should track boring details to make things simple for humans. What the means is that the Moodle question bank should track the metadata like who owns each question and what licence it is shared under. Presumably trying to encourage use of creative commons, while not mandating it.

There are several changes in Moodle 2.0 that we should pay attention to when thinking about these changes. Community hubs allow for sharing of courses and building communities of practice. We should make sure sharing of questions fits naturally into this framework in future.

Moodle 2.0 introduced new concepts for managing files (attachments and images). If users find these concepts and user-interfaces easy to understand, then we should ensure that the concepts and interfaces in the question bank are similar.

Moodle 2.0 has a new web-service layer. If the question bank exposes the right web services, then it would be possible for people to make rich client applications for editing questions and quizzes. Since there are complex things, a rich client might provide a better interface than a web application.

Please have any requirements for this part of Moodle to this page: Future_question_bank/sharing/versioning_requirements.

More interesting and varied question types

Assorted drag and drop question types, question types involving images, attachments on essay questions, better handling of units and scientific notation, more sophisticated text matching, ...

Convert the Lesson module to use the standard question bank and engine

This has been on the to-do list for the lesson since Moodle 1.6 time, if not before.

More activities using the question bank and engine

For example, a module to do CAT (computerised adaptive testing) would be nice. See these forum threads: [1], [2].

Ability for per-student and per-group quiz settings

For example, allow one student or group extra time or an extra attempt. Different Open and Close dates per group, ...

A richer model for which questions are in a quiz

At the moment, a quiz is just a list of questions. Perhaps the order is shuffled, and perhaps some of the questions are randomly shuffled. It is possible to imagine richer models. For example shuffle the first five questions and the last five questions, but do not mix the two halves. Also one could imagine a single quiz returning results against several different outcomes, rather than just a single score.


One thing we can say for certain with a system that is as rich as the Moodle quiz, which so many teachers are interested in pushing to its limits, is that no matter how good we make it, there will always be more ideas about how to make it even better. Looks like I won't ever be out of a job ;-)--Tim Hunt 13:37, 11 October 2009 (UTC)

See also