Goals of an online assessment system
While doing quiz/question development it is worth keeping the following issues in mind. Irrespective of whatever else we do, we want Moodle to have these properties. If you like, you can think of them as meta-requirements or non-functional requirements.
That list is in order of priority, most important first. Let me explain these in more detail, and summarise how I thing Moodle currently performs.
Correctness
This says that, assuming nothing untoward happens, given certain inputs (question definitions, student responses, ...); and an understanding (specification) of what is supposed to happen; then the quiz module will always produce the right outputs (feedback, grades, ...). This is most important, because without this, you can't rely on anything.
I think Moodle does quite well in this regard, mainly due to many committed users who will report bugs as they are encountered, and smaller group of developers who then fix those bugs.
Robustness
The previous section started, "assuming nothing untoward happens." Robustness is about what happens if something does go wrong. What if the server crashes in the middle of a test? What if the connection to the database is lost in the middle of processing one student's submission? What if the PHP process runs out of memory in the middle of a re-grade? What if an impatient user clicks submit repeatedly or closes their browser window? And so on.
In some of these cases, it is inevitable that some data will be lost. But robustness says that no matter what happens, the student's quiz attempt must be left in a consistent state (presumably either just before, or just after the last submission) and no more than the last submission's worth of data is lost.
I think that this is currently a weakness for Moodle. If the server goes down at the end of a timed test, bad things happen. See the list of bugs below.
Richness
This is about supporting a rich range of educational experiences. One of Martin's 'five laws' is that we learn when creating something for another person to see. An online assessment system can sometimes take the place of the other person. The question is, how effective a substitute can it be in how many situations?
Moodle does quite well here, with adaptive mode, plug-in-able question types and so on. However, we could do better. At the moment, writing a new question type is harder than it need be. Also, the interaction model like adaptive or non-adaptive mode is hard coded into the question engine which makes it harder to support new interactions like certainty based marking.
Efficiency
Given all the above, how many simultaneous users can be supported with a particular amount of hardware?
Moodle is neither flagrantly inefficient, nor as efficient as it could be. We currently lack easy-to-perform performance measurement. We should establish some, so we can estimate current hardware requirements, and observe the effect of other developments on efficiency.
