Question Engine 2:Overview

This page outlines how I think the question engine should work in Moodle 2.0 or 2.1.

Previous section: Rationale

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Clarifying question states

Althought there is a database table called question_states, the column there that stores the type of state is called event, and the values stored there are active verbs like open, save, grade. The question_process_responses function is written in terms of these actions too. One nasty part is that during the processing, the event type can be changed several times, before it is finally stored in the database as a state.

I would like to clearly separate the concept of the current state that a question is in, and the various actions that lead to a change of state.

(+Correctness, +Richness).

Question interaction models

At the moment, the various sequences of states that a question can move through is controlled by a combination of the compare_responses and grade_responses methods of the question types, and hard-coded logic in the question_process_responses function.

I would like to separate out the control of how a question moves through different states, into what I will call question interaction models. Currently there are two models supported by Moodle, and the choice is controlled by the adaptive mode setting in the quiz.

There is the deferred feedback model, which is what you get when adaptive mode is off. The student enters an answer to each question which is saved. Then, when they submit the quiz attempt, all the questions are graded. Then all the marks and feedback are available for review.

Alternatively, there is the adaptive model, where the student can submit one question at a time, get some feedback, and if they were wrong, immediately change their answer, with a reduced grade if they get it right the second time.

Currently, manually graded question use the same adaptive, or non-adaptive, models, with some slight changes. However, this does not work very well, as the number of essay question correctness bugs listed above shows. Really, manual grading should be treated as a separate interaction model.

So, what exactly does the interaction model control? Given the current state of the question, it limits the range of actions that are possible. For example, in adaptive mode, there is a submit button next to each question. In non-adaptive mode, the only option is to enter an answer and have it saved, until the submit all and finish button is pressed. That is, it rougly replaces question_extract_responses, question_process_responses, question_process_comment and save_question_state in the current code.

(+++Richness, ++Correctness, +Robustness)

Simplified API for question types

Although I said above "The student enters an answer to each question which is saved. Then when they submit the quiz attempt, all the questions are graded." In fact, that was a lie. Whenever a response is saved, the grade_responses method of the question type is called, even in the state is only being saved. This is confusing, to say the least, and very bad for performance in the case of a question type like JUnit (in contrib) which takes code submitted by the student, and compiles it in order to grade it.

So some of the API will only change to the extent that certian funcitons will in future only be called when one would expect them to be. I think this can be done in a backwards-compatible way.

Another change will be that, at the moment, question types have to implement tricky load/save_question_state methods that, basically, have to unserialise/serialise some of the state data in a custom way, so it can be stored in the answer column. This is silly design, and leads to extra, unnecessary database queries. By changing the database structure so that the students' responses are stored directly in a new table (after all, an HTTP POST request is really just an array of string key-value pairs, and there is a natural way to store that in a database) it should be posiblity to eliminate the need for these methods.

(+Richness, +Correctness, +Efficiency)

Changed database structure

At the moment, the some parts of the question_sessions and question_states tables and their relations are not normalised. This is a technical term for describing database designs. Basically, if the tables are normalised, then each 'fact' is stored in exactly one place. This makes it much less likely that the database can get into an inconsistent state. Changing to a normalised structure should therefore increase robustness.

In addition, I wish to change the tables, so the the responses recieved from the student are stored in a much more 'raw' form. That will mean that the responses can be saved much earlier in the sequence of processing, which will again increase robustness. It will also allow the sequence of responses from the student to be replayed more easily, making it easier to continue after a server crash, to regrade, and to write automatic test scripts.

(+++Robustness, +Correctness)

Reorganise the code

Already, in the Moodle 2.0 developments, I have made several changes to the how the quiz and question bank code is organised, in an attempt to make it easier to maintain and understand. It also makes it easier to write automated tests. I hope that this work will have a further beneficial effect on the code quality.

(+Robustness, +Efficiency)

Brief digression: who controls the model, the quiz or the question?

Before proceeding to decribe in detail the design for how I will implement these changes, I want to digress and discuss one of the design decisions that it took a long time for me to resolve.

The question is, is the interaction model a property of the question, or the quiz?

At the moment, there is a setting in the quiz that lets you change all the questions in the quiz from adaptive mode to non-adptive mode at the flick of a switch. Or, at least, it allows you to re-use the same questions in both adaptive and non-adaptive mode. This suggest that the interaction model is a property of the quiz.

On the other hand, suppose that you wanted to introduce the feature that, in adaptive mode, the student gets different amounts of feedback after each try at getting the question right. For example, the first time they get it wrong the only get a brief hint. If they are wrong the second time they get a more detailed comment, and so on. In order to do this, you need more data (the various hints) in order to define an adaptive question that is irrelevant to a non-adaptive question. Also, certain question types (e.g. Opaque, from contrib) have to, of necessity, ignore the adaptive, non-adaptive setting. And I suggested above that, manually graded question types like Essay should really be considered to have a separate interaction model. This suggests that the interaction model is a properly of the individual question. (Although usability considerations suggest that a single quiz should probably be constructed from questions with similar interaction models.)

I evenutally concluded that both answers are right. That is, it is a good idea for the quiz to have a setting like adaptive/non-adaptive, which states the intention of how each question in this quiz should behave. However, the exact choice of interaction model to use is up to the different quetsion types. When a quiz attempts is started, and told "A quiz attempt is starting using 'adaptive mode'. Exactly which interaction model should be used for questions of this type in this attempt?" This gives questions that can only work in a certain way (for example essay questions) a chance to override the quiz setting.

Documentation