Question Engine 2:Design

This page explains how I think the question engine should work in Moodle 2.0 or 2.1.

Previous section: Overview

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

New database structure

question_attempt_batches

This is just a rename of question_attempts.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id used to link attempt question data to other things, e.g. quiz_attempts.uniqueid and question_attempts.batchid
modulename	NOT NULL VARCHAR(255)	e.g. 'quiz' the thing linked to.

question_attempts

This replaces question_sessions. Question sessions is not a great name because session has other connotations in the context of web applications. I think it is right to use the question_attempt name here, because this tables has one row for each attempt at each question.

There is now no requirement for (attemptid, questionid) to be unique.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id. Linked to from question_states.attemptid.
batchid	INT(10) NOT NULL REFERENCES question_batch.id	Which attempt this data belongs to.
interactionmodel	VARCHAR(32) NOT NULL	The question interaction model that is managing this question attempt.
questionid	INT(10) NOT NULL REFERENCES question.id	Which question this is the attempt data for.
maxgrade	NUMBER(12,7) NOT NULL	The grade this question is marked out of in this attempt.
responsesummary	TEXT	This is a textual summary of the student's response (basically what you would expect to in the Quiz responses report).

• Need to store maxgrade becuase it could come from anywhere, (e.g. quiz_question_instances, question.defaultgrade, ...). We need it available at various times (e.g. when displaying a question) so it is better to store it explicitly here.

question_states

Same purpose as the old question_states table, but simplified.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id. Linked to from question_states.stateid.
attemptid	INT(10) NOT NULL REFERENCES question_attempts.id	Which question attempt this data belongs to.
timestamp	INT(10) NOT NULL	Timestamp of the event that lead to this state.
state	INT(4) NOT NULL	The type of state this is. One of the QUESTION_STATEXXX constants from the top of questionlib.php.
grade	NUMBER(12,7)	The grade the student has earned for this question, on a scale of 0..1. Needs to be multiplied by question_attempts.maxgrade to get the true grade.

• We store grade unscaled (as a value between 0.0 and 1.0) because that makes regrading easier. (You might think that you can adjust scaled grades later, and that is almost true, but if maxgrade used to be 0, then you can't change it to anything else.)

question_responses

This stores the data submitted by the student (a list of name => value pairs) that lead to the state stateid. This replaces the old question_states.answer.

There will be a convention that ordinary names like 'myvariable' should be used for submitted data belonging to the question type; names prefixed with a !, like '!myaction' should be used for data belonging to the question interaction model; and names prefixed with a _ can be used for internal things, for example, the random question might store '_realquestionid' attached to the 'open' state, or a question type that does a lot of expensive processing might store a '_cachedresult' value, so the expensive calculation does not need to be repeated when reviewing the attempt.

Note that, the old question_states.answer field used to save a lot of repetitive information from one state to the next, for example the other questionid for random questions, and the choices order for multiple-choice questions with shuffle-answers on. In future, this sort of repetitive information will not be saved. Instead, during question processing, the question types will be given access to the full state history.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id. Not used much.
stateid	INT(10) NOT NULL REFERENCES question_states.id	Which state the submission of this data lead to.
name	VARCHAR(20) NOT NULL	The name of the parameter received from the student.
value	TEXT	The value of the parameter.

New list of states that a question may be in

The aim here is to have as few states as necessary. What is necessary? To make it clear what is going on, for example in the quiz navigation. Of course, that is only one case to consider.

Incomplete

This is the state that questions start in. They stay in this state as long as the student still needs to give this question attention. In deferred feedback (non-adaptive) mode, that is until the student has entered an answer. (For a short-answer question, any answer in the input box moves you out of this state; for a matching question, you only move out of this state when you have answered all the sub-questions.) In adaptive mode, the question stays in this state until either you have got it right, or you have run out of tries.

In the state, the student can enter or change their answer.

Complete

This state is for questions where the student have done enough, but the attempt is still open, so they could change their answer if they wanted to. For example, this happens in deferred feedback mode when the student has entered a complete answer, and before they do submit all and finish. Also, a Description, after the student has seen it.

In the state, the student can enter or change their answer.

Graded(Correct/PartiallyCorrect/Incorrect)

For computer-graded questions, once the student can no longer interact with the question, it goes to one of the sub-states of the graded state.

Finished

For questions that do not have a grade, for example descriptions, after the attempt is over, they go into this state.

GaveUp

This state is used for questions where it is impossible to assign a grade because the student did submit all and finish when the question was in the incomplete state. However, this does not necessarily happen, for example, we may choose to grade an incomplete matching question if the student has completed at least one sub-question.

ManuallyGraded(Correct/PartiallyCorrect/Incorrect)

Commented

GaveUpCommented

These three states correspond the the previous three states after the teacher has added a comment and/or manually graded.

Interaction models to implement initially

It is the interaction model that controls how the student's attempt at the question moves through the state diagram in the previous section. When making the key decisions, that interaction model calls methods of the question type to supply question-type-specific information about what is happening.

Note that the main blocker to supporting negative scoring in the quiz is that it is not compatible with adaptive mode. So separating the models will allow MDL-1647 to finally be resolved.

Deferred feedback

This is how the quiz currently work when adaptive mode is off. The student enters a response to each question, then does Submit all and finish at the end of their attempt. Only then do they get feedback and/or grades, depending on the review settings.

Legacy adaptive mode

This re-implements how adaptive mode currently works, so old attempts can still be reviewed. In future, adaptive mode will be replaced with ...

Interactive

This will replace adaptive mode. This is the model that has been used successfully for several years in the OU's OpenMark system. The OU has also modified Moodle to work like this, but because of the way the quiz core currently works, I was not happy just merging the OU changes into Moodle core. In a sense, this whole document grew out of my thoughts about how to implement the OU changes in Moodle core properly.

Manually graded

This is a new mode, specifically tailored to manually graded questions. Should make it possible to fix a lot of the outstanding manual grading correctness bugs above.

Each attempt builds on last

This is for the benefit of the quiz feature of the same name. It is just like deferred feedback model, but subsequent attempts are seeded with the student's last response from the previous attempt. I think it leads to slightly cleaner code to do this as a separate interaction model. For example, it does not make any sense to use each attempt builds on last in combination with adaptive mode, although Moodle currently lets you do that.

Immediate feedback

This would be like cross between deferred feedback an interactive. Each question would have a submit button beside it, like in interactive mode, so the student can submit it immediately and get the feedback while they still remember their thought processes while answering the question. However, unlike the interactive model, they would not then be allowed to change their answer.

Certainty based marking with deferred feedback

This takes any question that can use the deferred feedback model, and adds three radio buttons to the UI (Higher, medium, lower) for the student to use to indicate how certain they are that their answer is correct. Their choice is used to alter the grade the receive for the question.

Certainty based marking with immediate feedback

This is like immediate feedback mode with the certainty based marking feature.

Delegate to remote system

This would be specific to the Opaque question type, which uses a web service link to another quetsion system to embed questions from those other system inside the Moodle quiz. The other system controls the states that the question moves through.

Changes to the question type API

Can this be backwards compatible?

Proposed robustness and performance testing system

A major change to the question engine should really only be contemplated in combination with the introduction of a test harness that makes it easy to run correctness, performance and reliablitity tests.

One adavntage of the way data will be stored in the new system is that everything originally submitted by the user will be stored in the database in a format very close to the one in which it was originally received by the web server. Therefore, it should be easy to write a script that replays saved quiz attempts. This is the basis of a test harness. I will create such a test script as part of this work.

See also

In the next section, Implementation plan outlines how I will implement this.

• Back to Question Engine 2