Question Engine 2:Design

This page explains how I think the question engine should work in Moodle 2.0 or 2.1.

Previous section: Overview

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

New database structure

question_attempts_groups

This is just a rename of question_attempts.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id used to link attempt question data to other things, e.g. quiz_attempts.uniqueid and question_attempts.batchid
modulename	NOT NULL VARCHAR(255)	e.g. 'quiz' the thing linked to.

question_attempts

This replaces question_sessions. Question sessions is not a great name because session has other connotations in the context of web applications. I think it is right to use the question_attempt name here, because this tables has one row for each attempt at each question.

There is now no requirement for (attemptid, questionid) to be unique.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id. Linked to from question_states.attemptid.
batchid	INT(10) NOT NULL REFERENCES question_batch.id	Which attempt this data belongs to.
interactionmodel	VARCHAR(32) NOT NULL	The question interaction model that is managing this question attempt.
questionid	INT(10) NOT NULL REFERENCES question.id	Which question this is the attempt data for.
maxgrade	NUMBER(12,7) NOT NULL	The grade this question is marked out of in this attempt.
responsesummary	TEXT	This is a textual summary of the student's response (basically what you would expect to in the Quiz responses report).

• Need to store maxgrade becuase it could come from anywhere, (e.g. quiz_question_instances, question.defaultgrade, ...). We need it available at various times (e.g. when displaying a question) so it is better to store it explicitly here.

question_states

Same purpose as the old question_states table, but simplified.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id. Linked to from question_states.stateid.
attemptid	INT(10) NOT NULL REFERENCES question_attempts.id	Which question attempt this data belongs to.
timestamp	INT(10) NOT NULL	Timestamp of the event that lead to this state.
state	INT(4) NOT NULL	The type of state this is. One of the QUESTION_STATEXXX constants from the top of questionlib.php.
grade	NUMBER(12,7)	The grade the student has earned for this question, on a scale of 0..1. Needs to be multiplied by question_attempts.maxgrade to get the true grade.

• We store grade unscaled (as a value between 0.0 and 1.0) because that makes regrading easier. (You might think that you can adjust scaled grades later, and that is almost true, but if maxgrade used to be 0, then you can't change it to anything else.)

question_responses

This stores the data submitted by the student (a list of name => value pairs) that lead to the state stateid. This replaces the old question_states.answer.

There will be a convention that ordinary names like 'myvariable' should be used for submitted data belonging to the question type; names prefixed with a !, like '!myaction' should be used for data belonging to the question interaction model; and names prefixed with a _ can be used for internal things, for example, the random question might store '_realquestionid' attached to the 'open' state, or a question type that does a lot of expensive processing might store a '_cachedresult' value, so the expensive calculation does not need to be repeated when reviewing the attempt.

Note that, the old question_states.answer field used to save a lot of repetitive information from one state to the next, for example the other questionid for random questions, and the choices order for multiple-choice questions with shuffle-answers on. In future, this sort of repetitive information will not be saved. Instead, during question processing, the question types will be given access to the full state history.

Column	Type	Comment
id	INT(10) NOT NULL AUTO INCREMENT	Unique id. Not used much.
stateid	INT(10) NOT NULL REFERENCES question_states.id	Which state the submission of this data lead to.
name	VARCHAR(20) NOT NULL	The name of the parameter received from the student.
value	TEXT	The value of the parameter.

Upgrading the database

TODO

New list of states that a question may be in

The aim here is to have as few states as necessary. What is necessary? To make it clear what is going on, for example in the quiz navigation. Of course, that is only one case to consider.

Incomplete

This is the state that questions start in. They stay in this state as long as the student still needs to give this question attention. In deferred feedback (non-adaptive) mode, that is until the student has entered an answer. (For a short-answer question, any answer in the input box moves you out of this state; for a matching question, you only move out of this state when you have answered all the sub-questions.) In adaptive mode, the question stays in this state until either you have got it right, or you have run out of tries.

In the state, the student can enter or change their answer.

Complete

This state is for questions where the student have done enough, but the attempt is still open, so they could change their answer if they wanted to. For example, this happens in deferred feedback mode when the student has entered a complete answer, and before they do submit all and finish. Also, a Description, after the student has seen it.

In the state, the student can enter or change their answer.

Graded(Correct/PartiallyCorrect/Incorrect)

For computer-graded questions, once the student can no longer interact with the question, it goes to one of the sub-states of the graded state.

Finished

For questions that do not have a grade, for example descriptions, after the attempt is over, they go into this state.

GaveUp

This state is used for questions where it is impossible to assign a grade because the student did submit all and finish when the question was in the incomplete state. However, this does not necessarily happen, for example, we may choose to grade an incomplete matching question if the student has completed at least one sub-question.

ManuallyGraded(Correct/PartiallyCorrect/Incorrect)

Commented

GaveUpCommented

These three states correspond the the previous three states after the teacher has added a comment and/or manually graded.

API for modules using the question engine

Here is some proposed code from an integration test method. It creates an attempt containing one true/false question and walks through a student getting it right, and then at teacher overriding the grade.

   public function test_delayed_feedback_truefalse() {
       // Create a true-false question with correct answer true.
       $tf = $this->make_a_truefalse_question();

       // Start a delayed feedback attempt and add the question to it.
       $qag = question_engine::make_question_attempts_group();
       $qag = set_preferred_interaction_model('delayedfeedback');
       $questionattemptid = $qag->add_question($tf);
       // Different from $tf->id since the same question may be used twice in
       // the same attempt.
       $qaid = $questionattemptid; // Abbreviate.

       // Verify.
       $this->assertEqual($qag->question_count(), 1);

       // Begin the attempt. Creates an initial state for each question.
       $qag->start_all_questions();

       // Output the question in the initial state.
       $html = $qag->render_question($qaid);

       // Verify.
       $this->assertEqual($qag->get_question_state($qaid), QUESTION_STATE_INCOMPLETE);
       $this->assertNull($qag->get_question_grade($qaid));
       $this->assertPattern('/' . preg_escape($tf->questiontext) . '/', $html);

       // Simulate some data submitted by the student.
       $prefix = $qag->get_field_prefix($qaid);
       $answername = $prefix . 'true';
       $getdata = array(
           $answername => 1,
       );
       $response = $qag->extract_responses($qaid, $getdata);

       // Verify.
       $this->assertEqual(array('true' => 1), $response);

       // Process the data extracted for this question.
       $qag->process_response($qaid, $response);
       $html = $qag->render_question($qaid);

       // Verify.
       $this->assertEqual($qag->get_question_state($qaid), QUESTION_STATE_COMPLETE);
       $this->assertNull($qag->get_question_grade($qaid));
       $this->assert(new ContainsTagWithAttributes('input',
               array('name' => $answername, 'value' => 1)), $html);
       $this->assertNoPattern('/class=\"correctness/', $html);

       // Finish the attempt.
       $qag->finish_all_questions();
       $html = $qag->render_question($qaid);

       // Verify.
       $this->assertEqual($qag->get_question_state($qaid), QUESTION_STATE_GRADED_CORRECT);
       $this->assertEqual($qag->get_question_grade($qaid), 1);
       $this->assertPattern(
               '/' . preg_escape(get_string('correct', 'question')) . '/',
               $html);

       // Process a manual comment.
       $qag->process_manual_grade($qaid, 0.5, 'Not good enough!');
       $html = $qag->render_question($qaid);

       // Verify.
       $this->assertEqual($qag->get_question_state($qaid), QUESTION_STATE_MANUALLYGRADED_PARTIALLYCORRECT);
       $this->assertEqual($qag->get_question_grade($qaid), 0.5);
       $this->assertPattern('/' . preg_escape('Not good enough!') . '/', $html);
   }

Note that this code does not interact with the database at all. Data is only stored to or loaded from the database if you call $qag->load_... or $qag_save... methods.

New classes

question_interaction_model_base

Will have subclasses like

• question_delayedfeedback_model
• question_interactive_model
• ...

qtype_renderer_base

Will have subclasses like

• qtype_truefalse_renderer
• qtype_multichoice_renderer - and possibly also qtype_multichoice_horizontal_renderer
• ...

qim_renderer_base

Will have subclasses like

• qim_delayedfeedback_renderer
• qim_interactive_renderer
• ...

Changes to the question type API

Can this be backwards compatible?

Proposed robustness and performance testing system

A major change to the question engine should really only be contemplated in combination with the introduction of a test harness that makes it easy to run correctness, performance and reliablitity tests.

One adavntage of the way data will be stored in the new system is that everything originally submitted by the user will be stored in the database in a format very close to the one in which it was originally received by the web server. Therefore, it should be easy to write a script that replays saved quiz attempts. This is the basis of a test harness. I will create such a test script as part of this work.

See also

In the next section, Implementation plan outlines how I will implement this.

• Back to Question Engine 2