MoodleDocs - User contributions [en]

Development:Changing the Moodle Question Engine

2009-03-16T20:20:37Z

Dooug: /* Code flow */

__TOC__

The Question Engine is the part of the question code in Moodle that deals with what happens when a question is attempted by a Student, or later reviewed by the Teacher or the student themselves or included in reports. Roughly speaking this is half the question code, the other part being the Question Bank, which allows people to create, edit, organise, import, export and share questions. The Question Engine is the part that has to perform when you have a large class of students taking a quiz simultaneously.

[[Image:Sydney_harbour.jpg|frame|An irrelevant but pretty picture of Sydney, taken from the hotel room where most of this was written.]]This document is a by-product of three years working on the Moodle quiz and question code. During that time I have fixed hundreds of bugs and feature requests. Some of the bugs with the Question Engine have requried careful fixing in order to avoid regressions, and there are other issues that are hard to do anything about at the moment.

As well as bug fixes, I have also overseen major developments in the quiz/question system. For example Jamie's changes to the Question Bank, Ollie's new quiz editing interface, and Mahmoud's new quiz navigation that I added to Core Moodle while refactoring the quiz attempt code.

Over the three years, I have also followed the Moodle quiz forum and seen what it is that people struggle with, complain about, want to change (and also what they like!). For example, having coached some people when writing new question types I realise that creating a new questions type is more difficult than it needs to be.

The changes proposed here fix parts of the question engine code that I have been dissatisfied with since about the first time I looked at it. There are several reasons why I have not addressed them before now.

# To fix them properly, we must change how the data is stored in some of the key question tables. In some sites, these tables contain millions of rows, and the processing required on update is non-trivial and is different for different question types. This is not something to undertake lightly.
# It took me about two and a half years to work out this solution. Some parts of this proposal where clear from the start, but the exact notion of question interaction models, and how they would relate to question types, only crystallised recently in my mind. Because of 1. I did not want to change anything until I was sure what I was proposing was right, not just in general, but in all the details.
# The mundane reason that there have been plenty of other things to work on, and I have not had time to address this until now.
# These are not critical problems. There have been no major changes to the question engine since the 1.6 release. Clearly there are not fundamental flaws with the current system. It is just not as good as it could be.

As this is a major change, I will spend some time justifying it, before describing the change itself. My estimate is that doing what I propose here is a couple of month's solid work.

==Goals of an online assessment system==

Before going further, I think it is worth being clear about what properties we require of a question engine. In my view, we want:
# Correctness
# Robustness
# Richness
# Efficiency

Let me explain in more detail, and summarise how I thing Moodle currently performs.

===1. Correctness===

This says that, assuming nothing untoward happens, given certain inputs (question definitions, student responses, ...); and an understanding (specification) of what is supposed to happen; then the quiz module will always produce the right outputs (feedback, grades, ...). This is most important, because without this, you can't rely on anything.

I think Moodle does quite well in this regard, mainly due to many committed users who will report bugs as they are encountered, and smaller group of developers who then fix those bugs.

===2. Robustness===

The previous section started, "assuming nothing untoward happens." Robustness is about what happens if something does go wrong. What if the server crashes in the middle of a test? What if the connection to the database is lost in the middle of processing one student's submission? What if the PHP process runs out of memory in the middle of a re-grade? What if an impatient user clicks submit repeatedly or closes their browser window? And so on.

In some of these cases, it is inevitable that some data will be lost. But robustness says that no matter what happens, the student's quiz attempt must be left in a consistent state (presumably either just before, or just after the last submission) and no more than the last submission's worth of data is lost.

I think that this is currently a weakness for Moodle. If the server goes down at the end of a timed test, bad things happen. See the list of bugs below.

===3. Richness===

This is about supporting a rich range of educational experiences. One of Martin's 'five laws' is that we learn when creating something for another person to see. An online assessment system can sometimes take the place of the other person. The question is, how effective a substitute can it be in how many situations?

Moodle does quite well here, with adaptive mode, plug-in-able question types and so on. However, we could do better. At the moment, writing a new question type is harder than it need be. Also, the interaction model like adaptive or non-adaptive mode is hard coded into the question engine which makes it harder to support new interactions like certainty based marking.

===4. Efficiency===

Given all the above, how many simultaneous users can be supported with a particular amount of hardware?

Moodle is neither flagrantly inefficient, nor as efficient as it could be. We currently lack easy-to-perform performance measurement. We should establish some, so we can estimate current hardware requirements, and observe the effect of other developments on efficiency.

==Why things need to change==

I have just summarised how well Moodle meets my four goals. I will now presents a different picture, by listing bugs and feature requests in the Moodle tracker that I think can best be resolved by the changes I propose.

===Correctness issues===

{| class="nicetable"
! Key !! Votes !! Watchers !! Summary
|-
| MDL-3030 || 16 || 7 || attempts should be auto-closed
|-
| MDL-3936 || 2 || 1 || Essay Question used in score before grading
|-
| MDL-9303 || 2 || 5 || General Feedback not shown for essay questions on review screen
|-
| MDL-9327 || 4 || 3 || Text in Essay type questions is not saved in second or more attempts when the teacher has not graded previous attempts
|-
| MDL-13289 || 0 || 0 || Essay question with adaptive mode not show feedback
|-
| MDL-17681 || 0 || 0 || Manually assigned grades cannot easily by re-scaled when the marks for a question is changed
|}

===Robustness issues===

{| class="nicetable"
! Key !! Votes !! Watchers !! Summary
|-
| MDL-11852 || 0 || 1 || Quiz does not score properly
|-
| MDL-12344 || 0 || 0 || Student takes quiz, upon completion of quiz receives the following
|-
| MDL-12665 || 0 || 0 || error: Must Specify course id, shorname, or idnumber quiz attempts sometimes show zero despite correct answers
|-
| MDL-13073 || 0 || 0 || Regrading (really) messes up grades
|-
| MDL-13246 || 0 || 1 || Quiz does not save score. Shows 0/20
|-
| MDL-14873 || 0 || 2 || quiz answers submit freezes (occationally)
|-
| MDL-15305 || 1 || 0 || Quiz doesn't work after submit all and finish button is selected - get a "No quiz with id=0" or "No attempt with id=0" error message
|-
| MDL-16451 || 0 || 1 || Quiz scores 0 with secure window; Grades do not transfer
|-
| MDL-16490 || 0 || 1 || timed quiz generates bad 2nd attempt
|-
| MDL-16663 || 0 || 1 || errors and missing work
|}

(Many of these are intermittent and only affect some people. Sometimes bugs like this turn out to be the result of a corrupt table in MySQL. Still, we really should not be getting problems like this with a system that is sometimes relied on as much as the quiz is.)

===Richness issues===

{| class="nicetable"
! Key !! Votes !! Watchers !! Summary
|-
| MDL-1647 || 4 || 6 || Allowing negative marks for questions
|-
|| MDL-4860 || 0 || 0 || Questiontype giving hints
|-
| MDL-12821 || 0 || 1 || Essay quiz question redundant submit button
|-
| MDL-17894 || 0 || 0 || Support certainty/Confidence based marking
|-
| MDL-17895 || 0 || 0 || Allow question types to validate responses before grading
|-
| MDL-17896 || 0 || 0 || Question types like Opaque should not require nasty hacks in question_process_responses
|}

===Issues I don't propose to fix with this change===

... but which would become easier to fix as a result:

{| class="nicetable"
! Key !! Votes !! Watchers !! Summary
|-
| MDL-3452 || 0 || 0 || Use saved answers when marking late submissions
|-
| MDL-4309 || 0 || 3 || Ability for late quizzes (with penalty)
|-
| MDL-15596 || 0 || 0 || Allow a question to be used more than once in an attempt
|}

==How this part of the quiz currently works==

Before saying what I would like to change, I think it is best to summarise how the question engine currently works.

===Database tables===

There are three database tables that store data about students' attempts at questions.

====question_attempts====

This table is basically used to generate unique ids for the other tables relating to question attempts, and allow one to link the data in those tables to the other parts of Moodle that use the question bank, for example from a quiz attempt.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id used to link attempt question data to other things, e.g. quiz_attempts.uniqueid
|-
| modulename || NOT NULL VARCHAR(20) || e.g. 'quiz' the thing linked to.
|}

====question_sessions====

There is one row in this table for each question in an attempt.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id. Not used much.
|-
| attemptid || INT(10) NOT NULL REFERENCES question_attempts.id || Which attempt this data belongs to.
|-
| questionid || INT(10) NOT NULL REFERENCES question.id || Which question this is the attempt data for.
|-
| newest || INT(10) NOT NULL REFERENCES question_states.id || The latest state of this question in this attempt.
|-
| newgraded || INT(10) NOT NULL REFERENCES question_states.id || The latest graded state of this question in this attempt.
|-
| sumpenalty || NUMBER(12,7) NOT NULL || Used for adaptive mode scoring.
|-
| manualcomment || TEXT || A teacher's manual comment.
|}

(attemptid, questionid) is an alternate key, but that makes MDL-15596 impossible to satisfy.

====question_states====

There are several rows here for each question session, recording the different states that the question went through as the student attempted it. For example open, save, submit, manual_grade.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id. Not used much.
|-
| attempt || INT(10) NOT NULL REFERENCES question_attempts.id || Which attempt this data belongs to.
|-
| question || INT(10) NOT NULL REFERENCES question.id || Which question this is the attempt data for.
|-
| seq_number || INT(10) NOT NULL || Numbers the states within this question attempt.
|-
| answer || TEXT NOT NULL || A representation of the student's response.
|-
| timestamp || INT(10) NOT NULL || Timestamp of the event that lead to this state.
|-
| event || INT(4) NOT NULL || The type of state this is. One of the QUESTION_EVENTXXX constants from the top of questionlib.php.
|-
| grade || NUMBER(12,7) NOT NULL || The grade that had been earned by this point, after penalties had been taken into account.
|-
| raw_grade || NUMBER(12,7) NOT NULL || The grade that had been earned by this point, before penalties had been taken into account.
|-
| penalty || NUMBER(12,7) NOT NULL || Used by the adaptive mode scoring.
|}

(attempt, question, seq_number) is an alternate key (see the remark in the previous sub-section).

The student's response is stored in the answer column. It is up to each question type to convert the data received from the student into a single string in whatever format seems best, even though the response come from a HTTP post, and so is just an array of key => value pairs, which could easily be stored directly in the database without each question type having to write some nasty string concatenation/parsing code.

===Code flow===

A part of Moodle, for example the Quiz module, wishes to use questions. First, mod/quiz/attempt.php will render a page of questions by calling
# load_questions,
# load_question_states, and
# print_question one or more times.
All three of these functions delegate part of the work to the appropriate question type.

This outputs each question in its current state. Any form controls that are output have a name="" attribute that is a combination of a prefix determined by the question engine (e.q. q123_), and a main part determined by the question type.

When this page is submitted, the response goes to whichever script the question using code designate. In the case of the quiz it is mod/quiz/processattemptattempt.php. This calls
# load_questions,
# load_question_states,
# question_extract_responses,
# question_process_responses, and
# save_question_state.
# In the case of the quiz, it then saves the quiz_attempts row, which has been updated by some of the above methods.
Again, steps 1., 2., 4. and 5. delegate part of the work to the question types.

Processing manual grading by the teacher is similar. The sequence of calls is:
# load_questions,
# load_question_states,
# question_process_comment, and
# save_question_state.
# In the case of the quiz, it then saves the quiz_attempts row, which has been updated by some of the above methods.

One thing to note here is that a lot of processing is done before any data can be stored in the database.

Another point to note is that the question type classes are singleton classes. That is, only one instance of each question type is created. All the methods used by the question engine are passed a $question and possibly a $state object to tell the question type which question instance they should be processing. This is good design from several points of view. It works well with batch database queries (efficiency) and the question types to not have state, which might become inconsistent (robustness). However, there is a slight price here, since very rich question types cannot cache state in memory, so they may have to repeat processing (richness <-> efficiency trade off) however, they can store state in the database since they can put whatever they like in the question_states.answer field.

==Overview of the changes==

===Clarifying question states===

Althought there is a database table called question_states, the column there that stores the type of state is called event, and the values stored there are active verbs like open, save, grade. The question_process_responses function is written in terms of these actions too. One nasty part is that during the processing, the event type can be changed several times, before it is finally stored in the database as a state.

I would like to clearly separate the concept of the current state that a question is in, and the various actions that lead to a change of state.

(+Correctness, +Richness).

===Question interaction models===

At the moment, the various sequences of states that a question can move through is controlled by a combination of the compare_responses and grade_responses methods of the question types, and hard-coded logic in the question_process_responses function.

I would like to separate out the control of how a question moves through different states, into what I will call question interaction models. Currently there are two models supported by Moodle, and the choice is controlled by the adaptive mode setting in the quiz.

There is the deferred feedback model, which is what you get when adaptive mode is off. The student enters an answer to each question which is saved. Then, when they submit the quiz attempt, all the questions are graded. Then all the marks and feedback are available for review.

Alternatively, there is the adaptive model, where the student can submit one question at a time, get some feedback, and if they were wrong, immediately change their answer, with a reduced grade if they get it right the second time.

Currently, manually graded question use the same adaptive, or non-adaptive, models, with some slight changes. However, this does not work very well, as the number of essay question correctness bugs listed above shows. Really, manual grading should be treated as a separate interaction model.

So, what exactly does the interaction model control? Given the current state of the question, it limits the range of actions that are possible. For example, in adaptive mode, there is a submit button next to each question. In non-adaptive mode, the only option is to enter an answer and have it saved, until the submit all and finish button is pressed. That is, it rougly replaces question_extract_responses, question_process_responses, question_process_comment and save_question_state in the current code.

(+++Richness, ++Correctness, +Robustness)

===Simplified API for question types===

Although I said above "The student enters an answer to each question which is saved. Then when they submit the quiz attempt, all the questions are graded." In fact, that was a lie. Whenever a response is saved, the grade_responses method of the question type is called, even in the state is only being saved. This is confusing, to say the least, and very bad for performance in the case of a question type like JUnit (in contrib) which takes code submitted by the student, and compiles it in order to grade it.

So some of the API will only change to the extent that certian funcitons will in future only be called when one would expect them to be. I think this can be done in a backwards-compatible way.

Another change will be that, at the moment, question types have to implement tricky load/save_question_state methods that, basically, have to unserialise/serialise some of the state data in a custom way, so it can be stored in the answer column. This is silly design, and leads to extra, unnecessary database queries. By changing the database structure so that the students' responses are stored directly in a new table (after all, an HTTP POST request is really just an array of string key-value pairs, and there is a natural way to store that in a database) it should be posiblity to eliminate the need for these methods.

(+Richness, +Correctness, +Efficiency)

===Changed database structure===

At the moment, the some parts of the question_sessions and question_states tables and their relations are not normalised. This is a technical term for describing database designs. Basically, if the tables are normalised, then each 'fact' is stored in exactly one place. This makes it much less likely that the database can get into an inconsistent state. Changing to a normalised structure should therefore increase robustness.

In addition, I wish to change the tables, so the the responses recieved from the student are stored in a much more 'raw' form. That will mean that the responses can be saved much earlier in the sequence of processing, which will again increase robustness. It will also allow the sequence of responses from the student to be replayed more easily, making it easier to continue after a server crash, to regrade, and to write automatic test scripts.

(+++Robustness, +Correctness)

==Brief digression: who controls the model, the quiz or the question?==

Before proceeding to decribe in detail the design for how I will implement these changes, I want to digress and discuss one of the design decisions that it took a long time for me to resolve.

The question is, is the interaction model a property of the question, or the quiz?

At the moment, there is a setting in the quiz that lets you change all the questions in the quiz from adaptive mode to non-adptive mode at the flick of a switch. Or, at least, it allows you to re-use the same questions in both adaptive and non-adaptive mode. This suggest that the interaction model is a property of the quiz.

On the other hand, suppose that you wanted to introduce the feature that, in adaptive mode, the student gets different amounts of feedback after each try at getting the question right. For example, the first time they get it wrong the only get a brief hint. If they are wrong the second time they get a more detailed comment, and so on. In order to do this, you need more data (the various hints) in order to define an adaptive question that is irrelevant to a non-adaptive question. Also, certain question types (e.g. Opaque, from contrib) have to, of necessity, ignore the adaptive, non-adaptive setting. And I suggested above that, manually graded question types like Essay should really be considered to have a separate interaction model. This suggests that the interaction model is a properly of the individual question. (Although usability considerations suggest that a single quiz should probably be constructed from questions with similar interaction models.)

I evenutally concluded that both answers are right, and more importantly, worked out how a question engine could allow that to happen.

-----

{{Work in progress}}

''TODO finish the rest of this document.''

==Detailed design==

===New database structure===

====question_attempt_batches====

This is just a rename of question_attempts.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id used to link attempt question data to other things, e.g. quiz_attempts.uniqueid and question_attempts.batchid
|-
| modulename || NOT NULL VARCHAR(255) || e.g. 'quiz' the thing linked to.
|}

====question_attempts====

This replaces question_sessions. Question sessions is not a great name because session has other connotations in the context of web applications. I think it is right to use the question_attempt name here, because this tables has one row for each attempt at each question.

There is now no requirement for (attemptid, questionid) to be unique.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id. Linked to from question_states.attemptid.
|-
| batchid || INT(10) NOT NULL REFERENCES question_batch.id || Which attempt this data belongs to.
|-
| interactionmodel || VARCHAR(32) NOT NULL || The question interaction model that is managing this question attempt.
|-
| questionid || INT(10) NOT NULL REFERENCES question.id || Which question this is the attempt data for.
|-
| maxgrade || NUMBER(12,7) NOT NULL || The grade this question is marked out of in this attempt.
|-
| responsesummary || TEXT || This is a textual summary of the student's response (basically what you would expect to in the Quiz responses report).
|}

* Need to store maxgrade becuase it could come from anywhere, (e.g. quiz_question_instances, question.defaultgrade, ...). We need it available at various times (e.g. when displaying a question) so it is better to store it explicitly here.

====question_states====

Same purpose as the old question_states table, but simplified.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id. Linked to from question_states.stateid.
|-
| attemptid || INT(10) NOT NULL REFERENCES question_attempts.id || Which question attempt this data belongs to.
|-
| timestamp || INT(10) NOT NULL || Timestamp of the event that lead to this state.
|-
| state || INT(4) NOT NULL || The type of state this is. One of the QUESTION_STATEXXX constants from the top of questionlib.php.
|-
| grade || NUMBER(12,7) || The grade the student has earned for this question, on a scale of 0..1. Needs to be multiplied by question_attempts.maxgrade to get the true grade.
|}

* We store grade unscaled (as a value between 0.0 and 1.0) because that makes regrading easier. (You might think that you can adjust scaled grades later, and that is almost true, but if maxgrade used to be 0, then you can't change it to anything else.)

====question_responses====

This stores the data submitted by the student (a list of name => value pairs) that lead to the state stateid. This replaces the old question_states.answer.

There will be a convention that ordinary names like 'myvariable' should be used for submitted data belonging to the question type; names prefixed with a !, like '!myaction' should be used for data belonging to the question interaction model; and names prefixed with a _ can be used for internal things, for example, the random question might store '_realquestionid' attached to the 'open' state, or a question type that does a lot of expensive processing might store a '_cachedresult' value, so the expensive calculation does not need to be repeated when reviewing the attempt.

Note that, the old question_states.answer field used to save a lot of repetitive information from one state to the next, for example the other questionid for random questions, and the choices order for multiple-choice questions with shuffle-answers on. In future, this sort of repetitive information will not be saved. Instead, during question processing, the question types will be given access to the full state history.

{| class="nicetable"
! Column !! Type !! Comment
|-
| id || INT(10) NOT NULL AUTO INCREMENT || Unique id. Not used much.
|-
| stateid || INT(10) NOT NULL REFERENCES question_states.id || Which state the submission of this data lead to.
|-
| name || VARCHAR(20) NOT NULL || The name of the parameter received from the student.
|-
| value || TEXT || The value of the parameter.
|}

===Upgrading the database===

===New list of states that a question may be in===

The aim here is to have as few states as necessary. What is necessary? To make it clear what is going on, for example in the quiz navigation. Of course, that is only one case to consider.

; Incomplete
: This is the state that questions start in. They stay in this state as long as the student still needs to give this question attention. In deferred feedback (non-adaptive) mode, that is until the student has entered an answer. (For a short-answer question, any answer in the input box moves you out of this state; for a matching question, you only move out of this state when you have answered all the sub-questions.) In adaptive mode, the question stays in this state until either you have got it right, or you have run out of tries.
: In the state, the student can enter or change their answer.
; Complete
: This state is for questions where the student have done enough, but the attempt is still open, so they could change their answer if they wanted to. For example, this happens in deferred feedback mode when the student has entered a complete answer, and before they do submit all and finish. Also, a Description, after the student has seen it.
: In the state, the student can enter or change their answer.
; Graded(Correct/PartiallyCorrect/Incorrect)
: For computer-graded questions, once the student can no longer interact with the question, it goes to one of the sub-states of the graded state.
; Finished
: For questions that do not have a grade, for example descriptions, after the attempt is over, they go into this state.
; GaveUp
: This state is used for questions where it is impossible to assign a grade because the student did submit all and finish when the question was in the incomplete state. However, this does not necessarily happen, for example, we may choose to grade an incomplete matching question if the student has completed at least one sub-question.
; ManuallyGraded(Correct/PartiallyCorrect/Incorrect)
; Commented
; GaveUpCommented
: These three states correspond the the previous three states after the teacher has added a comment and/or manually graded.

[[Image:Question_state_diagram.png]]

===Interaction models to implement initially===

It is the interaction model that controls how the student's attempt at the question moves through the state diagram in the previous section. When making the key decisions, that interaction model calls methods of the question type to supply question-type-specific information about what is happening.

Note that the main blocker to supporting negative scoring in the quiz is that it is not compatible with adaptive mode. So separating the models will allow MDL-1647 to finally be resolved.

====Deferred feedback====

This is how the quiz currently work when adaptive mode is off. The student enters a response to each question, then does Submit all and finish at the end of their attempt. Only then do they get feedback and/or grades, depending on the review settings.

====Legacy adaptive mode====

This re-implements how adaptive mode currently works, so old attempts can still be reviewed. In future, adaptive mode will be replaced with ...

====Interactive====

This will replace adaptive mode. This is the model that has been used successfully for several years in the OU's OpenMark system. The OU has also modified Moodle to work like this, but because of the way the quiz core currently works, I was not happy just merging the OU changes into Moodle core. In a sense, this whole document grew out of my thoughts about how to implement the OU changes in Moodle core properly.

====Manually graded====

This is a new mode, specifically tailored to manually graded questions. Should make it possible to fix a lot of the outstanding manual grading correctness bugs above.

====Each attempt builds on last====

This is for the benefit of the quiz feature of the same name. It is just like deferred feedback model, but subsequent attempts are seeded with the student's last response from the previous attempt. I think it leads to slightly cleaner code to do this as a separate interaction model. For example, it does not make any sense to use each attempt builds on last in combination with adaptive mode, although Moodle currently lets you do that.

====Immediate feedback====

This would be like cross between deferred feedback an interactive. Each question would have a submit button beside it, like in interactive mode, so the student can submit it immediately and get the feedback while they still remember their thought processes while answering the question. However, unlike the interactive model, they would not then be allowed to change their answer.

====Certainty based marking with deferred feedback====

This takes any question that can use the deferred feedback model, and adds three radio buttons to the UI (Higher, medium, lower) for the student to use to indicate how certain they are that their answer is correct. Their choice is used to alter the grade the receive for the question.

====Certainty based marking with immediate feedback====

This is like immediate feedback mode with the certainty based marking feature.

===Changes to the question type API===

Can this be backwards compatible?

===The interaction model API===

==Proposed robustness and performance testing system==

A major change to the question engine should really only be contemplated in combination with the introduction of a test harness that makes it easy to run correctness, performance and reliablitity tests.

One adavntage of the way data will be stored in the new system is that everything originally submitted by the user will be stored in the database in a format very close to the one in which it was originally received by the web server. Therefore, it should be easy to write a script that replays saved quiz attempts. This is the basis of a test harness. I will create such a test script as part of this work.

==Time estimates==

* Completing the above design: a few days.
* ...

===Time scales===

It seems highly unlikely that this could be done before Moodle 2.0 branches. Now that I have worked out the details of this proposal, I would really like to do it as soon as possible, so that would be for Moodle 2.1. However, that is dependant on my employer (either moodle.com or The Open University). This change is sufficiently big that I would not want to undertake it on my own time.

==Summary==

This change will cost about X months development. In exchange, the question engine will be more reliable, and will support a richer ranged of teaching and learning activities, through extending the range of interaction models supported, and maing it easier to write new question types.

==See also==

* The place to discuss this is the [http://moodle.org/mod/forum/view.php?f=121 quiz forum].
*There is also a [http://moodle.org/mod/forum/discuss.php?d=114527#p502692 Thread in Lesson forum].
* See the lists of [[#Why_things_need_to_change|related tracker issues]] above.

[[Category:Developer]]
[[Category:Quiz]]

Development:Quiz database structure

2009-03-06T18:45:43Z

Dooug:

{{Quiz developer docs}}
==Overview==

The following diagram shows the database structure for the quiz and question bank in Moodle 1.9.

There have been only small changes since Moodle 1.6. From memory, those changes are
* adding the quiz_feedback table in Moodle 1.7.
* renaming the question_sessions.comment column to manualcomment in Moodle 1.7.
* adding the column question.generalfeedback in Moodle 1.7.
* adding question.timecreated, timemodified, createdby and modifiedby. Replacing question_category.course with question_category.contextid. Removing question_category.published. This was Jamie Pratt's questionbank upgrade in Moodle 1.9.

The rest of this page refers to Moodle 1.5. There were big changes in the quiz database between Moodle 1.5 and 1.6 (basically separating out the question bank stuff from the quiz module). I still don't have time to update it for now, but a lot of the changes were just renaming tables, so if you can deduce the old table names from the new ones, there is useful information in there.

[[Image:Quiz_database.png]]

-----

The quiz data model has a fairly large pool of database tables, so the first step in explaining them is to provide some order. Conceptually it is possible to distinguish between the tables holding the teacher-supplied data, defining quizzes and questions, and the tables storing all the data that is generated when students interact with the quizzes and questions.

A further simplification is possible by separating out the questiontype specific tables. They are logical extensions to other tables and therefore are not necessary for understanding the general basic model. They are therefore explained on the developer documentation page for the individual [[Quiz developer docs#Question types|question types]].

The diagram below shows how the most important tables are linked to one another.

[[image:Quiz_database_tables.gif]]

==Teacher-supplied data==

===quiz===

The quiz table contains the definition for all the quizzes. Each quiz belongs to a course, reflected by the course id, has a name and a short descriptive text (intro), an opening and a closing time and several fields that store the settings of various quiz options, each of which is explained in the quiz help that is linked to from the quiz settings page.

The only field that requires additional information is the optionsflag, which... I will put an explanation here eventually --[[User:Gustav|Gustav]] 06:02, 5 February 2006 (WST)

For quizzes that contain random questions the $quiz object may acquire an additional property
;questionsinuse
:A comma separated list of questions that are being used in the quiz. This is used by the random questiontype to avoid choosing a question that is already being used.

===question===
All the questions code in moodle 1.6 has been transfered to the top-level question directory so that new question types could be more easily implemented as plug-ins and could be used in other components than just the quizzes.
The database has been modified so that the tables names do not refer to quiz. For example the quiz_questions table was renamed question.
Generally the table items where maintained so that the old code can be used but there has been specific modifications.
The 1.6 question table contains the id,category,parent,name,questiontext,questiontextformat,image,defaultgrade,penalty,qtype,length,stamp,version,hidden as the old quiz_questions table with the following modifications.

This table constitutes the item or question bank, i.e. the repository of defined questions. The quiz_questions table defines the data that is common to questions of all types.

;id
:int(10) NOT NULL auto_increment,
:Primary key. This is used as a foreign key in many other tables. for example the quiz_answers table allows to define an arbitrary number of answers that are part of the question. And many questiontypes have their own tables that hold more information about the question.

;category
:int(10) NOT NULL default '0',
:Foreign key: refers to an id in the table [[#question_categories|question_categories]]

;parent
:int(10) unsigned NOT NULL default '0',
:Foreign key: refers to an id in the table [[#quiz_questions|quiz_questions]]. This field is set to zero except in the case of wrapped questions as they are used for example in the multianswer questiontype. When a question wraps around any number of subquestions the subquestions will have their parent id field set to the id of the main question, thus allowing the question to find all its sub-questions (or wrapped questions). If parent is not '0' the question is never shown in a list of questions, because it is not a valid question by itself. This is also used for hiding random questions from the question list. Their parent field is simply set to their own id.

;name
:varchar(255) NOT NULL default '',''': in 1.6 utf8'''
:Question name that is used in question lists on the quiz editing pages

;questiontext
:text NOT NULL, ''': in 1.6 utf8'''
:Main text of the question

;questiontextformat
:tinyint(2) NOT NULL default '0',

;image
:varchar(255) NOT NULL default '', ''': modified in 1.6 to utf8'''

;defaultgrade
:int(10) unsigned NOT NULL default '1',
:Default grade for a question, usually '1'. In the wrapped parts of cloze questions defaultgrade represents the weight of the part.

;penalty
:float NOT NULL default '0.1',
:The penalty that is applied (if the respective quiz option is set) for an incorrect attempt at the question.

;commentarytext
:text NOT NULL default '',
:Extra feedback that is give to all studnets attempting the question, (as opposed to feedback which is different for each student depending on their repsonse).

;qtype
:varchar(20) NOT NULL '''utf8'''
:The questiontype of the question as text i.e. 'truefalse', 'calculated'

;length
:int(10) unsigned NOT NULL default '1',
:This defines how many question numbers are required for this question. It is generally set to "1", but the description questiontype, for example, sets it to "0", reflecting the fact that it doesn't have a question number.

;stamp
:varchar(255) NOT NULL default '',''': modified in 1.6 to utf8'''
:Unique identifier

;version
:int(10) NOT NULL default '1',''':
:A number representing how often the question was edited.
'''was modified to'''
:varchar(255) NOT NULL default ''

;hidden
:int(1) unsigned NOT NULL default '0',
:Controls whether to display a question in the question bank list in edit.php. Hiding questions is a mechanism to "delete" questions without removing them from the database and thus to allow them to be restored or "unhidden" at a later stage. Also the (unfinished and disabled) [[Quiz_developer_docs#Question_versioning|versioning feature]] uses the hidden field to prevent older versions of a question from cluttering the user interface.

In addition to these static fields, which are part of the generic question definitions, there are some additional fields that are only added to the object at runtime. There are two different kinds of fields: On the one hand there are questiontype specific fields, which are also part of the static question definition, but only apply to some questions. On the other hand there are fields that refer to the question instance, i.e. the question in the context of a particular quiz. The values for these later fields can differ when a question is used in different quizzes.

The runtime fields are added to question objects with the function <code>quiz_get_questions_options()</code> is called. This in turn calls the questiontype specific <code>get_question_options()</code> method for to add the options field.

;maxgrade
:This is the maximum grade that the teacher has assigned to this question in the context of the current quiz. This is by default equal to $questions->defaultgrade but the teacher can change this when editing the quiz. In the database it is stored in the [[#quiz_question_instances|quiz_question_instances table]].

;instance
:Foreign key: refers to an id in the table [[#quiz_question_instances|quiz_question_instances]]

;options
:Optional field. It provides a namespace for any questiontype specific information that may be stored in this field. It is generally set by the <code>get_question_options</code> method.

;name_prefix
:The prefix to be used on all interactions printed as part of the question.

We now deal in alphabetical order with the remaining tables for the teacher-provided data.

===question_answers===

This table allows a common way to store one or more teacher-defined answers for each question. It is not mandatory for a questiontype to make use of this table however. A questiontype may choose to store it's teacher-defined answers in an entirely different way, or even to do away with teacher-defined answers and use some other method to mark the student-supplied answer. For example it could calculate the correct answer on the fly.

;id
:int(10) unsigned NOT NULL auto_increment,
:Primary key

;question
:int(10) unsigned NOT NULL default '0',
:Foreign key to [[#quiz_questions|quiz_questions]] table.

;answer
:text NOT NULL,
:The string that represents the teacher-defined answer that will be compared to the student responses for grading and feedback purposes. The format of this field is entirely up to the question type.

;fraction
:varchar(10) NOT NULL default '0.0',
:The fraction of the total mark that the student should earn for giving this particular answer. This could be a negative number for wrong answers. Note that it is stored in a varchar(10) field.

;feedback
:text NOT NULL,
:Text that can be displayed to student as feedback when the student's responses match this particular answer.

In the past there was also a sequence number field was sometimes used to store the order of the different answers and was set to '0' if the order was of no importance.

===question_categories===

Categories are provided as a way to organize questions. Each category has a name and a descriptive text (info) and the sortorder as metadata. Categories allow hierarchical nesting via the parent id and can be private or published, i.e. they can be made available to teachers in other courses.

Since categories are simply a means for organising questions they are not vital for understanding how the quiz module works.

;id
:int(10) unsigned NOT NULL auto_increment,
:Primary key

;course
:int(10) unsigned NOT NULL default '0',
:Foreign key to the course table

;name
varchar(255) NOT NULL default '',

;info
:text NOT NULL,

;publish
:tinyint(4) NOT NULL default '0',

;stamp
:varchar(255) NOT NULL default '',

;parent int(10)
:unsigned NOT NULL default '0',

;sortorder int(10)
:unsigned NOT NULL default '999',

===quiz_question_instances===

Questions can be assigned different grades in different quizzes. These are stored in this table. This table can also be used to find all the questions assigned to a particular quiz. But note that this is not the same as the questions that are actually used because if the quiz contains random questions then these in turn choose other questions to be used when an attempt is started and these actually used questions are not stored in this table.

While, after a small extension, this table could also fulfill the purpose of storing the order of the questions in a quiz, this is currently still done in the questions field in the [[#quiz|quiz]] table.

;id
:int(10) unsigned NOT NULL auto_increment,
:Primary key

;quiz
:int(10) unsigned NOT NULL default '0',
:Foreign key to the id of the [[#quiz|quiz]] table.

;question
:int(10) unsigned NOT NULL default '0',
:Foreign key to the id of the [[#quiz_questions|quiz_questions]] table.

;grade
:smallint(6) NOT NULL default '0',
:The maximum grade assigned to this question in this quiz. This grade was set by the teacher on the [[mod/quiz/edit|quiz editing page]]. At runtime this information is stored in the $question->maxgrade field.

===quiz_question_versions===

This feature is not finished and disabled. The table structure may still change. See [[Quiz developer docs#Question versioning]] for a discussion.

;id
:int(10) unsigned NOT NULL auto_increment,

;quiz
:int(10) unsigned NOT NULL default '0',

;oldquestion
:int(10) unsigned NOT NULL default '0',

;newquestion
:int(10) unsigned NOT NULL default '0',

;userid
:int(10) unsigned NOT NULL default '0',

;timestamp
:int(10) unsigned NOT NULL default '0',

==Student-created data==

===quiz_attempts===

In the quiz_attempts table a record is created each time a user starts an attempt at a quiz. This is one of the important tables and the corresponding $attempt object is passed between many of the quiz module functions and methods.

;id :int(10) unsigned NOT NULL auto_increment,
:Primary key
;uniqueid :int(10) unsigned NOT NULL default '0',
:Primary key since Moodle 1.6 used by [[Quiz database structure#quiz_states|quiz_states table]]. See [[Separating_questions_from_quizzes]] for details.
;quiz
:int(10) unsigned NOT NULL default '0', The id of the quiz that is attempted
;userid
:int(10) unsigned NOT NULL default '0', The id of the user who is attempting the quiz
;attempt
:smallint(6) NOT NULL default '0', It is possible for a user to attempt a quiz several times, therefore the number of the attempt is stored in this field.
;sumgrades
:varchar(10) NOT NULL default '0.0', The (unscaled) grade for the attempt, i.e. if the grades assigned to the questions add up to 8, but the maximum grade for the quiz is set to 10, then the sumgrades field can contain 8 at maximum.
;timestart
:int(10) unsigned NOT NULL default '0', The timestart field is set to the current time when an attempt is started and is never changed afterwards.
;timefinish
:int(10) unsigned NOT NULL default '0', The timefinish field is set to "0" initially and to the current time when the attempt is closed. This is exploited at several places in the code to determine whether an attempt has been closed or not (i.e. closed = timefinish > 0). For all other modifications of an attempt record the timemodified field should be changed as well.
;timemodified
:int(10) unsigned NOT NULL default '0', This should be changed each time data in the record is modified.
;layout
:text NOT NULL, a comma separated list of question ids, with a "0" denoting a page break. Usually the comma separated list ends with ",0".
;preview
:tinyint(3) unsigned NOT NULL default '0', A flag that marks a teacher preview (i.e. an attempt by a user with teacher privileges) that may be automatically deleted when the quiz is previewed again, and which is not taken into account when viewing statistics.

===quiz_states===

States are saved for each interaction with a question. This allows a review of the complete history of a user's interactions with individual questions. The current state is the most important object used by the quiz module run-time code. This $state run-time object has a few additional fields not in the database that will be explained further down. The database fields are:

;id
:int(10) unsigned NOT NULL auto_increment,
:Primary key

;attempt
:int(10) unsigned NOT NULL default '0',
:Foreign key: refers to an id in the table [[#quiz_attempts|quiz_attempts]]. '''In Moodle 1.5 it referred to the id field of that table, since Moodle 1.6 it refers to the uniqueid field.''' This change was done while [[Separating questions from quizzes]].

;question
:int(10) unsigned NOT NULL default '0',
:Foreign key: refers to an id in the table [[#quiz_questions|quiz_questions]]. The attempt id and the question id together sufficiently identify a question instance that a state belongs to.

;originalquestion
:int(10) unsigned NOT NULL default '0',
:Foreign key: refers to an id in the table quiz_questions. It identifies which question was in use when the student attempted the question. This is part of the [[Quiz_developer_docs#Question_versioning|question versioning]] code.

;seq_number
:int(6) unsigned NOT NULL default '0',
:A counter that provides information about the sequence in which the states were created.

;answer
:text NOT NULL,
:This field is deleted during runtime and replaced with the responses field. For legacy reasons it is still called answer in the database. Questiontypes can store students' responses (usually in a serialized format) in this field using the method save_session_and_responses. Questiontypes are allowed to deviate from this and handle their response storage in any desired way. Some questiontypes do not use this field but instead store the student response in their own table extending this table.

;timestamp
:int(10) unsigned NOT NULL default '0',
:A time stamp to record when the state was created. This could mainly be useful for audit purposes.

;event
:int(4) unsigned NOT NULL default '0',
:Records the event or interaction type that lead from the previous state to the current one. The field can take the value of any of the following constants (defined in locallib.php):
:*EVENTOPEN: The attempt has just been opened and this is the initial state of a question, i.e. the user has seen the question did not interact with it yet.
:*EVENTSAVE: The responses are just being saved, either because the student requested this explicitly or because the student navigated to another quiz page.
:*EVENTVALIDATE: The student requested a validation of the responses. (This is not supported by all questiontypes.)
:*EVENTGRADE: The responses are being graded but the question session is not closed. This is generally the case for adaptive questions.
:*EVENTCLOSE: The responses are being graded and the question session is closed. Usually this happens because the whole attempt closes, either because the student requests it or because the time is up or because we are beyond the due date.
:*EVENTDUPLICATEGRADE: This is a strange one. It indicates that the responses would have been graded had they not been found to be identical to previous responses.

;grade
:varchar(10) NOT NULL default '0.0',
:Calculated from the raw grade by deducting the penalty and rescaling to the defaultgrade value of the question. Note that this is a varchar(10) field like most grade-related fields in the quiz module.

;raw_grade
:varchar(10) NOT NULL default '',
:The grade that was achieved for the question scaled to the question's weight or grade as assigned in the quiz_question_instances table

;penalty
:varchar(10) NOT NULL default '0.0',
:The penalty incurred during the transition from the previous state to this one. This is different to the cumulative penalty, which can be calculated by adding up all the penalties from previous marking events.

In addition to the database fields a few fields are added to the states at run time. They are dealt with by the questiontypes with the methods <code>create_session_and_responses</code>, <code>restore_session_and_responses</code> and <code>save_session_and_responses</code> and are defined as follows.

;sumpenalty
:The cumulative penalty, which can be calculated by adding up all the penalties from previous marking events. For efficiency these cumulative penalties are stored in the table [[#quiz_newest_states|quiz_newest_states]] so that they do not need to be re-calculated each time.

;changed
:This records whether a change has taken place to the run-time state. This is set to false when the state is restored from the database. Any code which changes the question state must set this field to true and must increment seq_number. The <code>quiz_save_question_session()</code> function will save the new state object to the database if the field is set to true.

;responses
:This is automatically set to the value of the database field <code>answer</code> before that one is removed. The responses field contains an array of responses. In the default case of a single response the value can be found in ->responses['']. For questiontypes using several HTML form elements for their responses the array contains a value for each of the interaction elements. The indices are determined by the name of the elements after the name_prefix is stripped.

;last_graded
:This field contains another state object representing the most recent graded state in the history of this question attempt. This is necessary, because if a state merely represents a save interaction, it should not affect the session in any way. Therefore another grade interaction has to proceed from the last graded state.

;options
:Optional field. Similarly to the options field in the question object, this field provides a namespace for any questiontype specific information. The difference is that the information in the state->options field should be state specific, whereas the question->options field should be static.

We now deal in alphabetical order with the remaining tables holding the student-created data.

===quiz_grades===

The quiz_grades table merely stores a student's awarded grade for a quiz. Since it is possible to allow several attempts on a quiz, the grade stored is calculated depending on the quiz setting grademethod. This table exists mainly for convenience, because the values of its fields can be recalculated.

;id
:int(10) unsigned NOT NULL auto_increment,
:Primary key

;quiz
:int(10) unsigned NOT NULL default '0',
:Foreign key refering to the id of the [[#quiz|quiz]] table

;userid
:int(10) unsigned NOT NULL default '0',
:Foreign key refering to the id of the user table

;grade
:double NOT NULL default '0',
:The grade awarded for this quiz to this user

;timemodified
:int(10) unsigned NOT NULL default '0',
:Unix timestamp to be updated each time the grade is changed

===quiz_newest_states===

This table exists only for efficiency reasons:
#Via its 'newest' and 'newgraded' fields it gives attempt.php a way to quickly find the newest state and the newest graded state for an attempt. It allows the construction of SQL to select all the states that need to be loaded on attempt.php or review.php.
#Via its 'sumpenalty' field it gives quiz_apply_penalty() a quick way for getting at the accumulated penalty that needs to be applied. Without this field the penalties from all previous graded states would have to be added up each time. Not a big deal actually because this could be achieved with a single SQL query (using SUM) but this field was introduced when we still had the multiplicative penalty scheme around which would have been more difficult to recompute.

This table was introduced in Moodle 1.5 and is not populated for all states during the upgrade because on sites with a lot of existing states that could take too long. Rather it is done whenever needed by quiz_upgrade_states().

;id
:int(10) unsigned NOT NULL auto_increment,
Primary key

;attemptid
:int(10) unsigned NOT NULL default '0',
Foreign key referring to the id in the [[#quiz_attempts|quiz_attempts]] table. In Moodle 1.5 it referred to the id field of that table, since Moodle 1.6 it refers to the uniqueid field. This change was done while [[Separating questions from quizzes]].

;questionid
:int(10) unsigned NOT NULL default '0',
Foreign key referring to the id in the [[#quiz_questions|quiz_questions]] table

;newest
:int(10) unsigned NOT NULL default '0',
Foreign key referring to the id in the [[#quiz_states|quiz_states]] table. This points to the newest state for this attempt and this question.

;newgraded
:int(10) unsigned NOT NULL default '0',
Foreign key referring to the id in the [[#quiz_states|quiz_states]] table. This points to the newest state created by a grading event.

;sumpenalty
:varchar(10) NOT NULL default '0.0',
This stores the total penalty that this user has accumulated over previous graded responses to this question in this attempt.

==See also==

* [[Development:Question_database_structure|Question database structure]]
* [[Development:Database_schema_introduction|Database schema introduction]]

[[Category:Developer|Quiz database structure]]
[[Category:Quiz]]

User:Douglas Reith

2009-03-02T21:02:23Z

Dooug: New page: New to Moodle. I live/work in Ann Arbor, Michigan.

New to Moodle.

I live/work in Ann Arbor, Michigan.

Development:Random question type

2009-03-02T21:01:28Z

Dooug: fixed typos

{{Questiontype developer docs}}

The random question is not a real question but only a device to randomly choose a question from a category. The $question->questiontext field is abused as a flag: 1 means choose question from the category and its subcategories, 0 means only use questions in the category itself.

When a new session is started for a random question then a question is chosen from the category. We need to make sure that no question is used more than once in the quiz. Therefore the following need to be excluded:
#All questions that are explicitly assigned to the quiz
#All questions that are already chosen by an other random question
#Random questions
#Other explicitly excluded question types
#Wrapped questions

To do the first the question type uses an additional property $quiz->questionsinuse that holds a comma separated list of all questions used in the current quiz. To do the second the questiontype class has a property 'catrandoms' which is an array indexed by category id and by $question->questiontext. Each entry is a randomized array of questions in that category which can be used.

==Explicitly excluded question types==

See <code>question/type/random/questiontype.php</code> for the list of explicitly excluded types:

var $excludedtypes = array("'random'", "'randomsamatch'", "'essay'", "'description'");

Please note that because of problems with grading, the [[Essay_question_developer_docs|essay question]] is not supported at the moment. See more information at the thread [http://moodle.org/mod/forum/discuss.php?d=50903 Random Question Test].

==Response storage==

The answer field for random questions come in two flavours:

*For responses stored by Moodle version 1.5 and later the answer field has the pattern random#-* where the # part is the numeric question id of the actual question shown in the quiz attempt and * represents the student response to that actual question.

*For responses stored by older Moodle versions - the answer field is simply the question id of the actual question. The student response to the actual question is stored in a separate record.

This means that prior to Moodle version 1.5, random questions needed two records for storing the response to a single question. From version 1.5 and later the question type random works like all the other question types in that it now only needs one record per question.

==$question->options==

The random question type does not have any options for $question->options.

==$state->options==

This has one property 'question' which is set to the fully instantiated question object for the randomly chosen question.

[[Category:Developer|Random question type]]
[[Category:Quiz]]

Development:vim

2009-02-23T14:50:08Z

Dooug: typo fix

= Setting up Vim for Moodle Development =

There are many powerful features available in vim, here are a few tips
to ensure you get powerful use out of vim with moodle development

== Indentation ==
Moodle [[Coding|Coding Guidelines]] state that indentation should be spaces of 4 spaces (not Tabs). Some .vimrc options will help you adhere to these guidelines with ease:

<pre>
" insert space characters whenever the tab key is presse
set expandtab

" insert 4 spaces characters when tab key is pressed
set tabstop=4

" insert 4 spaces wen autoindent indents
set shiftwidth=4

" automatically indent files
set autoindent

" Do smart indentation depending on code syntax (e.g. change after { }, keywords etc)
set smartindent
</pre>

== Syntax Highlighting ==
Recent versions of vim have robust syntax highlighting for php. In order to switch syntax highlighting on add the following to your .vimrc:
<pre>
" set syntax highlighting on
syntax on
</pre>

== Other Useful vimrc settings ==
<pre>
" show a ruler with line number, % through file on status line
set ruler
" show line number
set nu
" PHP syntax check
set makeprg=php\ -l\ %
set errorformat=%m\ in\ %f\ on\ line\ %l

</pre>
== File encoding and decoding ==
<pre>
set fileencodings=ucs-bom,utf-8,gbk,big5,latin1
set termencoding=utf-8,gbk
if has ('multi_byte') && v:version > 601
if v:lang =~? '^$zh$\|$ja$\|$ko$'
set ambiwidth=double
endif
endif
</pre>

== Status Line ==
<pre>
set statusline=%<%f\ %h%m%r%=%k[%{(&fenc==\"\")?&enc:&fenc}%{(&bomb?\",BOM\":\"\")}]\ %-14.(%l,%c%V%)\ %P
</pre>

== Ctags ==
See the [[Development:ctags|Ctags Article]] for details on how to setup ctags usage with moodle and vim