Question Engine 2:Rationale
- Goals
- Rationale
- How it currently works
- New system overview
- Detailed design
- Question Engine 2 Developer docs:
- Implementation plan
- Testing
The Question Engine is the part of the question code in Moodle that deals with what happens when a question is attempted by a Student, or later reviewed by the Teacher or the student themselves or included in reports. Roughly speaking this is half the question code, the other part being the Question Bank, which allows people to create, edit, organise, import, export and share questions. The Question Engine is the part that has to perform when you have a large class of students taking a quiz simultaneously.
This proposal is a by-product of more than three years working on the Moodle quiz and question code. During that time I have fixed hundreds of bugs and feature requests. Some of the bugs with the Question Engine have requried careful fixing in order to avoid regressions, and there are other issues that are hard to do anything about at the moment.
As well as bug fixes, I have also overseen major developments in the quiz/question system. For example Jamie's changes to the Question Bank, Ollie's new quiz editing interface, and Mahmoud's new quiz navigation that I added to Core Moodle while refactoring the quiz attempt code.
Over the three years, I have also followed the Moodle quiz forum and seen what it is that people struggle with, complain about, want to change (and also what they like!). For example, having coached some people when writing new question types I realise that creating a new questions type is more difficult than it needs to be.
The changes proposed here fix parts of the question engine code that I have been dissatisfied with since about the first time I looked at it. There are several reasons why I have not addressed them before now.
- To fix them properly, we must change how the data is stored in some of the key question tables. In some sites, these tables contain millions of rows, and the processing required on update is non-trivial and is different for different question types. This is not something to undertake lightly.
- It took me about two and a half years to work out this solution. Some parts of this proposal where clear from the start, but the exact notion of question interaction models, and how they would relate to question types, only crystallised recently in my mind. Because of 1. I did not want to change anything until I was sure what I was proposing was right, not just in general, but in all the details.
- The mundane reason that there have been plenty of other things to work on, and I have not had time to address this until now.
- These are not critical problems. There have been no major changes to the question engine since the 1.6 release. Clearly there are not fundamental flaws with the current system. It is just not as good as it could be.
As this is a major change. My estimate is that doing what I propose here is a couple of month's solid work.
Related tracker issues
One way to see why things need to change is this list of tracker issues which these changes should fix.
Correctness issues
| Key | Votes | Watchers | Summary |
|---|---|---|---|
| MDL-3030 | 31 | 21 | attempts should be auto-closed |
| MDL-3936 | 3 | 2 | Essay Question used in score before grading |
| MDL-9303 | 9 | 8 | General Feedback not shown for essay questions on review screen |
| MDL-9327 | 9 | 10 | Text in Essay type questions is not saved in second or more attempts when the teacher has not graded previous attempts |
| MDL-13289 | 0 | 1 | Essay question with adaptive mode not show feedback |
| MDL-17681 | 0 | 0 | Manually assigned grades cannot easily by re-scaled when the marks for a question is changed |
Robustness issues
| Key | Votes | Watchers | Summary |
|---|---|---|---|
| MDL-11852 | 0 | 1 | Quiz does not score properly |
| MDL-12344 | 0 | 0 | Student takes quiz, upon completion of quiz receives the following |
| MDL-12665 | 3 | 3 | quiz attempts sometimes show zero despite correct answers |
| MDL-13073 | 0 | 1 | Regrading (really) messes up grades |
| MDL-13246 | 0 | 2 | Quiz does not save score. Shows 0/20 |
| MDL-14873 | 0 | 2 | quiz answers submit freezes (occationally) |
| MDL-15305 | 1 | 1 | Quiz doesn't work after submit all and finish button is selected - get a "No quiz with id=0" or "No attempt with id=0" error message |
| MDL-16451 | 0 | 1 | Quiz scores 0 with secure window; Grades do not transfer |
| MDL-16490 | 1 | 2 | timed quiz generates bad 2nd attempt |
| MDL-16663 | 0 | 1 | errors and missing work |
(Many of these are intermittent and only affect some people. Sometimes bugs like this turn out to be the result of a corrupt table in MySQL. Still, we really should not be getting problems like this with a system that is sometimes relied on as much as the quiz is.)
Richness issues
| Key | Votes | Watchers | Summary |
|---|---|---|---|
| MDL-1647 | 14 | 13 | Allowing negative marks for questions |
| MDL-4860 | 0 | 0 | Questiontype giving hints |
| MDL-12821 | 0 | 1 | Essay quiz question redundant submit button |
| MDL-17894 | 1 | 1 | Support certainty/Confidence based marking |
| MDL-17895 | 0 | 1 | Allow question types to validate responses before grading |
| MDL-17896 | 1 | 1 | Question types like Opaque should not require nasty hacks in question_process_responses |
Issues I don't propose to fix with this change
... but which would become easier to fix as a result:
| Key | Votes | Watchers | Summary |
|---|---|---|---|
| MDL-3452 | 0 | 0 | Use saved answers when marking late submissions |
| MDL-4309 | 3 | 7 | Ability for late quizzes (with penalty) |
| MDL-11047 | 9 | 4 | Quiz limit to one question per page with no access to previous or next question |
| MDL-15596 | 0 | 0 | Allow a question to be used more than once in an attempt |
See also
The next section, How the question engine currently works, contains further remarks about why change is needed.
- Back to Question Engine 2
