Question Engine 2:Testing

This page gives some notes of what parts of the system most need to be tested, and what has been done already.

Previous section: Implementation plan

Overview

First note that an earlier version of this code (based on Moodle 1.9) has been running on the Open University's live servers since December 2010, and processing 30000+ quiz attempts each month, with very few problems.

Therefore, the following parts have already been well tested:

• All the basics of building and running quizzes
• The quiz reports (although we don't use manual grading much)
• Deferred feedback behaviour
• Interactive behaviour
• The system running on Postgres (which is the database that OU uses). Note that several people have run my work-in-progress code on MySQL, so I think we have also flushed out most of the MySQL-specific bugs by now.

The bits that particularly need more testing are:

• The upgrade from Moodle 2.0 to 2.1
  • Especially if your Moodle site was originally upgraded from Moodle 1.x, and contains a lot of old quiz attempt data.
• Adaptive behaviour (and Adaptive, no penalty)
• Manual grading report in the quiz
• Running the new code on MS SQL and Oracle.

When testing and reporting bugs

A good way to test is to have Moodle 2.0 and Moodle 2.1 beta both installed. That lets you easily compare the new and old behaviours in the places where things are not supposed to have changed significantly (for example Deferred feedback and Adaptive behaviours). Of course, the whole point of this upgrade is to change how things work for the better, so when you observe a difference between the old and new system you need to decide whether it is:

• something that has intentionally changed for the better;
• a bug in the old system that no longer arises; or
• a bug in the new system;
• something that was intentionally changed in the new system, but for reasons that affect you it was a bad change.

Obviously, only the last two categories of things need to be reported to the bug tracker.

I recommend you test with Debugging set to DEVELOPER level. The new code should not generate any debug output.

I just want to remind you that the most helpful thing you can include in a bug report is step-by-step instructions for triggering the problem (steps to reproduce) so I can make it happen in front of my eyes.

Also, screen-shots can help if the problem is at all visual. Also, if there is any error message or debug output, it is more helpful if you copy-and-paste the text of the error message, rather than (or as well as) taking a screen-shot.

You might also consider exporting the question that demonstrates a problem in Moodle XML format at attaching it to the bug report. That might make the bug easier to reproduce.

Further details about good bug reports on the Tracker page.

Testing the upgrade

TODO

Details to follow.

Testing adaptive behaviours

As noted above, the best way to test is to compare an adaptive quiz running in 2.1 with one running in 2.0. Ideally you should test all question types, and various sequences of getting each question right and wrong.

Testing manual grading

The manual grading report has been substantially rewritten. Hopefully the new report provides a much simpler interface for teachers with a more efficient workflow.

The new code has already had some testing at the OU, but it is not an area we use a lot, so more testing by experienced users would be good.

Finally

Thank you for your interest in helping with this testing.

See also

• Tracker
• Moodle_2.1_release_notes
• Upgrading_to_Moodle_2.1
• Back to Question Engine 2