Better handling of overdue quiz attempts

Handling of overdue quiz attempts
Project state	Implementation in progress
Tracker issue	MDL-3030, MDL-4309
Discussion	Quiz forum thread
Assignee	Tim Hunt

Moodle 2.3

There is a tricky issue to do with what happens exactly when time runs on on a quiz. On the one hand, we want to let students use every second of time. On the other hand, the end of a quiz is often a time of high server load, and so Moodle is likely to responding slowly. Therefore, we have to prevent students from cheating on the time-limit while still processing their last-minute submission even if the server is heavily loaded.

There is also the problem of what happens if the student just logs out and does not submit at all. At the moment the attempt goes into a sort of limbo where Moodle can't really do anything with it.

What happens at the moment

Currently there are only really two states a quiz attempt can be in, and they are distinguished by whether quiz_attempts.timefinish is 0 or contains a time-stamp.

At most one attempt is allowed to be in the In progress state at any time.

After time expires, (either because the time limit runs out, or the close date passes) the attempt is in a weird state. Well, not really, the attempt is still considered to be In progress, but the student is no longer allowed to access it, so there is no way to get at the Submit all and finish button to submit it. Also, if the quiz allows more than one attempt, the student cannot start a new attempt, because they already have an In progress attempt (which they are not allowed to access!)

The known work-around is, as teacher, to:

1. edit quiz settings to allow more time;
2. login-as the student and submit the quiz;
3. set the time-limit or close data back to what it should be.

This is a real pain.

Before reading on, you may wish to remind yourself of the Quiz user interface overview, so you know what the key scripts view/startattempt/attempt/processattempt/summar/review.php each do.

To document the two existing transitions in some detail (note, all transitions are logged, I have not bothered to say that every time):

Transition: Start attempt

Goes to: In progress

Allowed when: If the student does not currently have any In progress attempt, and if they meet all the conditions imposed by the access rules, they may start one.

How it is done: The student clicks the Start attempt button on view.php. Technically, this takes the form of a POST request to startattempt.php.

What happens:

1. The attempt, and corresponding $quba, are created.
2. They are initialised by adding all the questions, which may depend on randomisation, and 'each attempt builds on last'.
3. The attempt and all associated data is written to the database.
4. A quiz_attempt_started event is raised. (This even it not used in standard Moodle.)

Transition: Submit all and finish

Goes from: In progress

Goes to: Finished

Allowed when: The student has an In progress attempt, which they are currently allowed to access according to the access rules. In particular, where the time limit has not run out, and the close date has not passed.

How it is done: Either

• The student clicks the Submit all and finish button on summary.php; or
• Time runs out while the student is working on the quiz, and so the quiz is submitted automatically (if JavaScript is enabled).

Technically, this takes the form of a POST request to processattempt.php, althought the code that does the work is the quiz_attempt::finish_attempt method in attemptlib.php.

What happens:

1. All questions that are not already finishes are finished.
2. The updated question states are written to the database.
3. The quiz_attempt record is updated with the timefinish and the total mark.
4. The overall quiz grade (combining all the student's attempt scores) is updated, and stored in the gradebook.
5. A quiz_attempt_submitted event is raised. (This is used by the quiz itself to send out any tutor or student notification messages that are enabled.)

Rough outline of the solution

We will introduce a new quiz settings When time expires and Grace period. When time expires is the strategy we will use for overdue attempts. For some strategies, they will need an amount of extra time that is the grace period. The possible strategies are:

1. Submit attempt automatically,
2. Allow a grace period to submit, but not change any responses, or
3. That's it. Tough luck! - we need a better name for this!

1. Corresponds to doing an automatic submit all and finish. 3. Corresponds to automatically moving the attempt to the Abandoned state. See below. 2. is more interesting (and is what the OU wants for assessed tests).

If the student has not submitted their attempt when time runs out, then the attempt moves into the Overdue state. In this state, when the student tries to get to the quiz attempt, then can only go to the Summary page, where all they can do is to click the Submit all and finish button.

If they finish submitted before the grace period runs out, then their attempt is counted.

If the student still does not submit, then the attempt moves into the Abandoned state. This means that the student is not prevented from starting a new attempt, if the quiz allows it.

Question: should we allow the student to abandon the current attempt at any time? For example, if an interactive quiz allows multiple attempts, and the student has got off to a bad start, should we let them abandon and start again, or should we force them to Submit all and finish if they want a new attempt? - for now we will not allow students to abandon any attempt.

In addition, we will start storing the quiz_attempt.state explicitly in the database, rather than relying on deducing it from timefinish or other columns. In a sense this is redundant information, but since significant things happen when the state changes (e.g. a message might be sent) I think making the state, and hence the state transition, explicit, will make the code clearer.

We will need code that detects when an attempt is in the 'wrong' state. For example, if a state is stored in the database as In progress, but time has now expired, then we will need to detect that and correct it by triggering the correct state transition. I think we need to do this in two ways:

1. Whenever we load an attempt from the database. This ensure that if the user is actively working on the quiz when a state transition is supposed to happen, they will immediately see the attempt change state.
2. We will need to add a cron script that detects attempts becoming overdue when no-one is looking at them.

(Note there is a subtlety, the JavaScript timer that runs in the user's browser will try to automatically trigger a submit, they may be happening at the same time as a teacher is reviewing the attempt (which might trigger 1) and cron may also be running at the same time (which might trigger 2). Therefore, we need a clear protocol to avoid nasty race-conditions.

Actually, a possible third way (instead of, or in addition to cron). Perhaps we should detect this when the quiz settings (or overrides) are edited, and check all attempts then and trigger state transitions. This might be a performance problem on quizzes with thousands of attempts.

New state diagram

At most one attempt can be in either the In progress or Overdue states at any time.

The existing transitions need to be updated:

• They all need to set the new quiz_attempt.state column as part of What happens.
• The transition Submit all and finish can now also go from Overdue to Finished.
• When time runs out as a student is working on their quiz attempt, this should no longer trigger an automatic Submit all and finish. Instead it should trigger an automatic Time expires.

The new transitions are:

Transition: Time expires

Goes from: In progress

Goes to: Overdue

Allowed when: Time expires, either due to the time-limit, the close date, or some other custom access rule; or because the teacher has edited the time limit and/or close date and shortened them; and when the overdue strategy is "Allow a grace period to submit, but not change any responses".

How it is done: Detected automatically by the Moodle code, either because

• time has passed;
• the teacher has edited the quiz settings or overrides to shorten the time limit and/or close date; or
• triggered by the countdown timer JavaScript, in which case this will trigger a POST to processattempt.php.

What happens:

1. quiz_attempt.state is updated
2. quiz_attempt_overdue event is raised. (The quiz will use this itself to send a warning message to the student, if that is enabled.)
3. If the student is currently working on the quiz, they are redirected to the summary.php page.

Transition: Give up

(not sure that is the best name)

Goes from: Overdue / In progress

Goes to: Abandoned

Allowed when: Time expires, and the overdue strategy is "That's it. Tough luck!"; or the grace period expires when an attempt is in the Overdue state.

How it is done: Detected automatically by the Moodle code, either because

• time has passed;
• the teacher has edited the quiz settings or overrides to shorten the time limit, close date and/or grace period; or
• triggered by the countdown timer JavaScript, in which case this will trigger a POST to processattempt.php; or
• if we decide to allow it, the student clicks a 'Give up button', perhaps on the summary page.

What happens:

1. quiz_attempt.state is updated
2. quiz_attempt_abandoned event is raised. (For completeness. I don't foresee a use for this at the moment.)

Transition: Time limit extended

This shows the kind of tricky thing that can happen. We have to consider what happens if the teacher edits the quiz settings to extend the time limit when an attempt has already been marked overdue.

Goes from: Overdue

Goes to: In progress

Allowed when: The teacher edits the quiz settings or the overrides and extends the time limit.

How it is done: Detected automatically by the Moodle code.

What happens:

1. quiz_attempt.state is updated
2. quiz_attempt_restarted event is raised. (For completeness. I don't foresee a use for this at the moment.)

What can be accessed when

For an In progress' attempt: The student can access attempt.php and summary.php; the teacher can access review.php.

For an Overdue' attempt: The student can access review.php and summary.php; the teacher can access review.php.

For a Finished' attempt: The student and teacher can access review.php.

For an Abandoned' attempt: The student and teacher can access review.php.

Summary of changes

Database changes

• New column quiz.overduehandling (char) to store which strategy we will use for overdue attempts.
• New column quiz.graceperiod (int) to store the time an attempt is allowed to remain in the Overdue state.
• New column quiz_attempt.state (char) to explicitly store the attempt state.

New capability

• mod/quiz:emailwarnoverdue to control overdue warning messages.

UI changes

• New fields 'When time expires' and 'Grace period' on the quiz settings form, with corresponding admin settings.
• Do we need even more columns of 'review options' on the quiz settings form? (For example for Overdue and Abandoned attempts.) Perhaps it is time to move these settings to a separate page?
• Information about the grace period may need to be displayed on the quiz settings page, or on the summary page for overdue attempts.
• List of attempts on the view.php page needs to indicate the state of attempts.
• review.php page needs to indicate the attempt state.
• Display attempt state in the quiz reports where appropriate.

Rough work break-down

1. Done! New quiz.overduehandling, quiz.graceperiod and quiz_attempt.state DB columns definition, upgrade code, backup & restore.
2. Done! Overdue handling and Grace period setting on quiz form, with any validation. Verify create / edit works.
3. Implement back end of the three new transitions.
4. Code where attempts are loaded to identify attempts in the wrong state and trigger a transition if necessary.
5. Catch the quiz_attempt_overdue event and send an email.
6. Make attempt_state clear on the view page list of past attempts.
7. Update logic for the 'Start attempt button', and in startattempt.php so that for overdue attempts users are redirected to the summary page.
8. Update summary page to display appropriate information about overdue attempts.
9. Update processattempts.php so that it correctly handles one single submission after an attempt becomes overdue, and, for overdue attempts, will only redirect to the summary page.
10. Display the attempt state on the review page.
11. Add the attempt state to the overview and responses quiz reports (this will actually be a change in the base class).
12. Add new options to the attempt and overview reports to filter the report by attempt state. (We can probably usefully refactor the settings forms and preferences handling to extract a base class.)
13. While we are working here, we should really update these two reports to use the Moodle 2.x enrolment SQL, instead of get_users_by_capability.
14. Change the statistics report to analyse attempts with state = finished, rather than timefinish <> 0.
15. Cron to detect attempts in the wrong state and trigger state transitions. (Worry about race conditions!)

Bits that are not yet clear:

• Potential race conditions (affects 4, 7, 9, 15).

Test script

The 3 examples at http://moodle.org/mod/forum/discuss.php?d=188534 are a good starting point. They just need to be elaborated.

Hypothetical other transitions

The following state-transitions can also be conceived, and could be implemented in the future, but they are out-of-scope for this proposal.

Transition: Reopen attempt

Goes from: Finished / Abandoned

Goes to: In progress

Allowed when: Time has not expired, and the current user has the appropriate capability.

(For example, the student accidentally clicked Submit all and finish, and the teacher wants to be nice and undo that for them. Alternatively an attempt has become Abandoned because time ran out, but then the teacher extended the grace period or time limit, and so wants to re-open some attempts. In this case, the transition may go to Overdue, or perhaps that is a different transition, or perhaps the right way to handle that is to first transition to In progress and then transition to Overdue. I think it is clear why this is being considered out-of-scope.)

How it is done: Probably clicking a button on the review page or in the quiz reports. This would probably be a POST to a new reopenattempt.php

What happens: This is a bit tricky because lots of complex stuff happens on Submit all and finish that now needs to be undone, but I think a possible approach is to delete any question_attempts_steps added by the finish action (and any that follow them) and then use a regrade of the attempt to reset everything else.

Actually, it is much easier to re-open an Abandoned attempt than a Finished one. Perhaps we should consider them two different transitions, or perhaps reopening an Abandoned attempt can be treated the same as Time limit extended.

Transition: Count this anyway

Goes from: Abandoned

Goes to: Submitted

Allowed when: The current user has the appropriate capability.

(For example a teacher decides that even though the student abandoned their attempt, it should be counted anyway.)

How it is done: Probably clicking a button on the review page or in the quiz reports. This would probably be a POST either to processattempt.php with extra arguments, or to a new script.

What happens: Probably very similar to Submit all and finish.

See also

• MDL-3030 and MDL-4309
• Quiz developer documentation