Plans for adaptive mode

This page is currently work in progress.

At the OU we are planning to do some work on adaptive mode of the quiz, because we think that there are some usability problems with how it currently works, and because it does not quite meet our needs. This page outlines those plans so we can discuss them.

We have been thinking about the on-and off for quite a long time, and have recently come up with this definite proposal. We being three members of OU staff Tim Hunt (leading developer and Moodle quiz maintainer), Phil Butcher (eAssessment business process leader - that means the person who consults with academic staff to gather requirements) and Mahmoud Kassaei (developer, who will be doing most of the work on implementing this).

Please discuss this proposal in this quiz forum thread.

The main usability problem

To demonstrate the big usability problem:

A. In an adaptive quiz, you enter the wrong answer to a question, the click submit. This is what you see:

B. Then you think again, and enter the right answer (but don't click submit yet). This is what you see:

C. Then you click save, and go away, perhaps you want to check a resource in the course that explains the underlying theory. Then you come back. Now it looks subtly different:

D. Finally when you do click submit you get this:

We believe that what you see in B. and C. is confusing, and students should never see a question in this state.

So how do we avoid this?

The problem is that we only have a single way to display questions during the attempt in adaptive mode, which is "showing feedback to any previous response, and allowing the student to enter their next response". (Then, whether in adaptive or non-adaptive mode, there is the reviewing the quiz after the attempt mode.)

We propose to change and have two separate states during an adaptive attempt. First "letting the student enter their next response, and then "showing feedback for the previous response".

Something like this:

What do we lose by our proposed approach?

The one thing this loses is that when they student is entering their second try, they cannot see the feedback from their first try. Since the whole point of feedback is to help the student correct their own mistake, and in the process learn something, this at first appears to be a big loss.

The other side of the argument is that our way forces the student to read and absorb the feedback before clicking next. Also, the student may work out their next answer (perhaps making notes on paper) while the feedback is still visible, before clicking try again.

Other changes we propose

If you looked closely at the above state diagram, you will have seen these already.

Limiting the number of tries a student can have at each question

So if the student still has not got the right answer after a number of tries, they get 0 marks, and get told what the right answer is. The number of tries can be set on the question editing from, like the penalty factor. There will be an 'unlimited' option for people who like the current behaviour.

After the student has got a question right, they can no longer change their answer

This seems consistent with stopping them after three wrong tries.

Allowing different feedback to be given after each try

At the moment, questions can have two sorts of feedback.

a. Feedback that depends on what the student answered.

b. General feedback that is shown to all students irrespective of their answer.

To this we want to add

c. Hints that depends on the number of times the student has tried the question and got it wrong. The idea is that the second hint will give more guidance than the first hint, and so on.

Then, after an incorrect try at a question, the feedback that depends on the student's answer and the appropriate hint will be displayed.

And after the student has finished the question (either by getting it right, or getting it wrong too many times) they will be shown the specific feedback that depends on their last answer, and the general feedback.

This means that if a question allows three tries, you need two different hints.

We don't think that this is ideal. It would be nice if hint could depend on the student's answer as well as on the number of incorrect tries, but that would make the question editing form impossibly complicated. We think our proposal for hints allows quite a lot of flexibility and power, without making things too complicated, and so is the best compromise.

What this means for the question editing form

The main change here is that we want to move all the stuff that relates to adaptive mode That is, 'penalty factor', a new field 'number of tries allowed', and then an appropriate number of HTML areas for the hints (number of tries - 1). This means that people who are not using adaptive mode, can ignore this part of the form entirely.