Quiz Usability portal/Different paginating features in quiz

Return to: Quiz Usability portal

This document discusses Quiz functionality in Moodle 1.9 and Moodle 2 HEAD in Dec 2008, in order to solve issues related to tickets MDL-17333 and MDL-6353.

The issue

I read the description in MDL-6353 and thought this was a mistake until I tested it: "Well, if the option 'shuffle questions' is turned on, then they are automatically paginated.

But if the shuffle option is off, then they are actually manually paginated. The quiz 'questions per page' option then only has an effect when a new question is added."

I realized that this is right. So "I think what this setting does is it keeps the quiz organized permanently" in the description of this bug is wrong IF shuffle questions is set to "No".

Like, what?? This inconsistency of behaviour makes no sense to me whatsoever, and is not communicated by the UI at all. This is not an issue with the wording of the setting, but the whole model of the functionality.

Breaking it down

That is, there are three features for managing the paging of a quiz

• A. Repaginate on request - allows changing the paginating manually afterwards for a more intricate paginating
• B. Keep quiz paginated consistently throughout with N questions
• C. Paginate any questions added from now on with N per page

(B and C are currently strangely mixed between the two settings "questions per page" and "shuffle questions". ).

A is a command, B and C are modes/settings which need to be made apparent in the UI. To make C not a mode, it could be a selection the user makes at the time of each 'add question' operation.

So the wording "After adding N questions" does make sense when shuffle questions is NOT enabled, since that is all it does (feature C).

But it does not make sense when shuffle questions is enabled, in which case the feature in question turns into another feature (features B & C combined).

Storing the order when paginating is fixed

When B is set, the quiz editing UI should reflect this reality:

• the editing UI should show the questions according to B
• the controls for paging should be disabled (as they are)

This arises the question of whether or not the manual order of questions should still be stored when B is set - and if it is, should the UI reflect this reality - the fact that in a sense, the quiz is storing two different quiz layouts when B is set, and that the user has the option to "undo" the effect of setting B by turning it off.

Or should setting B result in

1. first running A with the number of questions per page set for B,
2. and then enforcing that order,
3. and when B is turned off only the enforcing of the order is stopped

This would mean that the manual ordering would be lost when the user would set B. This is a destructive operation, should the user be warned about it? Since (confirm this?) it is a non-destructive operation in earlier versions of Moodle, users might rely that it is so also in new versions.

The way it currently works

• Shuffle off, questions per page on: C -> explicitly repaginating with A still possible
• Shuffle on, questions per page on: B+C -> explicitly repaginating with A not possible, since the number is fixed in any case
• Questions per page off: none; questions are paged manually and put onto the last page of quiz when being added

Usage scenarios and relationship to features

Three usage scenarios I can think of:

1. User wants to make sure his/her quiz consistently has N questions per page
   1. Subscenario: The user basically does not want to think about it at all: 1. You know from the start you want your quiz to have one-question per page, so you set that on the quiz settings screen when you create the quiz. 2. Then you add 10 questions to the quiz from the question bank.
2. User wants to just add questions and afterwards think about paging: mostly keep a consistent paging scheme but user might want to deviate from it on some pages (for example, first page only has an introductory question, while further pages have a greater quantity)
3. The user needs to set the layout of their questions manually: User already has an intricate paging scheme realized in the quiz, but wants to add further questions with a consistent paging scheme only for the new questions
   1. But if the number of questions to add is big, it is useful to be capable of automatically page the *new* questions while adding them, requiring feature C.

---

In 1, the user is served by either

• adding their questions first and then applying A (and then redoing that after adding further questions)
• having B set, and then adding questions. In this case the user can also set B on after adding the questions

In 2, the user can use A after adding the questions he/she wants, and afterwards make modifications to the paging manually. But 3.1 applies here, too.

Options for a solution

DISCLAIMER: The reasoning was made ENTIRELY WITHOUT user testing or extensive research of further use cases that might be related to this functionality. Still, it seems the best we got for now.

Background

As B and C are modes, and thus their being active should be reflected by the edit page.

The safest model would seem to be having three distinct features. However, I am not at all sure if C is such a special case that it should be eliminated altogether.

If you remove functionality C altogether, that means not supporting the use case the UI currently supports when shuffle questions is off: "In 3, the user can add their questions manually. But if the number of questions to add is big, it is useful to be capable of automatically page the *new* questions while adding them, requiring feature C. ".

So in both options, the goal is this:

	Shuffle off	Shuffle on
Manual paging (=> disable explicit repaginate)	Consistently show questions exactly like edit view suggests	Keep as many questions on pages as edit view suggests, but shuffle order
Questions per page: N	Entire quiz layout with fixed N questions per page, order as in edit view;	Entire quiz layout with fixed N questions per page, shuffled order

Option 1

If you don't want to take the risk of removing functionality, add a separate setting on the quiz settings page for C. This is mutually exclusive with the actual keeping quiz paginated permanently. That is, I would create three radio buttons.

Note that this also has the advantage of simplifying the drop down menus by removing the "Never" or "Unlimited" option, as that option is moved to the first radio button, making the UI more obvious (as you do not have to search the option/remember whether or not that non-numeric option was there or not - Recognition rather than recall in Nielsen's heuristics).

( ) Manual paging
( ) Manual paging, but when I add questions, paginate 
    them with [ v] questions  per page
( ) Keep the entire quiz paginated with [ v] questions per page

The dropdowns that are not with the currently active radio button should be disabled using javascript.

Option 2

The other option, then would seem to be making the functionality work according to how it is worded in 1.9: that is, discard functionality C and make the setting on the settings page only toggle B, independent of shuffle questions - assuming that the dependency with shuffle options has no purpose in the first place.

Tim's thoughts

Thank you for summarising everything an covering so many possibilities. I would not have reached the following suggestion without reading all your options.

Also, I am very aware that I am likely to have a strong sub-conscious bias in favour of how the quiz has always worked, but actually, I still think that the following is right, even though it is only a small change from how the quiz used to work - all your other changes make the way the quiz used to work much more understandable.

I think we need to consider the two cases separately:

Shuffle questions: Yes

In this case, the other settings is "Questions per page: N", or "All questions on one page".

On both the Editing quiz and Order and paging tabs, the list of question should always be shown broken into pages of the specified size, and there should be no options to manually control page breaks. The controls for adding new questions should only appear at the end, possibly it would be clearer if they only appeared outside the grey box for the last page.

I would be inclined to keep the UI for moving questions up and down in the list, and reordering them on the order and paging tab, because getting the list of questions in order may help teachers ensure they have exactly the list of questions they intend in the quiz.

Shuffle questions: No

In this case, we should only support manual paging. That is, basically option C, but with the option of manually doing a repaginate (A).

My logic is that the new UI makes it very clear what the current state of the pagination of the quiz is. Therefore, there is very little value in a 'Keep the entire quiz paginated with [ v] questions per page' mode in the UI, especially if a repaginate is easy to trigger.

In this case, the other setting becomes "New page every N questions" or "Never start a new page".

Suppose it is set to new page every 3 questions. Then the way it should work is, when you add a question to the last page on the quiz, if, as a result, that last page contains three or more question, then we automatically add a blank page at the end of the quiz, as well as adding the question.

More radical suggestion

Actually, the whole problem is that the place you go to change these settings, the quiz settings form, is nowhere near the place where you can see the effect of these settings. That is what we need to change.

There is already a precedent for this, the $quiz->grade field. This can only be set on the quiz editing page.

So, I think that similarly we should remove the shuffle questions and questions per page setting from the quiz settings form, and instead change the line that looks like

Update this Quiz: *Shuffle questions: Yes | Questions per page: 1

To look like:

Shuffle questions: [Yes |v] | New page: [every N questions |v]

or

Shuffle questions: [No |v] | New page: [Never |v] | (Repaginate ...)

Both of those dropdowns will be the sort that submit a form when they are changed, so the new value can be saved, and the display of the questions in the quiz can be updated appropriately.

I have tried to find a consistent wording for the second drop-down, that works for both shuffle options. If necessary, we could have the second dropdown depend on the state of the first.

Do what I said above about how to show the list of questions, and what controls are available in each mode.

Am I making any sense here?