Assignment Advanced Marking

This is a proposal for additional marking features in the Assignment module.

References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management

Tracker Issue: http://tracker.moodle.org/browse/MDL-34902

Goals, rationale and fundamentals

The main goals of this change are to:

• Control the distribution
• Support the marking process
• Control the release of marking to students

Control the distribution

This is relevant in the context of large courses where groups/groupings are used to divide the students into groups and to associate dedicated staff with these groups. In terms of assignment marking the idea is that staff associated with groups are responsible for the marking of students in their groups and fulfil such marking independently from the marking of other groups.

An example would be: A course with 300 students is divided into 3 groups of students (based on different delivery locations or modes). Two or more staff are associated with each group (one teacher and several teaching assistants). The same assignment is setup for the whole course. The staff associated with each group of students are responsible for marking their groups.

When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) staff associated with a group should only see the students belonging to this group. It is not recorded which staff member is marking which students. The staff member in charge of the overall course might have a setting, which allows to see assignment marking for all students (of all groups).

This functionality is primarily of interest to the EAM project.

Support the marking process

The focus here is on assisting with the marking process, typically in situations where multiple teachers and/or teaching assistants form a marking team. The idea is to allow each individual marker to have an overview of marking assigned to them and of the status this marking is in. The teacher in charge of the assignment wants to be able to see how marking is progressing and what quality the marking is (per marker and across markers).

To achieve this focus two aspects need to be introduced:

• Marker allocations;
• Marking statuses.

With marker allocations, it is recorded which marker (teacher or teaching assistant) is marking which student. Only the marker allocated can mark a student. The teacher in charge of the assignment decides the marker allocation (the teacher selects all staff involved in marking and then distributes marking manually, randomly, or even based on groups). As the marking progresses, allocations can be changed (e.g., when a marker becomes unavailable).

The following marking statuses are suggested:

• Allocated (the marker has not yet started)
• In marking (the marker has started but not yet finished)
• Marking completed (the marker has finished but might need to go back for checking/corrections)
• In review (the marking is now with the teacher in charge for quality checking)
• Ready for release (the teacher in charge is satisfied with the marking but wait before giving students access to the marking)
• Released (students have access to the marking.

The marking statuses are for use by staff. The marking statuses are not visible to students. The default setting will be that students can see who has marked them (but this can be changed). Marking can be taken from one status to the next on individual student basis, yet there are also bulk transitions (e.g., to release of ‘ready’ marking). Marking can be taken back into a previous status (e.g., from in review back to in marking if more work is required).

When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) it is displayed who is marking which student (and only the staff assigned can edit the marking).

The functionality related to the marking process has its origins in Lightwork and is primarily of interest to the Lightwork online project.

Control the release of marking to students

A problem with the current Moodle assignment module (2.3) is that students can immediately see their marking results (if the teacher does not take extra steps for hiding the assignment from student view and hiding the grade book entry). This is not good, as it does not allow the teacher to make changes (e.g., as required to deal with marker drift or for checking across multiple markers).

The suggestion is to add a release control to the assignment module. This will have three possibilities, visible in the assignment marking overview screen (mod/assign/view.php?id=3&action=grading):

• Nothing shown (no-one has yet saved any marking for a student)
• Not released (someone has saved marking, but this marking is not yet visible to the student)
• Released (marking is visible to the student)

The sole focus here is on if the student can see the marking or not. This is not about trying to manage the marking process.

This functionality is already included in supporting the marking process as proposed by the Lightwork Online project. The EAM project is interested in this functionality as well.

How do these functionalities combine for the user and in implementation?

The functionalities would result in three settings to the Assignment update page (course/modedit.php?update=3&return=1).

• Control release of marking to students (Yes/No)
• Set distribution of marking (Yes/No; with Yes offering selection of groups/groupings)
• Manage marking process (Yes/No; with Yes offering allocation algorithms/manual allocation)

Controlling the release of marking to students is not in conflict with setting the distribution of marking, and is already included in managing the marking process.

Fundamentally there is no conflict between setting the distribution of marking and managing the marking process (in fact, it would be welcome as a ‘group’ will likely contain a large number of students and possibly several markers, making it very useful to be able to manage their marking). Yet, an initial implementation might keep things more simple and make these setting mutually exclusive.

The distribution of marking will build on already existing groups and groupings. No further database tables should be necessary to capture this information. Any changes to groups would be reflected during the marking process.

Capturing the marking allocations will require new database tables. If one allows building the allocation on groups, this will mean that the initial setting for the allocations is taken from the groups, but that any changes to groups will not be automatically reflected in marker allocations (but can be done manually, like any changes to allocations).

Overview

(The following sections currently do not match what is proposed under Goals above.)

Settings added to the mod_assign settings page

Release grades immediately (Checkbox, default to on - this is the current behaviour)

[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.

Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")

When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.

Grouping for marker allocations (Select from the available groupings - or None for all groups)

GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.

PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:

• Simple marking" (the default settings - grades are released immediately)
• Advanced marking - single marker" (Grades are released when status changed to 'Released')
• Advanced marking - multiple markers" (Grades are released when status changed to 'Released')

Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.

EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:

• Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)
• Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)
• Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)
• Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)

GW - I think my preference is for three settings: Release grades immediately Allocate markers to students

and the third Grouping for marker allocations (to appear only if group allocation is selected)

I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.

Changes to grading/grading interface

Marking release

If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).

Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.

Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release".

Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.

Marker allocations

If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.

Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page) An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students).

Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).

If the a submissions has not been allocated to the current user, the option to grade that submission is removed.

If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.

PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?

Use Cases

Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking

• Create an assignment and leave the new settings at their defaults
  • Release grades immediately: "checked"
  • Allocate markers to students: "None"

Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release

(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now) EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.

• Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities
• Create an assignment and de-select Release grades immediately
  • Release grades immediately: "not checked"
  • Allocate markers to students: "None"
• Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students

GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.

PC - How about "Provisional"?

• Teacher re-marks a random sample of submissions
• Teacher releases all the grades via the batch operations form

EH: The seems to be our 'In marking'

GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?

'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'

EH:

I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as

• The teacher wants to release marking to all students at the same time.

Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).

PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.

Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment

• Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities
• Each marker selects "Assign to me" for the submissions they wish to mark.
• Each marker then provides grades and feedback which is released immediately to students
• The markers can see which students have been allocated a marker and will not end up marking the same student

A teacher has assistants helping them mark and wants to review all marks before they are released 1

• Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities
• There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities
• The Teacher configures the assignment to have:
  • Release grades immediately : not checked
  • Allocate markers to students : Manually
• The teacher allocates a set of students to each of the Teaching Assistants
• The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students

EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it. DW: No the current proposal is that they would not see any students not allocated to them.

• When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"

EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.

• The teacher can see which grades/feedback are ready for them to review by the grading status.
• The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary

EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)? DW: yes they would see the grading interface If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? DW: This is a big change - it would currently show them the last person to mark them. We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered). DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")

EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.

• The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.

EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality.

Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.

DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.

GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above: 'Not marked' 'In marking/Being marked' 'Ready for review' 'Ready for release' 'Released'

A teacher has assistants helping them mark and wants to review all marks before they are released 2

• Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities
• There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities
• There are groups containing teaching assistants and sets of students contained in a grouping.
• The Teacher configures the assignment to have:
  • Release grades immediately : not checked
  • Allocate markers to students : Groups
  • Grouping for marker allocation : "Name of Grouping"
• The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students
• When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"
• The teacher can see which grades/feedback are ready for them to review by the grading status.
• The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary
• The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.

If there are multiple markers added to a group then all markers will be able to mark assignments in that group.

GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??

Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released

Not supported by this proposal

Multiple markers mark one student in sequence

EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.

This is not part of the current proposal.

Separate release of grades and feedback

EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something).

This is not part of the current proposal.

DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.

GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?

EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)

User Interface

Assignment settings screen

Submission menu in grading table

Batch operations menu

GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?

GW - will there be a batch operation for "Assign to me?"

Marker allocation

Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.