Assignment Advanced Marking

Revision as of 00:38, 17 August 2012 by Damyon Wiese (talk | contribs) (Separate release of grades and feedback)

Jump to: navigation, search

This is a proposal for additional marking features in the Assignment module.

References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management

Tracker Issue: http://tracker.moodle.org/browse/MDL-34902

Goals and rationale

The main goals of this change are to:

  • Enable dividing the marking of large classes among multiple markers
  • Enable markers to review each others marking before releasing marks to students
  • Enable markers to review their own marking before releasing marks to students
  • Not add any additional complexity for people who do not need these features

Overview

Settings added to the mod_assign settings page

Release grades immediately (Checkbox, default to on - this is the current behaviour)

[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.

Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")

When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.

Grouping for marker allocations (Select from the available groupings - or None for all groups)

Changes to grading/grading interface

Marking release

If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Yellow - Ready for Release, Green - Released).

Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for release" via the drop down menu on each submission, or via the batch operations form.

Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.

EH: There seem to be some inconsistencies about the statuses: Is what is named 'Draft' here the same as 'Ready for Review' later?

Marker allocations

If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.

Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page) An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students).

Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).

If the a submissions has not been allocated to the current user, the option to grade that submission is removed.

If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.

Use Cases

Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking

  • Create an assignment and leave the new settings at their defaults
    • Release grades immediately: "checked"
    • Allocate markers to students: "None"

Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release

(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now) EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.

  • Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities
  • Create an assignment and de-select Release grades immediately
    • Release grades immediately: "not checked"
    • Allocate markers to students: "None"
  • Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students
  • Teacher re-marks a random sample of submissions
  • Teacher releases all the grades via the batch operations form

EH:

I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as

  • The teacher wants to release marking to all students at the same time.

Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).

PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.

Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment

  • Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities
  • Each marker selects "Assign to me" for the submissions they wish to mark.
  • Each marker then provides grades and feedback which is released immediately to students
  • The markers can see which students have been allocated a marker and will not end up marking the same student

A teacher has assistants helping them mark and wants to review all marks before they are released 1

  • Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities
  • There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities
  • The Teacher configures the assignment to have:
    • Release grades immediately : not checked
    • Allocate markers to students : Manually
  • The teacher allocates a set of students to each of the Teaching Assistants
  • The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students

EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it. DW: No the current proposal is that they would not see any students not allocated to them.

  • When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"

EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.

  • The teacher can see which grades/feedback are ready for them to review by the grading status.
  • The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary

EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)? DW: yes they would see the grading interface If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? DW: This is a big change - it would currently show them the last person to mark them. We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered). DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")

EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.

  • The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.

EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality.

Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.

DW: The downside is that this adds an extra step to the workflow for all marking - when X percent of the time it is not needed (I have no idea what X is). ie - if your only goal is to release all marks at the same time - you now have to mark, -> ready for review -> ready for release -> release. If you have approve permission I guess you could go straight from mark -> ready for release -> release - but it still requires explaining the difference to teachers.

A teacher has assistants helping them mark and wants to review all marks before they are released 2

  • Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities
  • There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities
  • There are groups containing teaching assistants and sets of students contained in a grouping.
  • The Teacher configures the assignment to have:
    • Release grades immediately : not checked
    • Allocate markers to students : Groups
    • Grouping for marker allocation : "Name of Grouping"
  • The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students
  • When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"
  • The teacher can see which grades/feedback are ready for them to review by the grading status.
  • The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary
  • The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.

Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released

Not supported by this proposal

Multiple markers mark one student in sequence

EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.

This is not part of the current proposal.


Separate release of grades and feedback

EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something).

This is not part of the current proposal.

DW: I think it should be

User Interface

Assignment settings screen

Assignment settings screen

Submission menu in grading table

Submission grading menu

Batch operations menu

Grading batch operations menu

Marker allocation

Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.

Marker allocation

Database Changes