Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Assignment Advanced Marking: Difference between revisions

From MoodleDocs
Line 185: Line 185:


GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)


== User Interface ==
== User Interface ==

Revision as of 02:48, 17 August 2012

This is a proposal for additional marking features in the Assignment module.

References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management

Tracker Issue: http://tracker.moodle.org/browse/MDL-34902

Goals and rationale

The main goals of this change are to:

  • Enable dividing the marking of large classes among multiple markers
  • Enable markers to review each others marking before releasing marks to students
  • Enable markers to review their own marking before releasing marks to students
  • Not add any additional complexity for people who do not need these features

Overview

Settings added to the mod_assign settings page

Release grades immediately (Checkbox, default to on - this is the current behaviour)

[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.

Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")

When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.

Grouping for marker allocations (Select from the available groupings - or None for all groups)

GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.

PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:

  • Simple marking" (the default settings - grades are released immediately)
  • Advanced marking - single marker" (Grades are released when status changed to 'Released')
  • Advanced marking - multiple markers" (Grades are released when status changed to 'Released')

Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.

EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:

  • Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)
  • Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)
  • Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)
  • Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)

Changes to grading/grading interface

Marking release

If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).

Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.

Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release".

Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.

Marker allocations

If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.

Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page) An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students).

Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).

If the a submissions has not been allocated to the current user, the option to grade that submission is removed.

If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.

PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?

Use Cases

Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking

  • Create an assignment and leave the new settings at their defaults
    • Release grades immediately: "checked"
    • Allocate markers to students: "None"

Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release

(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now) EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.

  • Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities
  • Create an assignment and de-select Release grades immediately
    • Release grades immediately: "not checked"
    • Allocate markers to students: "None"
  • Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students

GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.

PC - How about "Provisional"?

  • Teacher re-marks a random sample of submissions
  • Teacher releases all the grades via the batch operations form

EH: The seems to be our 'In marking'

EH:

I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as

  • The teacher wants to release marking to all students at the same time.

Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).

PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.

Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment

  • Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities
  • Each marker selects "Assign to me" for the submissions they wish to mark.
  • Each marker then provides grades and feedback which is released immediately to students
  • The markers can see which students have been allocated a marker and will not end up marking the same student

A teacher has assistants helping them mark and wants to review all marks before they are released 1

  • Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities
  • There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities
  • The Teacher configures the assignment to have:
    • Release grades immediately : not checked
    • Allocate markers to students : Manually
  • The teacher allocates a set of students to each of the Teaching Assistants
  • The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students

EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it. DW: No the current proposal is that they would not see any students not allocated to them.

  • When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"

EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.

  • The teacher can see which grades/feedback are ready for them to review by the grading status.
  • The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary

EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)? DW: yes they would see the grading interface If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? DW: This is a big change - it would currently show them the last person to mark them. We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered). DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")

EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.

  • The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.

EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality.

Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.

DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.

GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".

A teacher has assistants helping them mark and wants to review all marks before they are released 2

  • Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities
  • There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities
  • There are groups containing teaching assistants and sets of students contained in a grouping.
  • The Teacher configures the assignment to have:
    • Release grades immediately : not checked
    • Allocate markers to students : Groups
    • Grouping for marker allocation : "Name of Grouping"
  • The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students
  • When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"
  • The teacher can see which grades/feedback are ready for them to review by the grading status.
  • The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary
  • The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.

GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?

Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released

Not supported by this proposal

Multiple markers mark one student in sequence

EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.

This is not part of the current proposal.


Separate release of grades and feedback

EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something).

This is not part of the current proposal.

DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.

GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?

EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)

User Interface

Assignment settings screen

Assignment settings screen

Submission menu in grading table

Submission grading menu

Batch operations menu

Grading batch operations menu

GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?

GW - will there be a batch operation for "Assign to me?"

Marker allocation

Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.

Marker allocation

Database Changes