https://docs.moodle.org/dev/api.php?action=feedcontributions&feedformat=atom&user=GrettewilkinsonMoodleDocs - User contributions [en]2026-06-26T17:55:23ZUser contributionsMediaWiki 1.43.5https://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=35013Assignment Advanced Marking2012-08-22T02:13:47Z<p>Grettewilkinson: /* Support the marking process */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals, rationale and fundamentals =<br />
<br />
The main goals of this change are to:<br />
<br />
* Control the distribution<br />
* Support the marking process<br />
* Control the release of marking to students<br />
<br />
<br />
== Control the distribution ==<br />
<br />
This is relevant in the context of large courses where groups/groupings are used to divide the students into groups and to associate dedicated staff with these groups. In terms of assignment marking the idea is that staff associated with groups are responsible for the marking of students in their groups and fulfil such marking independently from the marking of other groups. <br />
<br />
An example would be: A course with 300 students is divided into 3 groups of students (based on different delivery locations, modes, tutorial groups, or other criteria). Two or more staff are associated with each group (one teacher and several teaching assistants). The same assignment is setup for the whole course. The staff associated with each group of students are responsible for marking their groups. <br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) staff associated with a group should only see the students belonging to this group. It is not recorded which staff member is marking which students. <br />
The staff member in charge of the overall course might have a setting, which allows to see assignment marking for all students (of all groups).<br />
<br />
This functionality is primarily of interest to the EAM project.<br />
<br />
== Support the marking process ==<br />
<br />
The focus here is on assisting with the marking process, typically in situations where multiple teachers and/or teaching assistants form a marking team. The idea is to allow each individual marker to have an overview of marking assigned to them and of the status this marking is in. The teacher in charge of the assignment wants to be able to see how marking is progressing and what quality the marking is (per marker and across markers). <br />
<br />
To achieve this focus two aspects need to be introduced:<br />
* Marker allocations;<br />
* Marking statuses.<br />
<br />
With marker allocations, it is recorded which marker (teacher or teaching assistant) is marking which student. Only the marker allocated can mark a student. The teacher in charge of the assignment decides the marker allocation (the teacher selects all staff involved in marking and then distributes marking manually, randomly, or even based on groups). As the marking progresses, allocations can be changed (e.g., when a marker becomes unavailable).<br />
<br />
GW - my concern with the paragraph above is that it implies that the processes of allocating manually, randomly or by groups will all operate identically. It is my understanding that the spec below is suggesting that manual allocation happens within the assignment grading table, whereas random allocation will effectively occur via the use of groups and will take advantage of the random allocation mechanisms already built into the group manager in Moodle. Likewise allocating by groups will take advantage of the same existing Moodle functionality. It was my impression that changing allocations during the marking process if groups (including random groups) are used, will require changes to the group membership. This may be a minor point, but I think it is worth highlighting.<br />
<br />
EH - Grette, putting implementation issues aside, do you agree with the functionality proposed and importantly, with the fundamental difference to what I have called 'distribution of marking'? If we can agree on the fundamentals first we can then see how the implementation will work. What is currently described below (e.g., with the screenshots) suffers, in my opinion, from confusion about what we want to achieve.<br />
<br />
GW - I guess I am finding it difficult to see what the difference between allocation and distribution is in reference to identifying which marker will mark which students work. I am unsure what the implications of highlighting the difference will be. <br />
<br />
The following marking statuses are suggested:<br />
* Allocated (the marker has not yet started)<br />
* In marking (the marker has started but not yet finished)<br />
* Marking completed (the marker has finished but might need to go back for checking/corrections)<br />
* In review (the marking is now with the teacher in charge for quality checking)<br />
* Ready for release (the teacher in charge is satisfied with the marking but wait before giving students access to the marking)<br />
* Released (students have access to the marking.<br />
<br />
The marking statuses are for use by staff. The marking statuses are not visible to students.<br />
The default setting will be that students can see who has marked them (but this can be changed).<br />
Marking can be taken from one status to the next on individual student basis, yet there are also bulk transitions (e.g., to release of ‘ready’ marking). Marking can be taken back into a previous status (e.g., from in review back to in marking if more work is required).<br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) it is displayed who is marking which student (and only the staff assigned can edit the marking).<br />
<br />
The functionality related to the marking process has its origins in Lightwork and is primarily of interest to the Lightwork online project.<br />
<br />
GW - EAM are also interested in manual allocation of marking as an addition to using groups.<br />
<br />
EH: Are you talking about manual distribution of subsets of students to individual markers/several markers (with processes like I have described in the first point about 'Distribution', or, are you talking about supporting the marking process?<br />
GW - see my comment above about not being clear on the reason for the distinction between allocation and distribution.<br />
<br />
== Control the release of marking to students ==<br />
<br />
A problem with the current Moodle assignment module (2.3) is that students can immediately see their marking results (if the teacher does not take extra steps for hiding the assignment from student view and hiding the grade book entry). This is not good, as it does not allow the teacher to make changes (e.g., as required to deal with marker drift or for checking across multiple markers). <br />
<br />
The suggestion is to add a release control to the assignment module. This will have three possibilities, visible in the assignment marking overview screen (mod/assign/view.php?id=3&action=grading):<br />
* Nothing shown (no-one has yet saved any marking for a student)<br />
* Not released (someone has saved marking, but this marking is not yet visible to the student)<br />
* Released (marking is visible to the student)<br />
<br />
The sole focus here is on if the student can see the marking or not. This is not about trying to manage the marking process.<br />
<br />
This functionality is already included in supporting the marking process as proposed by the Lightwork Online project. The EAM project is interested in this functionality as well.<br />
<br />
PC: A concern that I have with adding a release control to the assignment module is that teachers already have a form of release control (hiding the grade book entry). It might be simpler if we could use this existing mechanism instead of adding a second release control that may confuse teachers.<br />
<br />
GW - my sense is that using the hide/show controls for the gradebook entry requires teachers to move to a different part of Moodle in order to ensure that students do not receive their grades/feedback immediately. A function that sits within the assignment module itself will be more visible and hopefully therefore more usable. We should definitely not build something however that prevents teachers from using the 'hide/show gradebook entry' method.<br />
<br />
EH: I think that the additional functionality would be well placed. Hiding marking from students needs to occur in two places: in the gradebook and in the assignment activity. Hiding the gragebook entry is fairly easy for the teacher (but, as Grette points out, it is in a different place in Moodle). Hiding the assignment activity has unwelcome side effects. As I understand it, the teacher needs to hide the assignment from student view so students cannot see marks and feedback. But, for example, this means that students cannot make late submissions (as they cannot see the assignment ...).<br />
<br />
== How do these functionalities combine for the user and in implementation? ==<br />
<br />
The functionalities would result in three settings to the Assignment update page (course/modedit.php?update=3&return=1).<br />
<br />
* Control release of marking to students (Yes/No)<br />
* Set distribution of marking (Yes/No; with Yes offering selection of groups/groupings)<br />
* Manage marking process (Yes/No; with Yes offering allocation algorithms/manual allocation)<br />
<br />
Controlling the release of marking to students is not in conflict with setting the distribution of marking, and is already included in managing the marking process.<br />
<br />
Fundamentally there is no conflict between setting the distribution of marking and managing the marking process (in fact, it would be welcome as a ‘group’ will likely contain a large number of students and possibly several markers, making it very useful to be able to manage their marking). Yet, an initial implementation might keep things more simple and make these setting mutually exclusive.<br />
<br />
The distribution of marking will build on already existing groups and groupings. No further database tables should be necessary to capture this information. Any changes to groups would be reflected during the marking process.<br />
<br />
Capturing the marking allocations will require new database tables. If one allows building the allocation on groups, this will mean that the initial setting for the allocations is taken from the groups, but that any changes to groups will not be automatically reflected in marker allocations (but can be done manually, like any changes to allocations).<br />
GW - this would be highly problematic for groups that get their membership from Student Information Systems.<br />
<br />
= Overview =<br />
<br />
''(The following sections currently do not match what is proposed under Goals above.)''<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
PC - My concern is that being able to choose a combination of settings in this way does not make it clear to the teacher what they will end up with at the end. For instance, a teacher who wants to take advantage of the advanced marking process should not have to work out that they have to select "Release grades immediately - No" in order to do this. Also, see my comment earlier about the use of the "hiding" facility in the grade book to control the release.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Answer from our last meeting : marker allocation tables will be used<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PC: I believe the following statuses would be good:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
PC - I don't agree with skipping the "Ready for Release" status since it is an important part of the review process. However, it is possible to make the switching between statuses fast and efficient by having bulk options such as "Release all" that automatically selects marking with one status and moves it to the next.<br />
<br />
Where it would be appropriate to reduce the number of statuses (but possibly not in our first Moodle 2.4 release) is when there is only a <br />
single marker. So, I think we need the following:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
If there are multiple markers added to a group then all markers will be able to mark assignments in that group. <br />
<br />
GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=35010Assignment Advanced Marking2012-08-22T01:02:03Z<p>Grettewilkinson: /* How do these functionalities combine for the user and in implementation? */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals, rationale and fundamentals =<br />
<br />
The main goals of this change are to:<br />
<br />
* Control the distribution<br />
* Support the marking process<br />
* Control the release of marking to students<br />
<br />
<br />
== Control the distribution ==<br />
<br />
This is relevant in the context of large courses where groups/groupings are used to divide the students into groups and to associate dedicated staff with these groups. In terms of assignment marking the idea is that staff associated with groups are responsible for the marking of students in their groups and fulfil such marking independently from the marking of other groups. <br />
<br />
An example would be: A course with 300 students is divided into 3 groups of students (based on different delivery locations, modes, tutorial groups, or other criteria). Two or more staff are associated with each group (one teacher and several teaching assistants). The same assignment is setup for the whole course. The staff associated with each group of students are responsible for marking their groups. <br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) staff associated with a group should only see the students belonging to this group. It is not recorded which staff member is marking which students. <br />
The staff member in charge of the overall course might have a setting, which allows to see assignment marking for all students (of all groups).<br />
<br />
This functionality is primarily of interest to the EAM project.<br />
<br />
== Support the marking process ==<br />
<br />
The focus here is on assisting with the marking process, typically in situations where multiple teachers and/or teaching assistants form a marking team. The idea is to allow each individual marker to have an overview of marking assigned to them and of the status this marking is in. The teacher in charge of the assignment wants to be able to see how marking is progressing and what quality the marking is (per marker and across markers). <br />
<br />
To achieve this focus two aspects need to be introduced:<br />
* Marker allocations;<br />
* Marking statuses.<br />
<br />
With marker allocations, it is recorded which marker (teacher or teaching assistant) is marking which student. Only the marker allocated can mark a student. The teacher in charge of the assignment decides the marker allocation (the teacher selects all staff involved in marking and then distributes marking manually, randomly, or even based on groups). As the marking progresses, allocations can be changed (e.g., when a marker becomes unavailable).<br />
<br />
GW - my concern with the paragraph above is that it implies that the processes of allocating manually, randomly or by groups will all operate identically. It is my understanding that the spec below is suggesting that manual allocation happens within the assignment grading table, whereas random allocation will effectively occur via the use of groups and will take advantage of the random allocation mechanisms already built into the group manager in Moodle. Likewise allocating by groups will take advantage of the same existing Moodle functionality. It was my impression that changing allocations during the marking process if groups (including random groups) are used, will require changes to the group membership. This may be a minor point, but I think it is worth highlighting.<br />
<br />
The following marking statuses are suggested:<br />
* Allocated (the marker has not yet started)<br />
* In marking (the marker has started but not yet finished)<br />
* Marking completed (the marker has finished but might need to go back for checking/corrections)<br />
* In review (the marking is now with the teacher in charge for quality checking)<br />
* Ready for release (the teacher in charge is satisfied with the marking but wait before giving students access to the marking)<br />
* Released (students have access to the marking.<br />
<br />
The marking statuses are for use by staff. The marking statuses are not visible to students.<br />
The default setting will be that students can see who has marked them (but this can be changed).<br />
Marking can be taken from one status to the next on individual student basis, yet there are also bulk transitions (e.g., to release of ‘ready’ marking). Marking can be taken back into a previous status (e.g., from in review back to in marking if more work is required).<br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) it is displayed who is marking which student (and only the staff assigned can edit the marking).<br />
<br />
The functionality related to the marking process has its origins in Lightwork and is primarily of interest to the Lightwork online project.<br />
<br />
GW - EAM are also interested in manual allocation of marking as an addition to using groups.<br />
<br />
== Control the release of marking to students ==<br />
<br />
A problem with the current Moodle assignment module (2.3) is that students can immediately see their marking results (if the teacher does not take extra steps for hiding the assignment from student view and hiding the grade book entry). This is not good, as it does not allow the teacher to make changes (e.g., as required to deal with marker drift or for checking across multiple markers). <br />
<br />
The suggestion is to add a release control to the assignment module. This will have three possibilities, visible in the assignment marking overview screen (mod/assign/view.php?id=3&action=grading):<br />
* Nothing shown (no-one has yet saved any marking for a student)<br />
* Not released (someone has saved marking, but this marking is not yet visible to the student)<br />
* Released (marking is visible to the student)<br />
<br />
The sole focus here is on if the student can see the marking or not. This is not about trying to manage the marking process.<br />
<br />
This functionality is already included in supporting the marking process as proposed by the Lightwork Online project. The EAM project is interested in this functionality as well.<br />
<br />
PC: A concern that I have with adding a release control to the assignment module is that teachers already have a form of release control (hiding the grade book entry). It might be simpler if we could use this existing mechanism instead of adding a second release control that may confuse teachers.<br />
<br />
GW - my sense is that using the hide/show controls for the gradebook entry requires teachers to move to a different part of Moodle in order to ensure that students do not receive their grades/feedback immediately. A function that sits within the assignment module itself will be more visible and hopefully therefore more usable. We should definitely not build something however that prevents teachers from using the 'hide/show gradebook entry' method.<br />
<br />
== How do these functionalities combine for the user and in implementation? ==<br />
<br />
The functionalities would result in three settings to the Assignment update page (course/modedit.php?update=3&return=1).<br />
<br />
* Control release of marking to students (Yes/No)<br />
* Set distribution of marking (Yes/No; with Yes offering selection of groups/groupings)<br />
* Manage marking process (Yes/No; with Yes offering allocation algorithms/manual allocation)<br />
<br />
Controlling the release of marking to students is not in conflict with setting the distribution of marking, and is already included in managing the marking process.<br />
<br />
Fundamentally there is no conflict between setting the distribution of marking and managing the marking process (in fact, it would be welcome as a ‘group’ will likely contain a large number of students and possibly several markers, making it very useful to be able to manage their marking). Yet, an initial implementation might keep things more simple and make these setting mutually exclusive.<br />
<br />
The distribution of marking will build on already existing groups and groupings. No further database tables should be necessary to capture this information. Any changes to groups would be reflected during the marking process.<br />
<br />
Capturing the marking allocations will require new database tables. If one allows building the allocation on groups, this will mean that the initial setting for the allocations is taken from the groups, but that any changes to groups will not be automatically reflected in marker allocations (but can be done manually, like any changes to allocations).<br />
GW - this would be highly problematic for groups that get their membership from Student Information Systems.<br />
<br />
= Overview =<br />
<br />
''(The following sections currently do not match what is proposed under Goals above.)''<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
PC - My concern is that being able to choose a combination of settings in this way does not make it clear to the teacher what they will end up with at the end. For instance, a teacher who wants to take advantage of the advanced marking process should not have to work out that they have to select "Release grades immediately - No" in order to do this. Also, see my comment earlier about the use of the "hiding" facility in the grade book to control the release.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Answer from our last meeting : marker allocation tables will be used<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PC: I believe the following statuses would be good:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
PC - I don't agree with skipping the "Ready for Release" status since it is an important part of the review process. However, it is possible to make the switching between statuses fast and efficient by having bulk options such as "Release all" that automatically selects marking with one status and moves it to the next.<br />
<br />
Where it would be appropriate to reduce the number of statuses (but possibly not in our first Moodle 2.4 release) is when there is only a <br />
single marker. So, I think we need the following:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
If there are multiple markers added to a group then all markers will be able to mark assignments in that group. <br />
<br />
GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=35009Assignment Advanced Marking2012-08-22T00:36:15Z<p>Grettewilkinson: /* Support the marking process */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals, rationale and fundamentals =<br />
<br />
The main goals of this change are to:<br />
<br />
* Control the distribution<br />
* Support the marking process<br />
* Control the release of marking to students<br />
<br />
<br />
== Control the distribution ==<br />
<br />
This is relevant in the context of large courses where groups/groupings are used to divide the students into groups and to associate dedicated staff with these groups. In terms of assignment marking the idea is that staff associated with groups are responsible for the marking of students in their groups and fulfil such marking independently from the marking of other groups. <br />
<br />
An example would be: A course with 300 students is divided into 3 groups of students (based on different delivery locations, modes, tutorial groups, or other criteria). Two or more staff are associated with each group (one teacher and several teaching assistants). The same assignment is setup for the whole course. The staff associated with each group of students are responsible for marking their groups. <br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) staff associated with a group should only see the students belonging to this group. It is not recorded which staff member is marking which students. <br />
The staff member in charge of the overall course might have a setting, which allows to see assignment marking for all students (of all groups).<br />
<br />
This functionality is primarily of interest to the EAM project.<br />
<br />
== Support the marking process ==<br />
<br />
The focus here is on assisting with the marking process, typically in situations where multiple teachers and/or teaching assistants form a marking team. The idea is to allow each individual marker to have an overview of marking assigned to them and of the status this marking is in. The teacher in charge of the assignment wants to be able to see how marking is progressing and what quality the marking is (per marker and across markers). <br />
<br />
To achieve this focus two aspects need to be introduced:<br />
* Marker allocations;<br />
* Marking statuses.<br />
<br />
With marker allocations, it is recorded which marker (teacher or teaching assistant) is marking which student. Only the marker allocated can mark a student. The teacher in charge of the assignment decides the marker allocation (the teacher selects all staff involved in marking and then distributes marking manually, randomly, or even based on groups). As the marking progresses, allocations can be changed (e.g., when a marker becomes unavailable).<br />
<br />
GW - my concern with the paragraph above is that it implies that the processes of allocating manually, randomly or by groups will all operate identically. It is my understanding that the spec below is suggesting that manual allocation happens within the assignment grading table, whereas random allocation will effectively occur via the use of groups and will take advantage of the random allocation mechanisms already built into the group manager in Moodle. Likewise allocating by groups will take advantage of the same existing Moodle functionality. It was my impression that changing allocations during the marking process if groups (including random groups) are used, will require changes to the group membership. This may be a minor point, but I think it is worth highlighting.<br />
<br />
The following marking statuses are suggested:<br />
* Allocated (the marker has not yet started)<br />
* In marking (the marker has started but not yet finished)<br />
* Marking completed (the marker has finished but might need to go back for checking/corrections)<br />
* In review (the marking is now with the teacher in charge for quality checking)<br />
* Ready for release (the teacher in charge is satisfied with the marking but wait before giving students access to the marking)<br />
* Released (students have access to the marking.<br />
<br />
The marking statuses are for use by staff. The marking statuses are not visible to students.<br />
The default setting will be that students can see who has marked them (but this can be changed).<br />
Marking can be taken from one status to the next on individual student basis, yet there are also bulk transitions (e.g., to release of ‘ready’ marking). Marking can be taken back into a previous status (e.g., from in review back to in marking if more work is required).<br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) it is displayed who is marking which student (and only the staff assigned can edit the marking).<br />
<br />
The functionality related to the marking process has its origins in Lightwork and is primarily of interest to the Lightwork online project.<br />
<br />
GW - EAM are also interested in manual allocation of marking as an addition to using groups.<br />
<br />
== Control the release of marking to students ==<br />
<br />
A problem with the current Moodle assignment module (2.3) is that students can immediately see their marking results (if the teacher does not take extra steps for hiding the assignment from student view and hiding the grade book entry). This is not good, as it does not allow the teacher to make changes (e.g., as required to deal with marker drift or for checking across multiple markers). <br />
<br />
The suggestion is to add a release control to the assignment module. This will have three possibilities, visible in the assignment marking overview screen (mod/assign/view.php?id=3&action=grading):<br />
* Nothing shown (no-one has yet saved any marking for a student)<br />
* Not released (someone has saved marking, but this marking is not yet visible to the student)<br />
* Released (marking is visible to the student)<br />
<br />
The sole focus here is on if the student can see the marking or not. This is not about trying to manage the marking process.<br />
<br />
This functionality is already included in supporting the marking process as proposed by the Lightwork Online project. The EAM project is interested in this functionality as well.<br />
<br />
PC: A concern that I have with adding a release control to the assignment module is that teachers already have a form of release control (hiding the grade book entry). It might be simpler if we could use this existing mechanism instead of adding a second release control that may confuse teachers.<br />
<br />
GW - my sense is that using the hide/show controls for the gradebook entry requires teachers to move to a different part of Moodle in order to ensure that students do not receive their grades/feedback immediately. A function that sits within the assignment module itself will be more visible and hopefully therefore more usable. We should definitely not build something however that prevents teachers from using the 'hide/show gradebook entry' method.<br />
<br />
== How do these functionalities combine for the user and in implementation? ==<br />
<br />
The functionalities would result in three settings to the Assignment update page (course/modedit.php?update=3&return=1).<br />
<br />
* Control release of marking to students (Yes/No)<br />
* Set distribution of marking (Yes/No; with Yes offering selection of groups/groupings)<br />
* Manage marking process (Yes/No; with Yes offering allocation algorithms/manual allocation)<br />
<br />
Controlling the release of marking to students is not in conflict with setting the distribution of marking, and is already included in managing the marking process.<br />
<br />
Fundamentally there is no conflict between setting the distribution of marking and managing the marking process (in fact, it would be welcome as a ‘group’ will likely contain a large number of students and possibly several markers, making it very useful to be able to manage their marking). Yet, an initial implementation might keep things more simple and make these setting mutually exclusive.<br />
<br />
The distribution of marking will build on already existing groups and groupings. No further database tables should be necessary to capture this information. Any changes to groups would be reflected during the marking process.<br />
<br />
Capturing the marking allocations will require new database tables. If one allows building the allocation on groups, this will mean that the initial setting for the allocations is taken from the groups, but that any changes to groups will not be automatically reflected in marker allocations (but can be done manually, like any changes to allocations).<br />
<br />
= Overview =<br />
<br />
''(The following sections currently do not match what is proposed under Goals above.)''<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
PC - My concern is that being able to choose a combination of settings in this way does not make it clear to the teacher what they will end up with at the end. For instance, a teacher who wants to take advantage of the advanced marking process should not have to work out that they have to select "Release grades immediately - No" in order to do this. Also, see my comment earlier about the use of the "hiding" facility in the grade book to control the release.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Answer from our last meeting : marker allocation tables will be used<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PC: I believe the following statuses would be good:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
PC - I don't agree with skipping the "Ready for Release" status since it is an important part of the review process. However, it is possible to make the switching between statuses fast and efficient by having bulk options such as "Release all" that automatically selects marking with one status and moves it to the next.<br />
<br />
Where it would be appropriate to reduce the number of statuses (but possibly not in our first Moodle 2.4 release) is when there is only a <br />
single marker. So, I think we need the following:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
If there are multiple markers added to a group then all markers will be able to mark assignments in that group. <br />
<br />
GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=35008Assignment Advanced Marking2012-08-22T00:26:27Z<p>Grettewilkinson: /* Control the release of marking to students */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals, rationale and fundamentals =<br />
<br />
The main goals of this change are to:<br />
<br />
* Control the distribution<br />
* Support the marking process<br />
* Control the release of marking to students<br />
<br />
<br />
== Control the distribution ==<br />
<br />
This is relevant in the context of large courses where groups/groupings are used to divide the students into groups and to associate dedicated staff with these groups. In terms of assignment marking the idea is that staff associated with groups are responsible for the marking of students in their groups and fulfil such marking independently from the marking of other groups. <br />
<br />
An example would be: A course with 300 students is divided into 3 groups of students (based on different delivery locations, modes, tutorial groups, or other criteria). Two or more staff are associated with each group (one teacher and several teaching assistants). The same assignment is setup for the whole course. The staff associated with each group of students are responsible for marking their groups. <br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) staff associated with a group should only see the students belonging to this group. It is not recorded which staff member is marking which students. <br />
The staff member in charge of the overall course might have a setting, which allows to see assignment marking for all students (of all groups).<br />
<br />
This functionality is primarily of interest to the EAM project.<br />
<br />
== Support the marking process ==<br />
<br />
The focus here is on assisting with the marking process, typically in situations where multiple teachers and/or teaching assistants form a marking team. The idea is to allow each individual marker to have an overview of marking assigned to them and of the status this marking is in. The teacher in charge of the assignment wants to be able to see how marking is progressing and what quality the marking is (per marker and across markers). <br />
<br />
To achieve this focus two aspects need to be introduced:<br />
* Marker allocations;<br />
* Marking statuses.<br />
<br />
With marker allocations, it is recorded which marker (teacher or teaching assistant) is marking which student. Only the marker allocated can mark a student. The teacher in charge of the assignment decides the marker allocation (the teacher selects all staff involved in marking and then distributes marking manually, randomly, or even based on groups). As the marking progresses, allocations can be changed (e.g., when a marker becomes unavailable).<br />
<br />
The following marking statuses are suggested:<br />
* Allocated (the marker has not yet started)<br />
* In marking (the marker has started but not yet finished)<br />
* Marking completed (the marker has finished but might need to go back for checking/corrections)<br />
* In review (the marking is now with the teacher in charge for quality checking)<br />
* Ready for release (the teacher in charge is satisfied with the marking but wait before giving students access to the marking)<br />
* Released (students have access to the marking.<br />
<br />
The marking statuses are for use by staff. The marking statuses are not visible to students.<br />
The default setting will be that students can see who has marked them (but this can be changed).<br />
Marking can be taken from one status to the next on individual student basis, yet there are also bulk transitions (e.g., to release of ‘ready’ marking). Marking can be taken back into a previous status (e.g., from in review back to in marking if more work is required).<br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) it is displayed who is marking which student (and only the staff assigned can edit the marking).<br />
<br />
The functionality related to the marking process has its origins in Lightwork and is primarily of interest to the Lightwork online project.<br />
<br />
== Control the release of marking to students ==<br />
<br />
A problem with the current Moodle assignment module (2.3) is that students can immediately see their marking results (if the teacher does not take extra steps for hiding the assignment from student view and hiding the grade book entry). This is not good, as it does not allow the teacher to make changes (e.g., as required to deal with marker drift or for checking across multiple markers). <br />
<br />
The suggestion is to add a release control to the assignment module. This will have three possibilities, visible in the assignment marking overview screen (mod/assign/view.php?id=3&action=grading):<br />
* Nothing shown (no-one has yet saved any marking for a student)<br />
* Not released (someone has saved marking, but this marking is not yet visible to the student)<br />
* Released (marking is visible to the student)<br />
<br />
The sole focus here is on if the student can see the marking or not. This is not about trying to manage the marking process.<br />
<br />
This functionality is already included in supporting the marking process as proposed by the Lightwork Online project. The EAM project is interested in this functionality as well.<br />
<br />
PC: A concern that I have with adding a release control to the assignment module is that teachers already have a form of release control (hiding the grade book entry). It might be simpler if we could use this existing mechanism instead of adding a second release control that may confuse teachers.<br />
<br />
GW - my sense is that using the hide/show controls for the gradebook entry requires teachers to move to a different part of Moodle in order to ensure that students do not receive their grades/feedback immediately. A function that sits within the assignment module itself will be more visible and hopefully therefore more usable. We should definitely not build something however that prevents teachers from using the 'hide/show gradebook entry' method.<br />
<br />
== How do these functionalities combine for the user and in implementation? ==<br />
<br />
The functionalities would result in three settings to the Assignment update page (course/modedit.php?update=3&return=1).<br />
<br />
* Control release of marking to students (Yes/No)<br />
* Set distribution of marking (Yes/No; with Yes offering selection of groups/groupings)<br />
* Manage marking process (Yes/No; with Yes offering allocation algorithms/manual allocation)<br />
<br />
Controlling the release of marking to students is not in conflict with setting the distribution of marking, and is already included in managing the marking process.<br />
<br />
Fundamentally there is no conflict between setting the distribution of marking and managing the marking process (in fact, it would be welcome as a ‘group’ will likely contain a large number of students and possibly several markers, making it very useful to be able to manage their marking). Yet, an initial implementation might keep things more simple and make these setting mutually exclusive.<br />
<br />
The distribution of marking will build on already existing groups and groupings. No further database tables should be necessary to capture this information. Any changes to groups would be reflected during the marking process.<br />
<br />
Capturing the marking allocations will require new database tables. If one allows building the allocation on groups, this will mean that the initial setting for the allocations is taken from the groups, but that any changes to groups will not be automatically reflected in marker allocations (but can be done manually, like any changes to allocations).<br />
<br />
= Overview =<br />
<br />
''(The following sections currently do not match what is proposed under Goals above.)''<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
PC - My concern is that being able to choose a combination of settings in this way does not make it clear to the teacher what they will end up with at the end. For instance, a teacher who wants to take advantage of the advanced marking process should not have to work out that they have to select "Release grades immediately - No" in order to do this. Also, see my comment earlier about the use of the "hiding" facility in the grade book to control the release.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Answer from our last meeting : marker allocation tables will be used<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PC: I believe the following statuses would be good:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
PC - I don't agree with skipping the "Ready for Release" status since it is an important part of the review process. However, it is possible to make the switching between statuses fast and efficient by having bulk options such as "Release all" that automatically selects marking with one status and moves it to the next.<br />
<br />
Where it would be appropriate to reduce the number of statuses (but possibly not in our first Moodle 2.4 release) is when there is only a <br />
single marker. So, I think we need the following:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
If there are multiple markers added to a group then all markers will be able to mark assignments in that group. <br />
<br />
GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=35007Assignment Advanced Marking2012-08-22T00:21:04Z<p>Grettewilkinson: /* Control the distribution */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals, rationale and fundamentals =<br />
<br />
The main goals of this change are to:<br />
<br />
* Control the distribution<br />
* Support the marking process<br />
* Control the release of marking to students<br />
<br />
<br />
== Control the distribution ==<br />
<br />
This is relevant in the context of large courses where groups/groupings are used to divide the students into groups and to associate dedicated staff with these groups. In terms of assignment marking the idea is that staff associated with groups are responsible for the marking of students in their groups and fulfil such marking independently from the marking of other groups. <br />
<br />
An example would be: A course with 300 students is divided into 3 groups of students (based on different delivery locations, modes, tutorial groups, or other criteria). Two or more staff are associated with each group (one teacher and several teaching assistants). The same assignment is setup for the whole course. The staff associated with each group of students are responsible for marking their groups. <br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) staff associated with a group should only see the students belonging to this group. It is not recorded which staff member is marking which students. <br />
The staff member in charge of the overall course might have a setting, which allows to see assignment marking for all students (of all groups).<br />
<br />
This functionality is primarily of interest to the EAM project.<br />
<br />
== Support the marking process ==<br />
<br />
The focus here is on assisting with the marking process, typically in situations where multiple teachers and/or teaching assistants form a marking team. The idea is to allow each individual marker to have an overview of marking assigned to them and of the status this marking is in. The teacher in charge of the assignment wants to be able to see how marking is progressing and what quality the marking is (per marker and across markers). <br />
<br />
To achieve this focus two aspects need to be introduced:<br />
* Marker allocations;<br />
* Marking statuses.<br />
<br />
With marker allocations, it is recorded which marker (teacher or teaching assistant) is marking which student. Only the marker allocated can mark a student. The teacher in charge of the assignment decides the marker allocation (the teacher selects all staff involved in marking and then distributes marking manually, randomly, or even based on groups). As the marking progresses, allocations can be changed (e.g., when a marker becomes unavailable).<br />
<br />
The following marking statuses are suggested:<br />
* Allocated (the marker has not yet started)<br />
* In marking (the marker has started but not yet finished)<br />
* Marking completed (the marker has finished but might need to go back for checking/corrections)<br />
* In review (the marking is now with the teacher in charge for quality checking)<br />
* Ready for release (the teacher in charge is satisfied with the marking but wait before giving students access to the marking)<br />
* Released (students have access to the marking.<br />
<br />
The marking statuses are for use by staff. The marking statuses are not visible to students.<br />
The default setting will be that students can see who has marked them (but this can be changed).<br />
Marking can be taken from one status to the next on individual student basis, yet there are also bulk transitions (e.g., to release of ‘ready’ marking). Marking can be taken back into a previous status (e.g., from in review back to in marking if more work is required).<br />
<br />
When looking at the assignment marking (mod/assign/view.php?id=3&action=grading) it is displayed who is marking which student (and only the staff assigned can edit the marking).<br />
<br />
The functionality related to the marking process has its origins in Lightwork and is primarily of interest to the Lightwork online project.<br />
<br />
== Control the release of marking to students ==<br />
<br />
A problem with the current Moodle assignment module (2.3) is that students can immediately see their marking results (if the teacher does not take extra steps for hiding the assignment from student view and hiding the grade book entry). This is not good, as it does not allow the teacher to make changes (e.g., as required to deal with marker drift or for checking across multiple markers). <br />
<br />
The suggestion is to add a release control to the assignment module. This will have three possibilities, visible in the assignment marking overview screen (mod/assign/view.php?id=3&action=grading):<br />
* Nothing shown (no-one has yet saved any marking for a student)<br />
* Not released (someone has saved marking, but this marking is not yet visible to the student)<br />
* Released (marking is visible to the student)<br />
<br />
The sole focus here is on if the student can see the marking or not. This is not about trying to manage the marking process.<br />
<br />
This functionality is already included in supporting the marking process as proposed by the Lightwork Online project. The EAM project is interested in this functionality as well.<br />
<br />
PC: A concern that I have with adding a release control to the assignment module is that teachers already have a form of release control (hiding the grade book entry). It might be simpler if we could use this existing mechanism instead of adding a second release control that may confuse teachers.<br />
<br />
== How do these functionalities combine for the user and in implementation? ==<br />
<br />
The functionalities would result in three settings to the Assignment update page (course/modedit.php?update=3&return=1).<br />
<br />
* Control release of marking to students (Yes/No)<br />
* Set distribution of marking (Yes/No; with Yes offering selection of groups/groupings)<br />
* Manage marking process (Yes/No; with Yes offering allocation algorithms/manual allocation)<br />
<br />
Controlling the release of marking to students is not in conflict with setting the distribution of marking, and is already included in managing the marking process.<br />
<br />
Fundamentally there is no conflict between setting the distribution of marking and managing the marking process (in fact, it would be welcome as a ‘group’ will likely contain a large number of students and possibly several markers, making it very useful to be able to manage their marking). Yet, an initial implementation might keep things more simple and make these setting mutually exclusive.<br />
<br />
The distribution of marking will build on already existing groups and groupings. No further database tables should be necessary to capture this information. Any changes to groups would be reflected during the marking process.<br />
<br />
Capturing the marking allocations will require new database tables. If one allows building the allocation on groups, this will mean that the initial setting for the allocations is taken from the groups, but that any changes to groups will not be automatically reflected in marker allocations (but can be done manually, like any changes to allocations).<br />
<br />
= Overview =<br />
<br />
''(The following sections currently do not match what is proposed under Goals above.)''<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
PC - My concern is that being able to choose a combination of settings in this way does not make it clear to the teacher what they will end up with at the end. For instance, a teacher who wants to take advantage of the advanced marking process should not have to work out that they have to select "Release grades immediately - No" in order to do this. Also, see my comment earlier about the use of the "hiding" facility in the grade book to control the release.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Answer from our last meeting : marker allocation tables will be used<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PC: I believe the following statuses would be good:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
PC - I don't agree with skipping the "Ready for Release" status since it is an important part of the review process. However, it is possible to make the switching between statuses fast and efficient by having bulk options such as "Release all" that automatically selects marking with one status and moves it to the next.<br />
<br />
Where it would be appropriate to reduce the number of statuses (but possibly not in our first Moodle 2.4 release) is when there is only a <br />
single marker. So, I think we need the following:<br />
<br />
Multiple markers - "Allocated"->"In Marking"->"Marking Completed"->"In Review"->"Ready for Release"->"Released"<br />
<br />
Single marker - "In Marking"->"In Review"->"Released"<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
If there are multiple markers added to a group then all markers will be able to mark assignments in that group. <br />
<br />
GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34927Assignment Advanced Marking2012-08-17T07:22:48Z<p>Grettewilkinson: /* A teacher has assistants helping them mark and wants to review all marks before they are released 2 */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
If there are multiple markers added to a group then all markers will be able to mark assignments in that group. <br />
<br />
GW - If there are no markers in a group, then there is the potential that student assignments would get missed....there is no visual queue that students are unallocated. In addition - our SIS-created groups systematically check for the addition of members who are not in the group in the SIS. Any staff therefore who are added to the groups in Moodle will be removed from the group on the next update...unless we changed our integration script to retain users in SIS-created groups as long as they were not students??<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34926Assignment Advanced Marking2012-08-17T07:19:11Z<p>Grettewilkinson: /* A teacher has assistants helping them mark and wants to review all marks before they are released 1 */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released". If statuses can be skipped..ie. as long as we don't force users through every status, then i don't think it will be a problem to have the five statuses as proposed above:<br />
'Not marked'<br />
'In marking/Being marked'<br />
'Ready for review'<br />
'Ready for release'<br />
'Released'<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34920Assignment Advanced Marking2012-08-17T04:19:26Z<p>Grettewilkinson: /* Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'In Marking' (?) 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34919Assignment Advanced Marking2012-08-17T04:19:06Z<p>Grettewilkinson: /* Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked' 'Ready for review' 'Ready for release' ' Released'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34918Assignment Advanced Marking2012-08-17T04:12:35Z<p>Grettewilkinson: /* Marker allocations */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. (GW - future dev? Implies the need for markers to be able to lock assignments while they are marking them). If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34916Assignment Advanced Marking2012-08-17T03:57:58Z<p>Grettewilkinson: /* Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
'Not marked'<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34915Assignment Advanced Marking2012-08-17T03:40:02Z<p>Grettewilkinson: /* Settings added to the mod_assign settings page */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
GW - I think my preference is for three settings:<br />
Release grades immediately<br />
Allocate markers to students<br />
<br />
and the third<br />
Grouping for marker allocations (to appear only if group allocation is selected)<br />
<br />
I think it may end up being clearer, rather than trying to explain in help text what Simple marking and the Advanced marking options are.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34914Assignment Advanced Marking2012-08-17T02:58:13Z<p>Grettewilkinson: /* Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
EH: Paul, with "Advanced marking - single marker", do you see that an automated marking allocation is made to one person (who has to have the capability to release grades), therefore enforcing that only this person can mark? I think this is a quite different approach to what Damyon is proposing, which basically says 'use marking statuses but do not record/control who is marking'. If I understand correctly what Paul is proposing, this means that one can use marking statuses only if one controls marker allocation. This might be a better solution, as it cuts out variety, simplifying things for the development and the user. We could add a fourth option for using marking statuses without marker allocation, this might give the following options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately, marker allocations are not recorded)<br />
* Advanced marking - no marker allocations" ((Grades are released when status changed to 'Released'. No control who is marking)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released'. Only single marker specified can mark and release)<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released'. Multiple markers work together)<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
PC - How will you be handling marker allocation in the database when "Grouping for allocating markers" isn't selected? Will you be creating new groups to hold the allocations?<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
<br />
PC - How about "Provisional"?<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: The seems to be our 'In marking'<br />
<br />
GW - what about "Ready to mark" - this might be more consistent with the other statuses "Ready for review" and "Ready for release". Is there any relationship between this status and the submission status...ie. if the submission status is "Draft" should the marking status be "Not ready to mark", which changes to "Ready to mark" when the submission status is "submitted"?<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
EH: I am with Grette there - feedback should only be visible to students when marking is considered as finished (otherwise we would be back to the current deficiencies in Moodle)<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34907Assignment Advanced Marking2012-08-17T01:25:11Z<p>Grettewilkinson: /* Separate release of grades and feedback */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
GW: The potential issue with this is that feedback might change during the marking process. Is a student-triggered process (ie. viewing the feedback) to release grades "complicated"?<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34906Assignment Advanced Marking2012-08-17T01:20:44Z<p>Grettewilkinson: /* Batch operations menu */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
GW - will there be a batch operation for "Assign to me?"<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34905Assignment Advanced Marking2012-08-17T01:20:21Z<p>Grettewilkinson: /* A teacher has assistants helping them mark and wants to review all marks before they are released 2 */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
GW - what would happen if there is more than one marker in a group? what would happen if there are no markers in a group?<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34904Assignment Advanced Marking2012-08-17T01:19:06Z<p>Grettewilkinson: /* Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
PC - I believe the assignment settings should be simplified or teachers will find it confusing. I would like to see the "Release grades immediately" and "Allocate markers to students" replaced with a single drop down of standard options:<br />
<br />
* Simple marking" (the default settings - grades are released immediately)<br />
* Advanced marking - single marker" (Grades are released when status changed to 'Released')<br />
* Advanced marking - multiple markers" (Grades are released when status changed to 'Released')<br />
<br />
Note that when choosing "Advanced marking - multiple markers" the teacher would be able choose whether or not to use a Grouping for allocating<br />
markers. If we are going to allow other combinations of "Release grades immediately" and "Allocate markers to students" then we could have an additional menu item to allow teachers to choose their own combination.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
GW - I think I know why this status is "Draft", but I think it might get confused with the submission status in the interface. I think we can come up with a better term..not that I have thought of one yet.<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34902Assignment Advanced Marking2012-08-17T01:12:07Z<p>Grettewilkinson: /* Batch operations menu */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
<br />
GW: Will there be an option for the Teacher with "mod/assign:releasegrades" capability to take all assignments that are in "Ready for Review" to either "Ready for Release" or "Release" without having to individually select them?<br />
<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34901Assignment Advanced Marking2012-08-17T01:09:45Z<p>Grettewilkinson: /* A teacher has assistants helping them mark and wants to review all marks before they are released 1 */</p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
GW - I think that the Teacher should have the option to skip the "Ready for Release" status and take the assignment from "Ready for review" straight to "Released".<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment_Advanced_Marking&diff=34900Assignment Advanced Marking2012-08-17T01:03:58Z<p>Grettewilkinson: </p>
<hr />
<div>This is a proposal for additional marking features in the Assignment module. <br />
<br />
References: https://docs.moodle.org/dev/Lightwork#User_Interface_-_Marker_allocation_and_management<br />
<br />
Tracker Issue: http://tracker.moodle.org/browse/MDL-34902<br />
<br />
= Goals and rationale =<br />
<br />
The main goals of this change are to:<br />
<br />
* Enable dividing the marking of large classes among multiple markers<br />
* Enable markers to review each others marking before releasing marks to students<br />
* Enable markers to review their own marking before releasing marks to students<br />
* Not add any additional complexity for people who do not need these features<br />
<br />
= Overview =<br />
<br />
== Settings added to the mod_assign settings page ==<br />
<br />
Release grades immediately (Checkbox, default to on - this is the current behaviour)<br />
<br />
[Help text] When enabled, grades will be visible to students as soon as they have been entered by a grader (If the grade column is visible in the gradebook). When disabled, any grades that are entered will be only visible to graders until they are approved for release.<br />
<br />
Allocate markers to students (Select list of options, "None - all markers can mark all students", "Manually allocated - markers are manually allocated students to mark", "Groups - Markers can mark students in their own group")<br />
<br />
When the grouping method is chosen above, an additional setting should be enabled to allow a grouping to be specified for the marker allocations.<br />
<br />
Grouping for marker allocations (Select from the available groupings - or None for all groups)<br />
<br />
GW - we currently create marking groups using the groups interface and then set assignment up using "Separate groups" so that markers can filter the list of assignments. One of the issues we have come up against is unenrolled (suspended) students still appearing in the groups interface. We have a proposal for better management of suspended users. We have identified 23 (possibly 25) areas where suspended users are currently an issue. Where possible, we’d like an option to toggle between hiding and showing suspended users, with the default option being hidden. We are looking to resource a change that will be all encompassing and a complete fix that can be contributed back to the Moodle community. Initially we will build for 2.1.6, but will then review for 2.3 towards the end of the year.<br />
<br />
== Changes to grading/grading interface ==<br />
<br />
=== Marking release ===<br />
<br />
If "Release grades immediately" is not enabled, an additional status should be added to the grading status column "Draft/Ready for release/Released". Use background colors as well to visually distinguish the statuses (Blue - draft, Fushia - Ready for Review, Yellow - Ready for Release, Green - Released).<br />
<br />
Anyone with mod/assign:grade capability would be able to move the status from "Draft" to "Ready for Review" via the drop down menu on each submission, or via the batch operations form.<br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" to "Ready for Release". <br />
<br />
Anyone with mod/assign:releasegrade (new) capability would be able to move the grade from either "Draft" or "Ready for Review" or "Ready for Release" to "Released". At this point the grade is pushed to the gradebook and made visible to the student.<br />
<br />
=== Marker allocations ===<br />
<br />
If "Allocate markers to students" is set to "Manually allocated", and the current user has "mod/assign:allocatemarkers" capability, the list of markers allocated to each student needs to be shown in a new column in the grading interface.<br />
<br />
Multiple markers should be able to be allocated to a single student. If the current user has mod/assign:allocategraders permission, a red X next to each markers will allow immediately removing this marker (Ajax if possible - don't refresh the full page)<br />
An option should be added to the menu for each submission and the batch operations menu which takes you to a separate page where you can choose from the users with mod/assign:grade permission to allocate to a student (or multiple students). <br />
<br />
Any student with no markers allocated should have "No markers" in this column and the column background should be shown in a different colour (Red?) to indicate that no-one has been assigned to mark this submission. If the current user has "mod/assign:allocatemarkers" and "mod/assign:grade" capability, there should be an entry in the menu for this submissions for "Assign to me" (preferably AJAX - do not refresh entire page).<br />
<br />
If the a submissions has not been allocated to the current user, the option to grade that submission is removed. <br />
<br />
If the current user does not have "mod/assign:allocatemarkers" capability, the list of submissions will be limited to only show the submissions where the current user has been allocated as a marker.<br />
<br />
== Use Cases ==<br />
<br />
=== Teacher is creating an assignment with a small number of students and they do not wish to use advanced marking ===<br />
* Create an assignment and leave the new settings at their defaults<br />
** Release grades immediately: "checked"<br />
** Allocate markers to students: "None"<br />
=== Teacher is creating an assignment with a small number of students and they wish to mark all submissions, then re-mark several submissions to check for "grading drift" before release ===<br />
(I am defining this as the perception of quality changing over time as the marker marks each assignment in turn - can't find a good reference right now)<br />
EH: This is called 'marker drift'. you'll find a brief description in http://etools.massey.ac.nz/documents/LitReview101006.pdf (p. 19 bottom), as well as references to further literature.<br />
* Teacher has "mod/assign:create", "mod/assign:grade" and "mod/assign:releasegrades" capabilities<br />
* Create an assignment and de-select Release grades immediately<br />
** Release grades immediately: "not checked"<br />
** Allocate markers to students: "None"<br />
* Teacher marks each submission individually. Grades are "Draft" and not released to gradebook or students<br />
* Teacher re-marks a random sample of submissions<br />
* Teacher releases all the grades via the batch operations form<br />
<br />
EH: <br />
<br />
I think we should point out as the main reason for the stepped release of grades via the transition from 'Ready for release' to 'Released' as<br />
* The teacher wants to release marking to all students at the same time.<br />
<br />
Marker drift or any other checking done by one person on their own marking is important. Yes, in this use case the teacher can use the statuses proposed for that, yet the need for this type of checking applies to all markers (teachers and teaching assistants). This is why we have proposed the statuses of 'In marking' and 'Marking finished', before the marking is being passed on the the teacher for further quality review (e.g., to check across markers).<br />
<br />
PS: The descriptions here focus on the release of grades - this portraits a very summative viewpoint; it does not acknowledge the importance of feedback (as in the comments hopefully written to guide the students). This is why I tend to talk about marking and marking results.<br />
<br />
=== Multiple Teachers are marking submissions for the same assignment and they want to know who is marking which assignment ===<br />
* Assignment has been created and there are multiple markers with "mod:assign/grade and mod:assign/allocatemarker" capabilities<br />
* Each marker selects "Assign to me" for the submissions they wish to mark. <br />
* Each marker then provides grades and feedback which is released immediately to students<br />
* The markers can see which students have been allocated a marker and will not end up marking the same student<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 1===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Manually<br />
* The teacher allocates a set of students to each of the Teaching Assistants<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
<br />
EH: It would be nice to have an option on the settings page where the teacher can set if the Teaching Assistants can see the marking done by other markers (not just the grade). Is the current proposal that a Teaching Assistant would see the entries for students not allocated to them (that means that can see the grade given by another Teaching Assistant)? There would have to be a 'read-only' grading option, allowing the Teaching Assistant to view marking done by others but not change it.<br />
DW: No the current proposal is that they would not see any students not allocated to them. <br />
<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
<br />
EH: Here it would be nice if the Teaching Assistant could pass on a message to the Teacher together with the status change.<br />
<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
<br />
EH: How will the screenshot look? Will the Teacher have access to the grading interface even if the Teacher is not allocated to mark (the Teacher will have to look at the marking details, not just at the grade)?<br />
DW: yes they would see the grading interface <br />
If the Teacher changes feedback or grade, will this mean an automated re-allocation to the Teacher as marker? <br />
DW: The teacher would not be able to provide grades or feedback unless they were the allocated marker<br />
Will the student see who has marked them (there probably should be a setting allowing the Teacher to decide)? <br />
DW: This is a big change - it would currently show them the last person to mark them.<br />
We also need to options here to return the marking to the Teaching Assistant (ideally with comments, explaining why) and to allocate another marker (who then will continue with the marking already started - maybe this is already covered).<br />
DW: This is quite an advanced workflow - we need to make sure we are not adding too much complexity (it is not prevented by this interface - it just does not include any statuses for it - ie "Needs allocation to another marker for a second round of marking")<br />
<br />
EH: This description skips the step of the Teacher taking assignments into 'Ready for release' after successful quality checking.<br />
<br />
* The teacher changes the status of the grades to 'Released' when they are satisfied with the grading quality.<br />
<br />
EH: I think this should read: The Teacher changes the status from 'Ready for review' to 'Ready for Release' when they are satisfied with the grading quality. <br />
<br />
Then the next step is: The Teacher changes the status from 'Ready for Release' to 'Released' when they want to release marking to students.<br />
<br />
DW: I have added the "ready for release" status - but this will require good help explaining the difference between the statuses to teachers. A visual flow chart of the statuses would help.<br />
<br />
=== A teacher has assistants helping them mark and wants to review all marks before they are released 2 ===<br />
* Assignment has been created and there are is a teacher with "mod/assign:grade", "mod/assign:allocatemarker" and "mod/assign:releasegrades" capabilities<br />
* There are multiple teaching assistants who have "mod/assign:grade" capability but not "mod/assign:releasegrades" or "mod/assign:allocatemarker" capabilities<br />
* There are groups containing teaching assistants and sets of students contained in a grouping.<br />
* The Teacher configures the assignment to have:<br />
** Release grades immediately : not checked<br />
** Allocate markers to students : Groups<br />
** Grouping for marker allocation : "Name of Grouping"<br />
* The teaching assistants login and can only grade their own students. The grades and feedback are "Draft" and not released to gradebook or students<br />
* When finished, for each submission the teaching assistant changes the status of the grades to "Ready for review"<br />
* The teacher can see which grades/feedback are ready for them to review by the grading status.<br />
* The teacher reviews all grades and feedback that are "Ready for review", making changes if necessary<br />
* The teacher changes the status of the grades to "Released" when they are satisfied with the grading quality.<br />
<br />
=== Multiple markers which to separately mark the same set of students and then a moderator either takes an average or chooses one of the marks to be released ===<br />
Not supported by this proposal<br />
<br />
=== Multiple markers mark one student in sequence ===<br />
<br />
EH: This is about two or more markers marking student work, e.g., marker 1 marks parts 1 and 2, marker 2 marks parts 3 and 4 of an assignment. This could be done with a different form of allocation: marker1 picks a student and then marks their part; the interface shows that marker 1 has done their part, marker2 can now allocate themselves and continue the marking. Once all markers involved (as setup initially, here markers 1 and 2 need to mark) have done their parts the status changes to 'ready for review'.<br />
<br />
This is not part of the current proposal.<br />
<br />
<br />
=== Separate release of grades and feedback ===<br />
<br />
EH: It would be nice to be able to release feedback (for formative purposes) first and marks/grades later (with a time delay or triggered by the Teacher, or, more sophisticated, dependent on the student doing something). <br />
<br />
This is not part of the current proposal.<br />
<br />
DW: I think it should be - but I don't what to make it complicated. I think that either the feedback is visible to the student during the entire marking process and they get the grade at the end - or they see nothing and they get the grades + feedback at the end (we could add those 2 options to the settings page). Note: The submission comments are visible to staff and students for the entire duration and provide 2 way comments between students and staff.<br />
<br />
== User Interface ==<br />
=== Assignment settings screen ===<br />
[[File:assign-advgrading-settings.png| Assignment settings screen]]<br />
=== Submission menu in grading table ===<br />
[[File:assign-advgrading-gradingmenu.png| Submission grading menu]]<br />
=== Batch operations menu ===<br />
[[File:assign-advgrading-gradingbatchmenu.png| Grading batch operations menu]]<br />
=== Marker allocation ===<br />
Notes: The current user is "Grader Gary" so the grade option has disappeared for all submissions that this user is not assigned to mark. The current user also has permission to assign graders so they see additional options in the submission menu.<br />
<br />
[[File:assign-advgrading-marker-allocation.png| Marker allocation]]<br />
<br />
== Database Changes ==</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=32650Assignment2012-03-07T02:34:21Z<p>Grettewilkinson: </p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = In Development<br />
|tracker = http://tracker.moodle.org/browse/MDL-26997<br />
|discussion = http://moodle.org/mod/forum/discuss.php?d=192642 https://github.com/netspotau/moodle-mod_assign<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Relevant Wiki pages<br />
<br />
* [[Assignment Subtypes Combined]]<br />
* [[Assignment Web Services]]<br />
* [[Assignment Offline Marking]]<br />
* [[Assignment Student Submission Receipts]]<br />
* [[Assignment Student Submission Statement]]<br />
* [[Assignment Team Based Submissions]]<br />
* [[Assignment Blind Marking]]<br />
* [[Assignment Submission Date Extensions]]<br />
* [[Assignment Marking Group Support]]<br />
* [[Assignment Send Notification UI Improvements]]<br />
<br />
Relevant Jira Tickets<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written instruction text to students <br />
** Teacher-supplied instruction files to students<br />
** Student-written submission text for teacher<br />
** Student-supplied submission files for teacher<br />
** Teacher-supplied grades (possibly using [[Advanced grading methods]])<br />
** Teacher-supplied text for feedback<br />
** Teacher-supplied feedback files to students (eg documents with inline comments)<br />
<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable blind marking.<br />
**Comment - with reference to MDL-29343 and MDL-30432. I tend to agree to NOT having subfolders and to rename student files. In the vast majority of cases, students submit a single file and having to go into a subfolder to get that file for every student in a large class would be tedious. MDL-29343 mentioned the benefits of not breaking filenames (renaming files that could break references). Whilst what we have done may potentially break references I would expect that references would only be used in a small percentage of submitted assignments. The fix code uses student name as the folder identifier. Names are NOT unique, so the user-id must be included. The choice between subfolders and renamed files is probably a personal preference. Is it possible to make it a choice that the teacher makes at the point of downloading the assignments. Perhaps a blue question mark alongside a checkbox option "create subfolders for each student" that advised to say no if students have uploaded a single file, say yes if the student have uploaded multiple files eg a little website. Re MDL-30432 - we have removed the ability for students to create a folder so haven't come across this issue. I am still not convinced that create a folder is required (aside from uploading a website, in which case a zip file would probably be more appropriate). If someone can provide a use case? (GW)<br />
**unless impractical for performance reasons, it would be useful to let administrators specify a RTF for the template response file, either as a string in the language settings or as an admin setting.<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
** Does this mean that the word count is a field that must be entered by the student? (DW) Yes (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW) Not all institutions require such a declaration so it's essential to be able to turn this off site-wide. (RC)<br />
** Note that this has been implemented in the quiz now, although what was actually done was to make it possible to do this in a plugin, so see https://docs.moodle.org/dev/Quiz_access_rules and https://github.com/timhunt/moodle-quizaccess_honestycheck. It would probably be good if there was some consistency in the UI/message here, if possible, although the people who wanted me to do this for the quiz wanted the declaration at the start of the quiz.<br />
<br />
* When students submit the assignment, a message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message (Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment. <br />
** Comments - The Marker may not be the person setting up the assignment. Does this assume the Marker(s) is(are) known by the time the first assignment is submitted? In which case, who is the default Marker, until otherwise specified? (RC) <br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
** This could be simplified to two options: a. student is sent email immediately after assignment is graded, and b. all students are sent an email at one point after marking is complete. The visibility of the grade+feedback could be tied to this (ie, no feedback is visible until email is sent). (Michael d)<br />
** The content of the notification email should just be a link to direct the student to the feedback visible on the Web. The notification email should not contain the grade or feedback as this can become out of date if the feedback is updated and does not allow feedback attention to be measured. (Michael d)<br />
<br />
* Feedback will be presented to students (this needs to be seriously considered - Michael d)<br />
** Ideally feedback should be presented in a Web form in Moodle. If there is feedback in an annotated document this should be collected from the feedback page.<br />
** Student views of feedback should be recorded in the database so feedback attention can be measured. Attachments downloaded should also be recorded. Even if statistics of feedback attention (if students are viewing feedback) are out of scope, the tables for feedback collection should be established up front.<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time.<br />
** Comment - could this drop down menu be available all the time if the Marker is in one or more Groups, not just when in 'Group Mode'? (RC)<br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a group declares it ready (Send for Marking), or when all students declare it ready?<br />
**My initial reaction was when 1 student says its ready (because that is what we have had previously), On further reflection having all student in the group say "yes - ready" is probably better – makes all students accountable (no comeback to say work submitted wasn’t what they agreed on). In practice I wonder how often an assignment will be late as a result of one student not having gone in and clicked "ready for submission" - so should the teacher be able to see which students of the group have declared that the work is ready for marking. (GW) Yes, and teacher needs to be able to specify a date on which the work will be auto-submitted if not previously declared ready by all students in the group. (RC)<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
** Should definitely integrate with plagiarism plugins. (Michael d)<br />
** Needs option to enable submission of '''drafts''' to text-matching service (Turnitin) and receive originality report, '''prior''' to final submission for assessment. (RC)<br />
<br />
* Blind marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more <del>descriptive</del> obvious statuses)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due date - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
** Since they are not all mutually exclusive, can more than one be highlighted simultaneously? For example, "submitted" and "late"? (RC)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback" so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
* Additional search filters on the grading page such as "not submitted", "dates extended (once this is supported)", more...<br />
<br />
== Use Cases ==<br />
not all of the use cases will be accommodated in the first phase and we'll need to go tough and show which ones will and won't be.<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process (will not be addressed in first phase)===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
Another use case from Mark Andrews in the forum:<br />
* Student submits work<br />
* Marker grades and leaves feedback<br />
* Second marker, checks grades and feedback - makes any amendments<br />
* Second Marker, releases grades to student(s), thus locking the assessment so no changes can be made<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
Name will be mod_assign.<br />
<br />
What is the plan for upgrading existing instances of mod_assignment?<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]<br />
<br />
<br />
[[Category:Project]]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=31022Assignment2011-12-22T00:24:18Z<p>Grettewilkinson: /* Scope */</p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = http://moodle.org/mod/forum/discuss.php?d=192642<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable blind marking.<br />
**Comment - with reference to MDL-29343 and MDL-30432. I tend to agree to NOT having subfolders and to rename student files. In the vast majority of cases, students submit a single file and having to go into a subfolder to get that file for every student in a large class would be tedious. MDL-29343 mentioned the benefits of not breaking filenames (renaming files that could break references). Whilst what we have done may potentially break references I would expect that references would only be used in a small percentage of submitted assignments. The fix code uses student name as the folder identifier. Names are NOT unique, so the user-id must be included. The choice between subfolders and renamed files is probably a personal preference. Is it possible to make it a choice that the teacher makes at the point of downloading the assignments. Perhaps a blue question mark alongside a checkbox option "create subfolders for each student" that advised to say no if students have uploaded a single file, say yes if the student have uploaded multiple files eg a little website. Re MDL-30432 - we have removed the ability for students to create a folder so haven't come across this issue. I am still not convinced that create a folder is required (aside from uploading a website, in which case a zip file would probably be more appropriate). If someone can provide a use case? (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
** Does this mean that the word count is a field that must be entered by the student? (DW) Yes (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
**My initial reaction was when 1 student says its ready (because that is what we have had previously), On further reflection having all student in the group say "yes - ready" is probably better – makes all students accountable (no comeback to say work submitted wasn’t what they agreed on). In practice I wonder how often an assignment will be late as a result of one student not having gone in and clicked "ready for submission" - so should the teacher be able to see which students of the group have declared that the work is ready for marking. (GW)<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Blind marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
Another use case from Mark Andrews in the forum:<br />
* Student submits work<br />
* Marker grades and leaves feedback<br />
* Second marker, checks grades and feedback - makes any amendments<br />
* Second Marker, releases grades to student(s), thus locking the assessment so no changes can be made<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? I think mod_assign is the favourite - add a comment if you disagree.<br />
<br />
What is the plan for upgrading existing instances of mod_assignment?<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=31021Assignment2011-12-22T00:23:48Z<p>Grettewilkinson: /* Scope */</p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = http://moodle.org/mod/forum/discuss.php?d=192642<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable blind marking.<br />
**Comment - with reference to MDL-29343 and MDL-30432. I tend to agree to NOT having subfolders and to rename student files. In the vast majority of cases, students submit a single file and having to go into a subfolder to get that file for every student in a large class would be tedious. MDL-29343 mentioned the benefits of not breaking filenames (renaming files that could break references). Whilst what we have done may potentially break references I would expect that references would only be used in a small percentage of submitted assignments. The fix code uses student name as the folder identifier. Names are NOT unique, so the user-id must be included. The choice between subfolders and renamed files is probably a personal preference. Is it possible to make it a choice that the teacher makes at the point of downloading the assignments. Perhaps a blue question mark alongside a checkbox option "create subfolders for each student" that advised to say no if students have uploaded a single file, say yes if the student have uploaded multiple files eg a little website.<br />
Re MDL-30432 - we have removed the ability for students to create a folder so haven't come across this issue. I am still not convinced that create a folder is required (aside from uploading a website, in which case a zip file would probably be more appropriate). If someone can provide a use case?<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
** Does this mean that the word count is a field that must be entered by the student? (DW) Yes (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
**My initial reaction was when 1 student says its ready (because that is what we have had previously), On further reflection having all student in the group say "yes - ready" is probably better – makes all students accountable (no comeback to say work submitted wasn’t what they agreed on). In practice I wonder how often an assignment will be late as a result of one student not having gone in and clicked "ready for submission" - so should the teacher be able to see which students of the group have declared that the work is ready for marking. (GW)<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Blind marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
Another use case from Mark Andrews in the forum:<br />
* Student submits work<br />
* Marker grades and leaves feedback<br />
* Second marker, checks grades and feedback - makes any amendments<br />
* Second Marker, releases grades to student(s), thus locking the assessment so no changes can be made<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? I think mod_assign is the favourite - add a comment if you disagree.<br />
<br />
What is the plan for upgrading existing instances of mod_assignment?<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=31020Assignment2011-12-21T23:54:42Z<p>Grettewilkinson: /* Scope */</p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = http://moodle.org/mod/forum/discuss.php?d=192642<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
** Does this mean that the word count is a field that must be entered by the student? (DW) Yes (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
**My initial reaction was when 1 student says its ready (because that is what we have had previously), On further reflection having all student in the group say "yes - ready" is probably better – makes all students accountable (no comeback to say work submitted wasn’t what they agreed on). In practice I wonder how often an assignment will be late as a result of one student not having gone in and clicked "ready for submission" - so should the teacher be able to see which students of the group have declared that the work is ready for marking. (GW)<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Blind marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
Another use case from Mark Andrews in the forum:<br />
* Student submits work<br />
* Marker grades and leaves feedback<br />
* Second marker, checks grades and feedback - makes any amendments<br />
* Second Marker, releases grades to student(s), thus locking the assessment so no changes can be made<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? I think mod_assign is the favourite - add a comment if you disagree.<br />
<br />
What is the plan for upgrading existing instances of mod_assignment?<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=31014Assignment2011-12-21T05:05:49Z<p>Grettewilkinson: /* Scope */</p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = http://moodle.org/mod/forum/discuss.php?d=192642<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
** Does this mean that the word count is a field that must be entered by the student? (DW) Yes (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Blind marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
Another use case from Mark Andrews in the forum:<br />
* Student submits work<br />
* Marker grades and leaves feedback<br />
* Second marker, checks grades and feedback - makes any amendments<br />
* Second Marker, releases grades to student(s), thus locking the assessment so no changes can be made<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? I think mod_assign is the favourite - add a comment if you disagree.<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=31013Assignment2011-12-21T04:50:02Z<p>Grettewilkinson: /* Scope */</p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = http://moodle.org/mod/forum/discuss.php?d=192642<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
** Does this mean that the word count is a field that must be entered by the student? (DW) Yes - but should be a setting at the assignment level (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Blind marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
Another use case from Mark Andrews in the forum:<br />
* Student submits work<br />
* Marker grades and leaves feedback<br />
* Second marker, checks grades and feedback - makes any amendments<br />
* Second Marker, releases grades to student(s), thus locking the assessment so no changes can be made<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? I think mod_assign is the favourite - add a comment if you disagree.<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=30980Assignment2011-12-16T04:55:35Z<p>Grettewilkinson: </p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = TBA<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Anonymous marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
*In the assignment grading page, rename "comment" column to "feedback so that it is consistent with the "grade" screens and also clearer that it will be visible to students. Let the width of the comment column increase if other columns are hidden, blocks are docked to make more screen real-estate etc. (GW)<br />
<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? This is just the name for the module folder, etc - users will just see "Assignment". Old assignment module is mod_assignment. Some not very good suggestions are: mod_assignment2, mod_assessment, mod_assign (from Martin in META issue in tracker)<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=30979Assignment2011-12-16T04:52:40Z<p>Grettewilkinson: </p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = TBA<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Anonymous marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
**Comment - this is 'blind' marking...the identity of the author/student is hidden. In anonymous marking the identity of the marker is hidden (to all but the marking manager/head teacher/topic coordinator). Blind marking is higher priority than anonymous marking. (GW)<br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
** The difference is that students would submit text via an uploaded document (eg. word, pdf, excel) and instead of markers having to download and upload again, that they could 'mark-up' on the fly. Possible? (GW)..see http://moodle.org/mod/data/view.php?d=13&rid=2350 (but preferably without the requirement for students to upload a pdf)<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
** Difference between draft and in progress largely semantic.... what happens at the due due - does "in progress" become "submitted" on the due date, probably not? Therefore is "draft" more appropriate as it makes more sense to remain draft past the due date? What about "overdue" instead of "missed", ie. not started and due date past. (GW)<br />
<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
**If/when it does come into scope...possible to block nasty file types (exe, lnk) at a system level and other restrictions at assignment level? (GW)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if blind marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
** If I have interpreted correctly this is similar to locking the return file, ie. having the ability to secure the document so that feedback and grades cannot be tampered with after return. Whilst I appreciate this feature I also appreciate the difficulty in terms of implementing. A copy of the un-altered return file should always exist on the server (which cannot be tampered with?), which may overcome some of the concerns in this area. (GW)<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
**I would refer to this as anonymous marking, not double blind (GW).<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
** Teacher should be able to provide grade and feedback (response) file to students via Moodle (GW)<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous (blind) marking option (Use case 3): if the activity is set to be anonymous (blind) the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
**Allow deleting is currently not operational (GW)<br />
# Student submits a file. **Submits, or uploads? (GW)<br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
**What happens after the due date - will they still be allowed to delete? Should students be allowed to retract once they have clicked "Submit"/"Send for marking"? How will these two settings work together? (GW)<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
** We have had a request to be able to set a word limit on the text submissions (GW)<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
** We have been referring to this type of second marking as 'moderation' (GW).<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
** Issues to consider - do the markers know the identity of the author? Do the markers know the identity of each other?<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
** Online text type doesn't currently have a "Send for Marking" option. Email receipts are sent every time an update is made. Perhaps highlights the need to add a "Send for Marking"/Finalise type option to online text type. Unless the purpose of the assignment is iterative, in which case finalise is not required, but an email receipt shouldn't be sent (GW)<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? This is just the name for the module folder, etc - users will just see "Assignment". Old assignment module is mod_assignment. Some not very good suggestions are: mod_assignment2, mod_assessment, mod_assign (from Martin in META issue in tracker)<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=30976Assignment2011-12-16T04:03:12Z<p>Grettewilkinson: </p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = TBA<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
**Comments - could probably do with some structured usability testing (GW)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
** My understanding of this function is that if the gradebook column is hidden (which we recommend as best practice during marking) that a notification email is not sent. There is nothing in the marking interface that indicates this, therefore markers may be under a false impression that students have been notified that grades and feedback are available. Perhaps some text in the interface along the lines (Notification emails not sent when item hidden in the gradebook). (GW)<br />
**In addition, currently the process of 'showing' the gradebook column does not automatically notify students that grades and feedback are available. Suggestion - when 'showing' something in the gradebook, include an option to "send notification to students that grades and feedback are available"....and related - we have had a suggestion to automatically hide the gradebook column on the due date - however to revise this perhaps it would be better to add this as an option in the assignment (near due date) "Auto-hide gradebook column on due date"...but I am not sure what effect this would have on accepting late submissions...needs more thought. (GW)<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Anonymous marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if anonymous marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous marking option (Use case 3): if the activity is set to be anonymous the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
# Student submits a file. <br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? This is just the name for the module folder, etc - users will just see "Assignment". Old assignment module is mod_assignment. Some not very good suggestions are: mod_assignment2, mod_assessment, mod_assign (from Martin in META issue in tracker)<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Assignment&diff=30972Assignment2011-12-16T01:01:54Z<p>Grettewilkinson: </p>
<hr />
<div>{{Work in progress}}<br />
{{Infobox Project<br />
|name = Assignment module<br />
|state = Scoping<br />
|tracker = TBA<br />
|discussion = TBA<br />
|assignee = Netspot<br />
}}<br />
{{Moodle 2.3}}<br />
<br />
== Project goals ==<br />
To redevelop the Moodle Assignment Module to address common problems around usability and extensibility and to implement web services to ease integration with third party tools, and address common needs of the user community which are currently not met.<br />
<br />
The following four problems have been highlighted requiring attention:<br />
* The four core subtypes are confusing to new users, and are in fact all very similar. (MDL-26997)<br />
* There is no good support for team-based assignments.<br />
* The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks<br />
* Third party tools such as Lightwork and the Moodle mobile app require the use of web services to be able to access all functionality of Moodle including Rubrics ([[Advanced grading methods]])<br />
<br />
== Progress ==<br />
Some information on the progress of this project is here:<br />
[[Talk:Assignment#Anonymous_marking_and_Blind_marking]]<br />
<br />
== Scope ==<br />
* resolution of the four problems identified by HQ<br />
* implementing features desired by the Electronic Assignment Management (EAM) Consortium and the Moodle community as time and funding permits. The project may be split into phases to allow prioritisation of development work. <br />
<br />
The following requirements are ordered in '''descending''' order of priority. the aim is to complete as many of the features as time permits.<br />
<br />
* The four problems may be addressed by collapsing the four assignment subtypes into one assignment type which supports:<br />
** Teacher-written text to students <br />
** Teacher-supplied files to students<br />
** Student-written text for teacher<br />
** Student-supplied files for teacher<br />
The removal of the facility of assignment sub-types will trigger a need for other third-party assignment sub-types to be redeveloped. The "Old Assignment" will probably need to continue to be included in Moodle 2 until at least the next major release to allow this to happen. (eg. Mahara View Assignment)<br />
<br />
* Customisations already made to the Assignment module by Netspot will be included in the new module (notably offline marking): zip&download package with descriptive file names, a template spread sheet for marks and feedback, zip&upload to populate the return files and the gradebook. <br />
<br />
* Offline Marking will consist of an extension to the existing zip&download facility which will include a template grading sheet and a template response file in the zip file. A zip file containing the appropriate file names can then be uploaded again to enter grades, feedback and response files for the students. Privacy of students will need to be taken into account - usernames and Moodle user IDs are secret information in some institutions. These same privacy options may be used to enable anonymous marking.<br />
**Comment - My understanding is that hiding the student identity should be referred to as blind marking. Anonymous marking is when the marker identify is hidden from the student (GW)<br />
<br />
* When students submit an assignment a declaration screen will ask them to confirm that the assignment is their own work (and word count? seems too specific.) <br />
** Comments - word count only really works for the equivalent of the "online" assignment type. It is error prone and therefore, not useful to try and support reading of X different file types to extract the word count for uploaded. Word count is also not always useful - e.g. a programming assignment, or a picture). Suggest an option to show word count for online text only.<br />
**The use case here is about academic integrity, ie. is the word count that the student declares actually accurate? It is currently used on some assignment cover sheets and provides a checking mechanism for markers. It should apply to files uploaded as well on online text. No expectation with file upload that the accuracy is 'checked' by Moodle. (GW)<br />
**Comment on student declaration - can this be made a setting that can be switched on/off at the assignment level? ie. on/off at the institution level and if on at the institution level an additional level of on/off control within an assignment. Use case - not all assignments ask for original work, eg. citation of case law. (GW)<br />
<br />
* When students submit the assignment, an message (messaging or email) is sent to the '''student''' confirming receipt of the submission (MDL-28651, MDL-14855). <br />
** Template for assignment receipt should be editable in the global assignment settings.<br />
<br />
* message Use messaging preferences) to marker upon first submission of an assignment/ upon submission of any assignment/ upon submission of late assignment<br />
<br />
* The submission screen (for students) and the submission overview screen (for teachers) in the 'Old Assignment' module is unclear - the layout, information shown and language used needs to be improved. (partially completed already by work commissioned by Flinders, most important here is a rewording of the various status lines)<br />
<br />
* The "send notification email" setting is confusing to markers, especially if the gradebook column is hidden. This interface needs to be improved.<br />
** Comments - Needs more information<br />
<br />
* The framework of the assignment module should allow for other activity modules to declare themselves "Assignments" such that they can be grouped in the UI if required.<br />
<br />
* The assignment module will support the use of [[Advanced grading methods]] (Rubrics). <br />
<br />
* The assignment activity observes groups and groupings for markers: if in Separate Groups mode, the submission overview screen when viewed as a marker contains a drop-down list which will let the marker choose between showing all submissions from all participants, or any groups that the marker is part of. This way the marker can mark only a particular group at a time. <br />
<br />
* The assignment activity should support groups and groupings for students, such that, if desired, teams of students can submit assignments by sharing uploading to a common file area and entering text into a shared text response (MDL-27250). A grade for a group submission would apply to all students but would allow overriding for individual students. This may be achieved though a new setting in the assignment module.<br />
** (The module would use the Common Module settings for the group membership information. If using Separate Groups then markers will need to be allocated into the groups they are meant to mark.)<br />
** see MDL-27250 attachment: "Team Based Assignments.docx": when will the assignment be sent for marking? When one student in a groiup declares it ready (Send for Marking), or when all students declare it ready?<br />
<br />
* Logging of activity in the assignment module will be even more important than previously if multiple students can modify each others' (common) work: who views the assignment when, who submits which file, who removes/edits which file or text, who views which submission, who downloads which file, who grades which submission, etc.<br />
<br />
* Files submitted by students and then retracted (while in "draft" mode) are logged. <br />
<br />
* A facility will need to be provided (through lib/assignmentlib.php) to convert assignment from the old four-pronged Assignment module to the new Assignment module with the appropriate settings. <br />
The "New Assignment" module will be able to co-exist with the "Old Assignment" module on the same instance (similar to how the new update Quiz is handled). When upgrading a Moodle instance to the first major release which includes the "New Assignment" module, existing assignment activities are not upgraded, but the "Old Assignment" module will be hidden. New assignment activities will be of the "New Assignment" type. Any backups restored will be of the "Old Assignment" type. Import will convert from old to new. <br />
There will be a "Convert All" button in the Administrator settings (similar to how Tim handled the upgrade of the Quiz module) which will convert all assignments on the system to the "New Assignment" type. <br />
The second major release to include the "New Assignment" module will upgrade any remaining assignment activities to the new type and remove the "Old Assignment" module. Any backups restored will also convert from old to new.<br />
<br />
* The Assignment module needs to provide '''web services''' which expose functional areas of the assignment module. Key areas are:<br />
** Student submission<br />
** Student feedback<br />
** Grading<br />
** Advanced grading<br />
<br />
* interface with text-matching software plugins. (TurnItIn)<br />
** Needs to support an API for this - if possible the same API as used currently<br />
<br />
* Anonymous marking masks the identity of the assignment submitter: if this setting is enabled, students are not identified by name or student ID. Markers can not identify the student (unless the student writes their name on the submitted file, which is impossible to prevent). At a specified date the anonymity is lifted. <br />
<br />
* The Assignment module will have facilities for user overrides similar to the quiz module: Extended access date/time for specific students. This extension date needs to be displayed to markers on the submission overview screen. <br />
<br />
* draft mode as it currently exists (but more descriptive statuses)<br />
<br />
* facilitate on-line markup of submitted text (this could be achieved through the use of a rich text editor field.)<br />
** Comment - More information? How is this different to the existing online assignment type?<br />
<br />
* assignment progress status: not started / in progress / late / draft(/resubmit if impemented) / submitted / marked / missed (largely aesthetic) <br />
** Comments - What is the difference between "in progress" and "draft" (for a student). These are not mutually exclusive - an assignment can be late+submitted.<br />
<br />
== Out of Scope ==<br />
<br />
* The module settings page may have a setting in which specific file types can either be blacklisted or whitelisted. (out of scope for the moment)<br />
<br />
* A cover sheet (in RTF or HTML format?) may be included when assignments are downloaded as a zip file. This cover sheet would include Surname, Firstname, Student ID, Username, Due date, Extension date if granted, Subject/unit/topic code and number, subject/unit/topic name, Assignment title, Name of marker (if not using anonymous marking), Name of group, Email address, Date and time submitted, Submission alert (eg. submitted 2 days late). Student details should not be included if anonymous marking selected (student numbers or a secret ID hash will be required for identification purposes) (out of scope for the moment)<br />
<br />
* Grade distribution tables and graphs, similar to quiz module, which also allows filtering by groups (out of scope for the moment)<br />
<br />
* Ability to submit on behalf of a student, by member of teaching team, or person in authority (from EAM wishlist) (out of scope for the moment)<br />
<br />
* Ability to see all the logs for an assignment/student from the grading interface (out of scope for the moment)<br />
<br />
* prevent editing of return file (out of scope as it would have to handle too many file formats)<br />
** Comment - More information?<br />
<br />
* The Assignment module should allow for self-assessment and peer-assessment (from EAM wishlist) - the workshop module may already satisfy this. (out of scope for the moment)<br />
<br />
* option to only release grade after the student has viewed the feedback (out of scope for the moment)<br />
<br />
* option to mask the identity of the marker. Double blind marking (out of scope for the moment)<br />
<br />
* Ability to flag an assignment for review by the marker. (out of scope for the moment)<br />
<br />
* multiple markers to one submission: this would allow multiple markers to mark different parts of an assignment and the mark would be aggregated. (too complicated for now? out of scope for the moment)<br />
<br />
* A method to request an extension which allows the student to attach a medical certificate (from EAM wishlist) (keep out of scope for the time being)<br />
<br />
== Use Cases ==<br />
=== Use Case 1: File Upload Assignment ===<br />
In a traditional use case a teacher may require the students to write an essay and submit the file to Moodle for grading. So far, the "Upload a Single File" and "Advanced Uploading of Files" Assignment types have been used. ("Upload a Single File" does not allow the teacher to upload a response file, whereas "Advanced Uploading" does). <br />
<br />
With editing turned on, the teacher would select "Assignment" from the "Add an Activity" dropdown menu. <br />
The teacher specifies a name, description, available from and available to dates and times and a grade for the assignment. <br />
The teacher specifies the maximum number of files a student is allowed to submit, and the size per file (or all files together?).<br />
<br />
Once the assignment opens, each student opens the assignment activity and uploads their files form their computer. This may be done over several visits. When the required files are uploaded, the student presses a button to indicate this and the assignment is locked from further changes by the student. (THis is the Send for marking/Draft mode behaviour of the Advanced file upload)<br />
<br />
This process described does not have any facility for the teacher to upload a file with instructions for the students, or a data file to process in the assignment. THis is a common request and should be added.<br />
<br />
=== Use Case 2: Assignment starter file ===<br />
The Teacher wishes to provide the students with a starter file (eg. data file) which the assignment is based on. This file should only be visible at a certain date and time. The students will have to work with the file supplied and submit one or more file (eg. written report) to complete the assignment. <br />
<br />
This is similar to Use Case 1 but the teacher uploads the data file as a assignment instruction file when creating the assignment.<br />
<br />
Can be done as attachmnet to description already.<br />
<br />
=== Use Case 3: Anonymous marking ===<br />
# Teacher creates the assignment with normal settings<br />
# Teacher select to use Anonymous marking for this assignment<br />
# Teacher selects the "post" or go live date for the results (this is also the date which the teacher will see student names for assignments)<br />
# Students upload the assignments<br />
# The papers are giving a unique ID for referencing (submissionID doesn't work here as they are only allocated on assignment submission. Students who don't submit at all should also be able to get a grade). This ID could be a hash of assignmentID and userID. In the case of an anonymous group assignment, group names will need to be anonymised as well.<br />
# Teacher / or Non-editing teacher then downloads the file with no access to knowing who the student is just the ID/hash<br />
# Teacher then grades the assignment <br />
# When the post date / go live date arrives, grades are posted to the gradebook and the teacher can now see names in the activity<br />
<br />
=== Use Case 4: Assignment activity that has no Moodle component. (previously Offline Assignment) ===<br />
THis assignmnet type is just a place holder for an assessable activity that doesn't have anything to do with Moodle. It creates a column in the gradebook and does nothing else.<br />
# Teacher creates the assignment with default settings, specifies Assignment name & description.<br />
# Teacher ticks "This is an off-line activity".<br />
# All options relating to upload of files and input of text are deselected. Students will not be able to upload anything. <br />
# Student visits the assignment. Instructions for the assignment are visible if within the date range. There is a notice that this is an offline activity and that the grades will appear in the gradebook.<br />
<br />
=== Use Case 5: Zip & download, offline marking ===<br />
This is a common scenario for teachers of large classes. <br />
# Teacher creates assignment with all the desired parameters, name, description, etc. <br />
# Students submit assignments. <br />
# Teacher opens assignment submission overview screen and clicks the link or button labelled "Download submissions for offline marking"<br />
# A ZIP file is created which contains all files that students have submitted, renamed to a consistent file name which includes the username and the assignment ID. The package also contains a spread sheet with columns for student name, username, grade and feedback, and one row for each student. <br />
#* At some institutions usernames are sensitive private information - there should be an admin setting which allows the administrator to choose which is used as the student identifier in the file names: username, email address, userID or always anonymous (using assignmentID+userID hash)<br />
#* This needs to play nice with the anonymous marking option (Use case 3): if the activity is set to be anonymous the filename and grade sheet will not include the student identifier specified in admin setting, but instead will show hash of assignmentID+userID.<br />
# The teacher is prompted to download the file. <br />
# THe teacher unzips the file to a desired location. <br />
# The teacher grades the submitted files and records the grades in the spread sheet. The teacher may also include a return file which needs to be named in a particular way. <br />
# The teacher zips up the folder with the return files and the spread sheet<br />
# The teacher visits the assignment submission overview screen and clicks the button "upload graded assignments" and uploads the zip file<br />
# THe system unpacks the zip file. Grades are entered in to the assignment and gradebook, return files are made available to students as return files.<br />
<br />
=== Use Case 6: Student uploads assignment, would like to retract file ===<br />
# Assignment is set up according to desired settings, but the "Allow Deleting" setting is set. <br />
# Student submits a file. <br />
# Student decides that the file was not the right file to upload, and clicks a little [x] icon next to the file whch removes it fro mthe submission.<br />
<br />
=== Use Case 7: Students work in teams and submit one common set of files. All students receive the same grade ===<br />
# Students are allocated into groups A, B, C, etc. in the course. (Optional: The groups are part of a grouping)<br />
# Teacher creates assignment. <br />
# Group setting is set to Group mode: "Separate Groups", Grouping if appropriate.<br />
# When the Group mode setting is set to Separate or Visible Groups, the following settings choice becomes visible: <br />
#* (o) Groups only apply for marking (ie. individual assignments, submission overview screen is groupified to ease allocation of marking to different teachers)<br />
#* ( ) Students submit assignment as groups. <br />
# Teacher selects "Students submit assignment as groups"<br />
#* students in a group now share a common submissionm "pool". Every student in the group can upload files, etc.<br />
# Student 1 in group A opens the assignment activity and uploads a file. <br />
# Student 2 in group A opens the assignment activity and sees the file submitted by Student 1. Student 2 can delete the file or upload another file (if max number of files setting permits)<br />
# * *** should there be a "Ready" (Send for marking) button? Should it work the way game setup works in multiplayer computer games? <br />
# One of the students in group A presses the "Submit for marking" button. <br />
# Teacher views the assignment submission overview screen: new column: Group<br />
# When the teacher presses "Grade" for any of the students in group A, the grading screen opens. <br />
#* Grading screen lists all students who are member of Group A and contains the usual Feedback, Response File and Grade inputs. <br />
#* Each of the three inputs also has a new checkbox: [ ] Apply for all group members. <br />
# Teacher enters Feedback, Grade and Response file for the Group submission, and ticks the three checkboxes to apply the feedback and grade to all students in group A. <br />
<br />
(This will need some design work to make sure the Grading screen doesn't get too confusing)<br />
<br />
=== Use Case 8: Students work in teams and submit one common set of files. All students receive individual grades based on their contributions ===<br />
like use case 7 above, however, at step<br />
12. Teacher enters Feedback, Grade and Response file for the Group submission, and DOES NOT tick the three checkboxes to apply the feedback and grade to all students in group A: The feedback is only applied to this student. <br />
13. Teacher clicks "Save and Next in Group". THe Grading popup advances to the next student in group A and the teacher enters feedback and grade for this student.<br />
<br />
=== Use Case 9: Individual assignments, but a teacher only marks assignments in one group ===<br />
<br />
<br />
=== Use Case 10: Online Text ===<br />
* Assignment is set up according to desired settings, but the "Online Text Submissions" setting is enabled.<br />
* Student views assignment submission page. <br />
* Student enters text directly into a rich text editor directly on the assignment submission page. <br />
* If resubmissions are allowed, the rich text editor is populated with their previous submission text.<br />
* Student clicks "Save" to save the submission or "Discard Changes" to reload the previous submission text.<br />
<br />
=== Use Case 11: Second Marking process ===<br />
<br />
There are a number of second marking methods this is one common process.<br />
<br />
The first marker assesses and provides comments and awards a mark<br />
The second marker checks the grading and comments of the first marker to check for fairness and consistency in standards. This includes the calculation of the grade/marks.<br />
<br />
<br />
This may be done for (and option should be available for)<br />
* all the submissions<br />
* a random sample of submissions<br />
* any fails<br />
* firsts<br />
* borderlines<br />
<br />
<br />
There is also another common use case<br />
<br />
Double blind marking <br />
* This is where both markers mark the paper without reference to each other<br />
* They ensure there is a clear audit trail for the rationale of the marks<br />
* they then reconcile the marks through discussion and agree final mark and comments.<br />
<br />
=== Use Case 12: Email Receipt ===<br />
# Teacher sets up assignment activity which has "Draft Mode" or "Send for Marking" enabled. (language to be decided on)<br />
# Student upload files or enters online text. <br />
# Student presses "Send for marking" button (language to be decided on). <br />
# page is presented with the assigment confirmation (as in [[http://moodle.org/mod/forum/discuss.php?d=182752]]): <br />
#: University Name<br />
#: Assignment Submission Confirmation<br />
#: You have successfully submitted your work<br />
#: Student Details<br />
#: Name: {firstname} {lastname}<br />
#: Username: {Username}<br />
#: ID Number: {ID number}<br />
#: Email: {email}<br />
#: Assignment Details<br />
#: Course: {courseshortname+courseidnumber} {course name}<br />
#: Assignment: {assignmenttitle}<br />
#: Due Date: {duedate}<br />
#: If Moodle Groups then:<br />
#: Group: {groupname} (eg. Day/Time of Class/Tutors Name)<br />
#: Submission Details<br />
#: File(s) submitted: {filename(s)}<br />
#: Or if online assignment:<br />
#: Submission Text:<br />
#: {text}<br />
#: Date/Time Submitted: {submitteddate}<br />
#: Submission Alert: {submissionalert} (eg. submitted 5 days early, submitted 2 days late)<br />
# an email is sent to student:<br />
#: Email subject: “Assignment submission confirmation – course short name – Assignment name”<br />
<br />
<br />
A user's profile should contain delivery options for the Assignment Submission Confirmation in the Messaging options.<br />
Administration settings should contain a section in which an administrator can specify the template for the Assignment Submission Confirmation template: should the username, IDnumber, course short name, ,long name, ID number be shown, etc.<br />
<br />
=== Use Case 13: Cover Sheet / Confirmation ===<br />
<br />
=== Use Case 14: Status message ===<br />
<br />
=== Use Case 15: Anonymous marking of Group Assignments ===<br />
<br />
<br />
=== (MDL-10434) Simple Deletion of submitted assignments ===<br />
<br />
== Mock-Ups ==<br />
[[File:Mockup_with_groups_mode.png|600px|thumb|center|Mock-up showing the settings associated with group mode.]]<br />
== Questions ==<br />
What is the name for the new assignment module? This is just the name for the module folder, etc - users will just see "Assignment". Old assignment module is mod_assignment. Some not very good suggestions are: mod_assignment2, mod_assessment, mod_assign (from Martin in META issue in tracker)<br />
<br />
== Implementation Plan ==<br />
Once specifications have been collected the project will be split into phases based on the importance of features - the four problems will be addressed in the first phase.<br />
<br />
<br />
== Database Schema ==<br />
<br />
== Related Tracker Issues ==<br />
This list is not complete...<br />
<br />
* MDL-10434: [http://tracker.moodle.org/browse/MDL-10434 Simple Deletion of submitted assignments]<br />
* MDL-29343: [http://tracker.moodle.org/browse/MDL-29343 Separate folder for each student's work in "Download all assignments as a zip"] - Although feedback from Flinders is that folders in a zip file is too complex<br />
* MDL-30432: [http://tracker.moodle.org/browse/MDL-30432 "Download all assignments as a zip" in assignments don't copy files with same name from the same student] - Needs to be fixed in the naming convention for files in the zip. <br />
* MDL-30481: [http://tracker.moodle.org/browse/MDL-30481 Advanced upload of files displays confusing message before available from date]<br />
* MDL-27250: [http://tracker.moodle.org/browse/MDL-27250 Group/team assignments]<br />
* MDL-28651: [http://tracker.moodle.org/browse/MDL-28651 Assignment Receipt]<br />
* MDL-14855: [http://tracker.moodle.org/browse/MDL-14855 Be able to send confirmation email when student submits an assignment]<br />
* MDL-29400: [http://tracker.moodle.org/browse/MDL-29400 Assignment shown as submitted before student sends files for marking]<br />
* MDL-15641: [http://tracker.moodle.org/browse/MDL-15641 Allows Studendts to self select their groups with max user setting for each group. Includes also a "group assignment" mod]<br />
* MDL-11968: [http://tracker.moodle.org/browse/MDL-11968 Can uploaded assignments be made anonymous?]<br />
* MDL-26982: [http://tracker.moodle.org/browse/MDL-26982 Download All button for online text type assignments]<br />
* MDL-28155: [http://tracker.moodle.org/browse/MDL-28155 Can't implement "download all submissions" feature in non-standard assignment type]<br />
* MDL-28335: [http://tracker.moodle.org/browse/MDL-28335 Bulk upload of feedback files]<br />
* MDL-29419: [http://tracker.moodle.org/browse/MDL-29419 The new UI of advanced uploading file assignment in 2.x is misleading students]<br />
* MDL-29154: [http://tracker.moodle.org/browse/MDL-29154 Optimize group features for assignments]<br />
* MDL-30144: [http://tracker.moodle.org/browse/MDL-30144 "Download all as a zip" contains submissions from unenrolled students]<br />
* MDL-26997: [http://tracker.moodle.org/browse/MDL-26997 META: Refactor all Assignment subtypes into a single activity module]<br />
* MDL-12687: [http://tracker.moodle.org/browse/MDL-12687 Add Submission Time for ALL submitted assignments]<br />
* MDL-30256: [http://tracker.moodle.org/browse/MDL-30256 Upload single file assignment does not permit updating of response files]</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Talk:Assignment&diff=30971Talk:Assignment2011-12-16T00:16:22Z<p>Grettewilkinson: /* Problem 1 */</p>
<hr />
<div>Martin originally wrote: <br />
<br />
===Problem 1===<br />
The four core subtypes we have are really confusing from a GUI point of view for new users, and are in fact all very similar.<br />
<br />
SOLUTION:<br />
Collapse all these four into a single new assignment module that can support any or all of these at once:<br />
* Teacher-written text to students <br />
* Teacher-supplied files to students<br />
* Student-written text for teacher<br />
* Student-supplied files for teacher<br />
<br />
??Will the functionality of the "offline type" be retained??<br />
- Grette<br />
<br />
===Problem 2===<br />
The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks.<br />
<br />
SOLUTION:<br />
# Remove subtypes from the default assignment module and implement all core features as above.<br />
# Allow normal activity modules to declare themselves as "Assignment" modules (similar to resource modules) so that the GUI can group them if required.<br />
# Provide lib/assignmentlib.php with some core classes to help the migration of subtypes to become full modules.<br />
<br />
<br />
I understand "Provide lib/assignmentlib.php with some core classes" in two ways: <br />
- write the new Assignment module in a way that makes it possible to migrate an instance with the old four-pronged Assignment module. All migrated Assignment modules would turn into new Assignment modules with the appropriate settings. <br />
OR<br />
- make available a library with classes so that other modules that currently are not Assignment modules ("hacks") to be upgraded to become Assignment modules as well. Examples for this would be the Mahara View Assignment and BIM.<br />
<br />
<br />
<br />
===Problem 3===<br />
We have no good support for team-based assignments.<br />
<br />
SOLUTION:<br />
<br />
# Support course groups and groupings. <br />
# Add a switch to settings to "Enable team assignments" which will allow users in groups to share one file area and one text response per group. <br />
# Add good logging of changes made by the users.<br />
<br />
<br />
===Problem 4===<br />
Implement web services for all functionality of Moodle to support LightWork and mobile<br />
<br />
== Project meetings ==<br />
<br />
==Assignment Module Redevelopment meetings==<br />
The NetSpot development team will use this space to document notes from meetings as they happen.<br />
===Meeting 1 - 22 Nov 2011===<br />
Project team:<br />
* Mark Drechsler - NetSpot project sponsor<br />
* Damyon Weise - Lead developer<br />
* Victor Herrera - Project Manager<br />
* Ashley Holman - Code Architect, all round good guy<br />
* Minh-Tam Nguyen - Functional Consultant<br />
* Plus future input from the consortium of Australian universities who will support NetSpot's development effort through a range of means. Details to be added soon.<br />
<br />
First step is to establish the functional specification for the 'bare minimum' of changes needed to meet the design goals listed by Moodle HQ. Once we have these we can start to look in more detail at the requirements of the Consortium and see how many of these are feasible and how many overlap with existing Tracker 'wish list' items from the broader community.<br />
<br />
Project team will be meeting regularly in the lead up to Xmas 2011 to try and be ready to hit some development tasks in early 2012.<br />
<br />
== Anonymous marking and Blind marking ==<br />
<br />
both terms "Anonymous marking" and "blind marking" are used in vaious wish lists and dicuments I have been given, with Anonymous marking referring to both hidden student identities and hidden marker identities. <br />
<br />
I believe that the language used here needs to be decided on. Any suggestions?<br />
<br />
I think they both refer to the same thing except that blind marking is more common when talking about "Double Blind" marking (which would be a nice feature!) - Damyon<br />
<br />
My interpretation is:<br />
Blind marking - the identity of the author (or student) is hidden from the marker<br />
Anonymous marking - the identity of the marker is hidden from the student<br />
Double marking - the work can be marked by more than one person (usually two) - identities of all parties is known to all<br />
Double blind marking - the work can be marked by more than one person but the identity of the author/student is hidden from both markers<br />
- Grette</div>Grettewilkinsonhttps://docs.moodle.org/dev/index.php?title=Talk:Assignment&diff=30970Talk:Assignment2011-12-16T00:14:16Z<p>Grettewilkinson: /* Anonymous marking and Blind marking */</p>
<hr />
<div>Martin originally wrote: <br />
<br />
===Problem 1===<br />
The four core subtypes we have are really confusing from a GUI point of view for new users, and are in fact all very similar.<br />
<br />
SOLUTION:<br />
Collapse all these four into a single new assignment module that can support any or all of these at once:<br />
* Teacher-written text to students <br />
* Teacher-supplied files to students<br />
* Student-written text for teacher<br />
* Student-supplied files for teacher<br />
<br />
<br />
===Problem 2===<br />
The support of subtypes makes it slightly easier to write new assignment types but also makes them less flexible, and so people tend to create hacks.<br />
<br />
SOLUTION:<br />
# Remove subtypes from the default assignment module and implement all core features as above.<br />
# Allow normal activity modules to declare themselves as "Assignment" modules (similar to resource modules) so that the GUI can group them if required.<br />
# Provide lib/assignmentlib.php with some core classes to help the migration of subtypes to become full modules.<br />
<br />
<br />
I understand "Provide lib/assignmentlib.php with some core classes" in two ways: <br />
- write the new Assignment module in a way that makes it possible to migrate an instance with the old four-pronged Assignment module. All migrated Assignment modules would turn into new Assignment modules with the appropriate settings. <br />
OR<br />
- make available a library with classes so that other modules that currently are not Assignment modules ("hacks") to be upgraded to become Assignment modules as well. Examples for this would be the Mahara View Assignment and BIM.<br />
<br />
<br />
<br />
===Problem 3===<br />
We have no good support for team-based assignments.<br />
<br />
SOLUTION:<br />
<br />
# Support course groups and groupings. <br />
# Add a switch to settings to "Enable team assignments" which will allow users in groups to share one file area and one text response per group. <br />
# Add good logging of changes made by the users.<br />
<br />
<br />
===Problem 4===<br />
Implement web services for all functionality of Moodle to support LightWork and mobile<br />
<br />
== Project meetings ==<br />
<br />
==Assignment Module Redevelopment meetings==<br />
The NetSpot development team will use this space to document notes from meetings as they happen.<br />
===Meeting 1 - 22 Nov 2011===<br />
Project team:<br />
* Mark Drechsler - NetSpot project sponsor<br />
* Damyon Weise - Lead developer<br />
* Victor Herrera - Project Manager<br />
* Ashley Holman - Code Architect, all round good guy<br />
* Minh-Tam Nguyen - Functional Consultant<br />
* Plus future input from the consortium of Australian universities who will support NetSpot's development effort through a range of means. Details to be added soon.<br />
<br />
First step is to establish the functional specification for the 'bare minimum' of changes needed to meet the design goals listed by Moodle HQ. Once we have these we can start to look in more detail at the requirements of the Consortium and see how many of these are feasible and how many overlap with existing Tracker 'wish list' items from the broader community.<br />
<br />
Project team will be meeting regularly in the lead up to Xmas 2011 to try and be ready to hit some development tasks in early 2012.<br />
<br />
== Anonymous marking and Blind marking ==<br />
<br />
both terms "Anonymous marking" and "blind marking" are used in vaious wish lists and dicuments I have been given, with Anonymous marking referring to both hidden student identities and hidden marker identities. <br />
<br />
I believe that the language used here needs to be decided on. Any suggestions?<br />
<br />
I think they both refer to the same thing except that blind marking is more common when talking about "Double Blind" marking (which would be a nice feature!) - Damyon<br />
<br />
My interpretation is:<br />
Blind marking - the identity of the author (or student) is hidden from the marker<br />
Anonymous marking - the identity of the marker is hidden from the student<br />
Double marking - the work can be marked by more than one person (usually two) - identities of all parties is known to all<br />
Double blind marking - the work can be marked by more than one person but the identity of the author/student is hidden from both markers<br />
- Grette</div>Grettewilkinson