Advanced grading methods: Difference between revisions

Revision as of 23:49, 28 August 2011

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Advanced grading methods
Project state	Planning
Tracker issue	MDL-29108
Discussion	TODO
Assignee	David Mudrak

Moodle 2.2

Project goals

Long-term goal: To be able to use advanced grading methods (like rubrics and other multi-criteria assessment forms) to assess all reasonable places (like assignment submissions, forum posts, wiki pages, database records etc.), not just the submissions in the workshop module.
Short-term goal: To allow using rubric as a grading tool for the assignment module.

Project scope

This project does NOT change the gradebook internals or its API for obtaining and storing grades. It has also nothing to do with other gradebook issues like aggregating hidden grades and friends.

This project aims to:

provide a way to define grading forms
allow activity modules to use defined forms to calculate grades

Things that must be decided and designed carefully:

export/import - how to easily publish forms definitions (XML + embedded files) and re-use them
backup/restore - grading forms must become part of the course module structures (XML structures similar to export/import)

Relevant tracker issues

MDL-795 (44 votes, 27 watchers) - 'workshop-like' teacher assessment methods (rubrics, accumulative, etc) for assignments
MDL-2255 (13 votes, 3 watchers) - Rubrics for all Activities
CONTRIB-2058 - Ability to edit/delete rubrics (Assignment Rubric activity module)

Use case 1: Using rubric in the assignment

When editing the Assignment settings, there is a new field in the "Grade" section called "Grading method" (see the screenshots attached). The teacher can define whether the grade (value or scale) will be put directly by the marker (as it is now) by selecting the grading method "Simple direct grading". Alternatively the teacher can select some advanced grading method to use. She sets the grade for this Assignment to a value 20 and decides to use advanced grading method "Rubric" to calculate that grade. See MDL-29151 for how the mod_form would look like if the Assignment module supported multiple gradable areas.

At the assignment page, the Settings side block contains new node to define the rubric. The teacher clicks at "Rubric editor" link.
The editor allows the teacher to design the rubric from scratch, to import it from XML file or use some existing rubric at the site as a template to start off from. She prepares a rubric consisting of three criteria, each of them having four levels with scores 3, 2, 1, 0. Hence the total score obtained from the rubric will be something between 9 and 0 points. When the teacher is happy with the preview, goes back to the assignment.
The list of submissions does not support the Quick grading mode now. The teacher clicks the "Grade" link at a submission row and is taken to /mod/assignment/submissions.php where the grading of that single submission happens.
Instead of the simple direct grade selector 0-20, the defined rubric is embedded now. The teacher clicks into the cells of the rubric grid to choose the most appropriate assessment of each criteria. Selected cells are highlighted. It is also possible to put a side plain-text remark for each criterion and the overall remark for the whole rubric.

By clicking at the "Update" button, the rubric is saved via AJAX and the displayed grade is replaced with a new calculated value. In this example, the calculated grade value would be 11.1 of the maximum possible grade 20 (the selected cells give the total score 5 out of the 9 possible points: 5 / 9 * 20 = 11.1). There is also a very basic support for browsers with javascript disabled planned for the future: the rubric would open in a pop-up window and the teacher would just use the calculated value from that pop-up to set the grade in the grading screen manually.
Using the link at the top right corner of the rubric, the rubric grid can be hidden to save space on the screen.

By clicking at "Save and show next" the teacher would continue with the next student. The calculated grade is pushed into the gradebook together with the information that the value was obtained from a particular rubric. The grader report is then able to generate a link to a standalone page (or some sort of pop-up like we have for in-built help maybe) that shows the filled rubric.

How it works

The Assignment module declares FEATURE_ADVANCED_GRADING in its lib.php so that standard_grading_coursemodule_elements() can put the new field into mod_assignment_mod_form.
A simple modification of mod_assignment_grading_form class is done so that the form replaces the simple direct grade selector with a new element for the rubric.
As we know what grading method should be used in the given area, the appropriate plugin (here it is rubric) is called and it provides the grading form. If the rubric was not defined yet (or it is being currently edited and it was not released for usage yet), the element displays a message and a link to the rubric editor.
When the teacher fills the rubric and updates it, the rubric plugin is asked to calculate a value representing the result of the assessment. All plugins return normalized value from 0.00000 to 100.00000 for this purpose. This value is returned as a part of the AJAX response and the Assignment module is able to transform it into the grade value
When modules push the grades into the gradebook, they will be able to record the source of the value (probably into two new columns in the grade_grades table with the component name and instanceid). Using this information, the grader report must be able to identify the source plugin and ask it for the URL of a page where the detailed grade analysis is displayed. Note that this part is actually independent on this project and can be developed and used separately. This way, quiz grades in the gradebook could lead to a page in quiz with the detailed analysis, for example.

Highlights

This proposed solution has minimal impact on the current grading logic in modules. Advanced grading forms (like rubrics) are considered as helper tools that are supposed to produce the grade value, as if the marker used some spreadsheet application and then inserted the calculated value into the current grading forms. Changes in the gradebook are minimal (only for the ability to link back to the grade value explanation). This way the project is doable for Moodle 2.2 release.

Implementation plan

New feature called FEATURE_ADVANCED_GRADING to be registered in lib/moodlelib.php
New core subsystem called "grading" (core_grading in frankenstyle) to be added into get_core_subsystems(), located in $CFG->dirroot/grading/
New plugin type "gradingform" to be added into get_plugin_types(), located in $CFG->dirroot/grading/form/. Advanced grading methods are implemented as plugins of this type, having the frankenstyle name like "gradingform_rubric"
The files course/modedit.php, course/moodleform_mod.php are modified so that they add support for selecting the active grading method in module.
The class mod_assignment_grading_form to be modified to support grading form embedding
Rubric editor to be implemented
Rubric renderer and the submitted data handler to be implemented
Gradebook API and the table grade_grades to be extended so that each grade can store the source of the value in a way that allows to display a deeper analysis/overview/more information on the grade (independent on this project, can be used outside the scope of advanced grading).

Database schema

Core scope tables

grading_gradable_areas: identifies areas where advanced grading can happen. For each area, the current active plugin can be set. Example: for the assignment in the use case above, there would be a record in this table with contextid set to the module instance's context, component set to the assignment subtype, areaname set to "submission" and activemethod set to "rubric" (as that is what the teacher selected in mod_form).

grading_form_definitions: each plugin can define one grading form in the given gradable area. That means for the given Assignment, there can be just one rubric defined. If there are more plugins available, the teacher can define one instance per gradable area for each of them and switch between them. This can be used for example to produce a quick initial feedback on the work in progress using one plugin and then switch to another method to grade the final products. This scenario is supported in the Workshop module, where the initial assessment can be done using a lite grading method like "Comments only" and then it can be replaced with heavyweight "Rubric" during the second grading iteration.

grading_form_instances: for each item inside the gradable area, one instance of the defined form is created when the grading form is filled. It holds the information about the user who did the assessment. Note that we do not store the user who is being graded here intentionally. The framework is robust enough that it can be used for any item inside the gradable area. The item itself can be for example a team work - the interpretation and handling is done by modules.

Plugin scope tables

gradeform_rubric_criteria: stores the rows of the rubric grid

gradeform_rubric_levels: stores the columns of the rubric grid

gradeform_rubric_fillings: stores the data of how the rubric is filled by a particular rater

@@ Line 59: / Line 59: @@
 === How it works ===
-I am currently working on a description of the flow - see the [https://cacoo.com/diagrams/fCft1sg26GwGBdVD UML sequence diagram] (work in progress). Just to quickly summarize now:
+* The Assignment module declares FEATURE_ADVANCED_GRADING in its lib.php so that standard_grading_coursemodule_elements() can put the new field into mod_assignment_mod_form.
+* A simple modification of mod_assignment_grading_form class is done so that the form replaces the simple direct grade selector with a new element for the rubric.
-* The Assignment module must declare FEATURE_GRADING_FORMS in its lib.php so that standard_grading_coursemodule_elements() can put the new fields into mod_assignment_mod_form.
+* As we know what grading method should be used in the given area, the appropriate plugin (here it is rubric) is called and it provides the grading form. If the rubric was not defined yet (or it is being currently edited and it was not released for usage yet), the element displays a message and a link to the rubric editor.
-* A simple if-condition in /mod/assignment/submissions.php determines whether advanced grading methods should be used or not. If not, the module works the same as it does now. If grading methods are enabled, the link leads to a page /grade/form/grading.php passing the information about the graded submission via URL parameters contextid, component, grading area name and item id. In the case of assignment, the context is module instance's context, the component is the assignment subtype (written in frankenstyle), the grading area is "submission" and item id is the submission id.
+* When the teacher fills the rubric and updates it, the rubric plugin is asked to calculate a value representing the result of the assessment. All plugins return normalized value from 0.00000 to 100.00000 for this purpose. This value is returned as a part of the AJAX response and the Assignment module is able to transform it into the grade value
-* The page /grade/form/grading.php uses a callback back to the assignment to obtain information about the graded item (ie submission in our case). The callback function will return the HTML to print at the grading page and some other data so that the rading page may actually look as if it was part of the assignment module scope.
+* When modules push the grades into the gradebook, they will be able to record the source of the value (probably into two new columns in the grade_grades table with the component name and instanceid). Using this information, the grader report must be able to identify the source plugin and ask it for the URL of a page where the detailed grade analysis is displayed. Note that this part is actually independent on this project and can be developed and used separately. This way, quiz grades in the gradebook could lead to a page in quiz with the detailed analysis, for example.
-* As we know what grading method should be used in the given area, the appropriate plugin (here it is rubric) is called and it provides the grading form. If the rubric was not defined yet (or it is being currently edited and it was not released for usage yet), the page displays a message and a link to the rubric editor.
-* When the teacher fills the rubric and submits it, the rubric plugin is asked to calculate a value representing the result of the assessment. All plugins return normalized value from 0.00000 to 100.00000 here. This value is sent back to the assignment module via another callback function so that it can handle it. In our scenario, the handler would convert the normalized score to the assignment grade (0 - 20 in this use case) and processes it in the same way as it does now. As a part of the response, the callback handler returns URL where the user should be taken to after finishing one assessment. In our case it would be the page with the list of all submissions.
-* When modules push the grades into the gradebook, they will be able to record the source of the value (probably into two new columns in the grade_grades table with the component name and instanceid). Using this information, the grader report must be able to identify the source plugin and ask it for the URL of a page where the detailed grade analysis is displayed. Note that this part is actually independent on this project and can be developed and used separately. This way, quiz grades in the gradebook could lead to a page in quiz with the detailed analysis.
 === Highlights ===
@@ Line 74: / Line 71: @@
 == Implementation plan ==
-* New core plugin type called "Grading method" to be added into get_plugin_types() as 'grademethod' with the location in /grade/method/*
+* New feature called FEATURE_ADVANCED_GRADING to be registered in lib/moodlelib.php
-* All grading methods to be implemented as plugins - "Rubric", "Multi-scale" (aka Accumulative grading strategy in the Workshop), "Number of errors" etc.
+* New core subsystem called "grading" (core_grading in frankenstyle) to be added into get_core_subsystems(), located in $CFG->dirroot/grading/
+* New plugin type "gradingform" to be added into get_plugin_types(), located in $CFG->dirroot/grading/form/. Advanced grading methods are implemented as plugins of this type, having the frankenstyle name like "gradingform_rubric"
+* The files course/modedit.php, course/moodleform_mod.php are modified so that they add support for selecting the active grading method in module.
+* The class mod_assignment_grading_form to be modified to support grading form embedding
+* Rubric editor to be implemented
+* Rubric renderer and the submitted data handler to be implemented
+* Gradebook API and the table grade_grades to be extended so that each grade can store the source of the value in a way that allows to display a deeper analysis/overview/more information on the grade (independent on this project, can be used outside the scope of advanced grading).
 === Database schema ===
@@ Line 84: / Line 87: @@
 ==== Core scope tables ====
-; '''grade_gradable_areas''' : identifies areas where advanced grading can happen. For each area, the current active plugin can be set. Example: for the assignment in the use case above, there would be a record in this table with contextid set to the module instance's context, component set to the assignment subtype, areaname set to "submission" and activemethod set to "rubric" (as that is what the teacher selected in mod_form).
+; '''grading_gradable_areas''' : identifies areas where advanced grading can happen. For each area, the current active plugin can be set. Example: for the assignment in the use case above, there would be a record in this table with contextid set to the module instance's context, component set to the assignment subtype, areaname set to "submission" and activemethod set to "rubric" (as that is what the teacher selected in mod_form).
-; '''grade_form_definitions''' : each plugin can define one grading form in the given gradable area. That means for the given Assignment, there can be just one rubric defined. If there are more plugins available, the teacher can define one instance per gradable area for each of them and switch between them. This can be used for example to produce a quick initial feedback on the work in progress using one plugin and then switch to another method to grade the final products. This scenario is supported in the Workshop module, where the initial assessment can be done using a lite grading method like "Comments only" and then it can be replaced with heavyweight "Rubric" during the second grading iteration.
+; '''grading_form_definitions''' : each plugin can define one grading form in the given gradable area. That means for the given Assignment, there can be just one rubric defined. If there are more plugins available, the teacher can define one instance per gradable area for each of them and switch between them. This can be used for example to produce a quick initial feedback on the work in progress using one plugin and then switch to another method to grade the final products. This scenario is supported in the Workshop module, where the initial assessment can be done using a lite grading method like "Comments only" and then it can be replaced with heavyweight "Rubric" during the second grading iteration.
-; '''grade_form_instances''' : for each item inside the gradable area, one instance of the defined form is created when the grading form is filled. It holds the information about the user who did the assessment. Note that we do not store the user who is being graded here intentionally. The framework is robust enough that it can be used for any ''item'' inside the gradable area. The item itself can be for example a team work - the interpretation and handling is done by modules.
+; '''grading_form_instances''' : for each item inside the gradable area, one instance of the defined form is created when the grading form is filled. It holds the information about the user who did the assessment. Note that we do not store the user who is being graded here intentionally. The framework is robust enough that it can be used for any ''item'' inside the gradable area. The item itself can be for example a team work - the interpretation and handling is done by modules.
 ==== Plugin scope tables ====

Documentation