Advanced grading methods

Advanced grading methods
Project state	In production since 2.2
Tracker issue	MDL-29108
Discussion	[1]
Assignee	Marina Glancy and David Mudrak

Moodle 2.2

Warning This page contains the original specification. Significant changes were done during the development. The database schema is still valid but the overall workflow is different in the final implemented version.

Project goals

Long-term goal: To be able to use advanced grading methods (like rubrics and other multi-criteria assessment forms) to assess all reasonable places (like assignment submissions, forum posts, wiki pages, database records etc.), not just the submissions in the workshop module.
Short-term goal: To allow using rubric as a grading tool for the assignment module.

Project scope

This project does NOT change the gradebook internals or its API for obtaining and storing grades. It has also nothing to do with other gradebook issues like aggregating hidden grades and friends.

This project aims to:

provide a way to define grading forms
allow activity modules to use defined forms to calculate grades

Things that must be decided and designed carefully:

export/import - how to easily publish forms definitions (XML + embedded files) and re-use them
backup/restore - grading forms must become part of the course module structures (XML structures similar to export/import)

Relevant tracker issues

MDL-795 (44 votes, 27 watchers) - 'workshop-like' teacher assessment methods (rubrics, accumulative, etc) for assignments
MDL-2255 (13 votes, 3 watchers) - Rubrics for all Activities
CONTRIB-2058 - Ability to edit/delete rubrics (Assignment Rubric activity module)

Use case 1: Using rubric in the assignment

When editing the Assignment settings, there is a new field in the "Grade" section called "Grading method" (see the screenshots attached). The teacher can define whether the grade (value or scale) will be put directly by the marker (as it is now) by selecting the grading method "Simple direct grading". Alternatively the teacher can select some advanced grading method to use. She sets the grade for this Assignment to a value 20 and decides to use advanced grading method "Rubric" to calculate that grade. See MDL-29151 for how the mod_form would look like if the Assignment module supported multiple gradable areas.

Preview of the current development version of the Assignment settings form with the advanced grading method selector available

At the assignment page, the Settings side block contains new node to define the rubric. The teacher clicks at "Rubric editor" link.
The editor allows the teacher to design the rubric from scratch, to import it from XML file or use some existing rubric at the site as a template to start off from. She prepares a rubric consisting of three criteria, each of them having four levels with scores 3, 2, 1, 0. Hence the total score obtained from the rubric will be something between 9 and 0 points. When the teacher is happy with the preview, goes back to the assignment.
The list of submissions does not support the Quick grading mode now. The teacher clicks the "Grade" link at a submission row and is taken to /mod/assignment/submissions.php where the grading of that single submission happens.
Instead of the simple direct grade selector 0-20, the defined rubric is embedded now. The teacher clicks into the cells of the rubric grid to choose the most appropriate assessment of each criteria. Selected cells are highlighted. It is also possible to put a side plain-text remark for each criterion and the overall remark for the whole rubric.

Rubric embedded into the submission grading page

On each change of the rubric, it is saved via AJAX and the displayed grade is replaced with a new calculated value (so the teacher can see how the total calculated grade change on the fly). In this example, the finally displayed calculated grade value would be 11.1 of the maximum possible grade 20 (the selected cells give the total score 5 out of the 9 possible points and 5 / 9 * 20 = 11.1). There is also a very basic support for browsers with javascript disabled planned for the future. However, implementation details of it are not defined yet.
Using the link at the top right corner of the rubric, the rubric grid can be hidden to save space on the screen.

Collapsed rubric shows just the result of the grading

By clicking at "Save and show next" the teacher would continue with the next student. The calculated grade is pushed into the gradebook together with the information that the value was obtained from a particular rubric. The grader report is then able to generate a link to a standalone page (or some sort of pop-up like we have for in-built help maybe) that shows the filled rubric.

How it works

UML sequence diagram of the communication among browser, assignment module and rubric plugin

A: Teacher clicks the "Grade" link in the list of all submission to grade a particular submission
B: Assignment module realizes that an advanced grading method should be used (because there is an active method defined for the grading area "submission" in the module's context) so it adds an element into the form representing the rubric instead of the default simple grade/scale item selector. If the rubric was not defined yet (or it is being currently edited and it was not released for usage yet), the element displays a message and a link to the rubric editor.
C: If there is no grading_instances table record existing yet for this particular submission (itemid), one is created for the current user as the rater. If a previous instance is found, it is loaded.
D: The rubric plugin renders HTML code to be embedded into the form and requires the current page loads the needed YUI modules for the rubric.
E: The whole page including the form with embedded rubric is sent to the browser.
F: The teacher clicks a cell in the rubric grid. That triggers a function from the rubric YUI module.
G: AJAX request is sent to the rubric, informing the plugin about a change in the assessment. As a part of the AJAX request, the browser sends the information about the assignment grade value (maximum grade 20 in our case).
H: The rubric plugin saves the new data into its own table.
J: The rubric plugin loads how the whole rubric is filled now, calculates the current total score and converts it to the assignment grade value (points out of 20 in our case).
K: The rubric plugin stores the new total score as a raw grade between 0.00000 and 100.00000 into grading_instances.
L: AJAX response contains the updated grade for display purposes.
M: The teacher is informed visually about the successful save of the change and the displayed current grade is updated. The teacher continues with the grading (goto F until the whole rubric is filled and the teacher is happy with it).
N: The teacher clicks "Save and show next" and the main form is submitted to the assignment for processing.
O: The assignment asks the rubric for the most recent result of the grading, providing the grade range it wants to convert it to (points range 0 - 20 in our case).
P: The rubric module reads the current raw grade from grading_instances
Q: The rubric module converts the loaded raw grade into the required assignment grade (scale or points range)
R: The grade is provided back to the assignment
S: The assignment module process the whole form as it does not, having the final grade calculated by rubric available.
T: The assignment module pushes the new grade into the gradebook (as it does now). Gradebook API are improved so the caller is able to record the source of the value (probably into two new columns in the grade_grades table with the component name and instanceid). Using this information, the grader report must be later able to identify the source plugin and ask it for the URL of a page where the detailed grade analysis is displayed. Note that this part is actually independent on this project and can be developed and used separately. This way, quiz grades in the gradebook could lead to a page in quiz with the detailed analysis, for example.
U: The browser is redirected to the new page as requested by user (as it does now)

Highlights

This proposed solution has minimal impact on the current grading logic in modules. Advanced grading forms (like rubrics) are considered as helper tools that are supposed to produce the grade value, as if the marker used some spreadsheet application and then inserted the calculated value into the current grading forms. That is why they are to be implemented as a new subsystem and not part of the gradebook. Related changes in the gradebook are therefore minimal, too (only for the ability to link back to the grade value explanation). This way the project is doable for Moodle 2.2 release.

The grading forms are defined separately for each gradable area, they do not belong to a course or some higher context (as scales, outcomes or questions). So for example, each Assignment defines its own rubric. When the same grading form is to be re-used in another module, a new copy of its definition is created and eventually modified only there. Therefore the UI of the forms editor must support this cloning smoothly. It is believed that this feature would minimize the problems with modifying some form once it was already used (Once a form was used for real assessment, it can't be changed any more - otherwise the grades it produces wouldn't be comparable and the assessment would loose significant amount of objectivity).

Implementation plan

New feature called FEATURE_ADVANCED_GRADING to be registered in lib/moodlelib.php
New core subsystem called "grading" (core_grading in frankenstyle) to be added into get_core_subsystems(), located in $CFG->dirroot/grade/grading/
New plugin type "gradingform" to be added into get_plugin_types(), located in $CFG->dirroot/grade/grading/form/. Advanced grading methods are implemented as plugins of this type, having the frankenstyle name like "gradingform_rubric"
The files course/modedit.php, course/moodleform_mod.php are modified so that they add support for selecting the active grading method in module.
The class mod_assignment_grading_form to be modified to support grading form embedding
Rubric editor to be implemented
Rubric renderer and the submitted data handler to be implemented
Gradebook API and the table grade_grades to be extended so that each grade can store the source of the value in a way that allows to display a deeper analysis/overview/more information on the grade (independent on this project, can be used outside the scope of advanced grading).

Concept map

Concept map of the framework

Database schema

ER diagram of new tables

grading_areas

Identifies gradable areas where advanced grading can happen. For each area, the current active plugin can be set. Example: for the assignment in the use case above, there would be a record in this table with contextid set to the module instance's context, component set to the assignment subtype, areaname set to "submission" and activemethod set to "rubric" (as that is what the teacher selected in mod_form).

Field	Type	Default	Description
id	bigint		auto-numbered
contextid	bigint		The context of the gradable area, eg module instance context
component	varchar (100)		Frankenstyle name of the component holding this area, eg assignment_online
areaname	varchar (100)		The name of gradable area, eg submission
activemethod	varchar (100)	null	The currently active grading method (plugin) that should be used for this area

grading_definitions

Each plugin can define one grading form in the given gradable area. That means for the given Assignment, there can be just one rubric defined. If there are more plugins available, the teacher can define one instance per gradable area for each of them and switch between them. This can be used for example to produce a quick initial feedback on the work in progress using one plugin and then switch to another method to grade the final products. This scenario is supported in the Workshop module, where the initial assessment can be done using a lite grading method like "Comments only" and then it can be replaced with heavyweight "Rubric" during the second grading iteration.

Field	Type	Default	Description
id	bigint		auto-numbered
areadid	bigint		The gradable area this form is defined for
method	varchar (100)		The name of the plugin that provides this form
name	varchar (255)		The title of the form that helps users to identify it
description	text		More detailed description of the form
descriptionformat	int		Format of the description field
status	int	0	Status of the form definition, by default in the under-construction state
copiedfromid	bigint		The id of the original definition that this was initially copied from or null if it was from scratch
timecreated	int		The timestamp of when the form definition was created initially
usercreated	int		The ID of the user who created this definition and is considered as its owner for access control purposes
timemodified	int		The time stamp of when the form definition was modified recently
usermodified	int		The ID of the user who did the most recent modification
timecopied	int	0	The timestamp of when the form definition was cloned last time
options	text		General field to be used by plugins as a general storage place for their own settings

grading_instances

For each item inside the gradable area, one instance of the defined form is created when the grading form is filled. It holds the information about the user who did the assessment. Note that we do not store the user who is being graded here intentionally. The framework is robust enough that it can be used for any item inside the gradable area. The item itself can be for example a team work - the interpretation and handling is done by modules.

Field	Type	Default	Description
id	bigint		auto-numbered
definitionid	bigint		The ID of the form definition this is instance of
raterid	int		The ID of the user who did the assessment
itemid	int		This identifies the graded item within the grabable area
rawgrade	numeric (10, 5)		The raw normalized grade 0.00000 - 100.00000 as a result of the most recent assessment
status	int	0	The status of the assessment. By default the instance is under-assessment state
feedback	text		Overall feedback from the rater for the author of the graded item
feedbackformat	int		The format of the feedback field
timemodified	int		The timestamp of when the assessment was most recently modified

gradingform_rubric_criteria

Stores the rows of the rubric grid.

Field	Type	Default	Description
id	bigint		auto-numbered
definitionid	bigint		The ID of the form definition this criterion is part of.
sortorder	int		Defines the order of the criterion in the rubric.
description	text		The criterion description.
descriptionformat	int		The format of the description field

gradingform_rubric_levels

Stores the columns of the rubric grid.

Field	Type	Default	Description
id	bigint		auto-numbered
criterionid	int		The rubric criterion we are level of
score	numeric (10,5)		The score for this level
definition	text	null	The optional text describing the level
definitionformat	int		The format of the definition field

gradingform_rubric_fillings

Stores the data of how the rubric is filled by a particular rater

Field	Type	Default	Description
id	bigint		auto-numbered
instanceid	bigint		The ID of the grading form instance
criterionid	bigint		The ID of the criterion (row) in the rubric
levelid	bigint	null	If a particular level was selected during the assessment, we store its ID here
remark	text		Side note feedback regarding this particular criterion
remarkdformat	int		The format of the remark field

Grading method plugins

Various grading methods, each providing its own type of the grading form and the way it calculates the final grade, can be implemented. For the beginning, we plan to convert all current grading strategies from the Workshop module into advanced grading methods available for all other modules. Here is a quick summary of the planned methods.

All supported grading methods must have the following common characteristics:

They are producing a number between 0.00000 and 100.00000 as a final result of the grading (or null if the total score can't be calculated yet). The lower value corresponds the worst student's performance, the higher corresponds the best student's performance. Activity modules can then convert such a raw value into their grades, scales or use them for whatever purpose. Example: A rubric with the score between 0 and 10 would report the score 8 as a raw grade 80.00000.
They generally consist of a number of assessed criteria, each of them being again represented with a raw grade 0 - 100 or null. They can report this in a form of vector with the dimension n where n is the number of assessment criteria. This is required by Workshop module that needs this information to compare the "internal structure" of peer assessments and calculate the grade for assessment from it. Example: In case of rubric, each row of the grid is a dimension of the assessment.

Rubric

Rubric is represented as a grid.
Each row in the grid forms a dimension (criterion) of the assessment.
Each row can have various number of columns although usually rubrics have the constant number of columns in all rows (typically 4).
Each column represents a level for the given criterion and have a numerical score and description defined.
The total score is obtained as a sum of scores from all rows.

Notes and things to discuss

It seems that teachers round the world use rubric levels with scores like 4, 3, 2 and 1. It is not clear whether a "hidden" (implicit) score 0 can be used too for the worst observed performance or whether the rubric forces to choose from 4 to 1. If the minimum possible score per row was 1 then the student would get some score even for the worst assessment. Example: A rubric with five criteria (rows), each of them having levels 4, 3, 2, 1. The maximum possible score is 20, the minimum is 4. So the score 20 would be mapped to 100% and the score 4 to 0% raw grade.
The editor will allow to select the score for each cell individually. So it will be possible to have a rubric like:

   +---+---+---+---+
   | 4 | 3 | 2 | 1 |
   +---*---*---*---+
   |   1   |   0   |
   +---*---*---*---+
   | 3 | 2 | 1 | 0 |
   +---*---*---*---+
   |  8  |  3  | 1 |
   +-----+-----+---+

In this (very unusual but still possible) rubric, the minimum score is 2 and the maximum score is 16. Note the last row has much bigger impact on the total score than the others and can itself form up to one half of the total score.

Accumulative

Represented as a set of criteria, each of them can be graded either via a numerical score or using a scale.
Regardless the actual grading tool used for each criterion, it is always normalized first to a value 0 - 100 before the total score is calculated.
The total score is calculated as a weighted mean of normalized values from criteria. This way, a scale consisting of three items have the same influence on the total score as a grade of 80.
Each criterion can have its explicit weight defined.

Number of errors

In fact this can be considered as a simplified rubric where each criterion is assessed by exactly two levels - one for the absence of an error, one for the presence of error. Example: "The essay has at least 2000 words - Yes/No". Here "Yes" is for the absence of error and "No" is for the presence of an error.
The total score is obtained from the number of errors (negative responses) using a mapping table.

Other custom grading methods

Custom grading methods can be implemented as plugins that reflect the individual needs and/or habits.

Documentation