Advanced grading methods

Jump to: navigation, search
Advanced grading methods
Project state In production since 2.2
Tracker issue MDL-29108
Discussion [1]
Assignee Marina Glancy and David Mudrak

Moodle 2.2


Warning This page contains the original specification. Significant changes were done during the development. The database schema is still valid but the overall workflow is different in the final implemented version.

Project goals

  • Long-term goal: To be able to use advanced grading methods (like rubrics and other multi-criteria assessment forms) to assess all reasonable places (like assignment submissions, forum posts, wiki pages, database records etc.), not just the submissions in the workshop module.
  • Short-term goal: To allow using rubric as a grading tool for the assignment module.

Project scope

This project does NOT change the gradebook internals or its API for obtaining and storing grades. It has also nothing to do with other gradebook issues like aggregating hidden grades and friends.

This project aims to:

  • provide a way to define grading forms
  • allow activity modules to use defined forms to calculate grades

Things that must be decided and designed carefully:

  • export/import - how to easily publish forms definitions (XML + embedded files) and re-use them
  • backup/restore - grading forms must become part of the course module structures (XML structures similar to export/import)

Relevant tracker issues

  • MDL-795 (44 votes, 27 watchers) - 'workshop-like' teacher assessment methods (rubrics, accumulative, etc) for assignments
  • MDL-2255 (13 votes, 3 watchers) - Rubrics for all Activities
  • CONTRIB-2058 - Ability to edit/delete rubrics (Assignment Rubric activity module)

Use case 1: Using rubric in the assignment

  • When editing the Assignment settings, there is a new field in the "Grade" section called "Grading method" (see the screenshots attached). The teacher can define whether the grade (value or scale) will be put directly by the marker (as it is now) by selecting the grading method "Simple direct grading". Alternatively the teacher can select some advanced grading method to use. She sets the grade for this Assignment to a value 20 and decides to use advanced grading method "Rubric" to calculate that grade. See MDL-29151 for how the mod_form would look like if the Assignment module supported multiple gradable areas.
Preview of the current development version of the Assignment settings form with the advanced grading method selector available


  • At the assignment page, the Settings side block contains new node to define the rubric. The teacher clicks at "Rubric editor" link.
  • The editor allows the teacher to design the rubric from scratch, to import it from XML file or use some existing rubric at the site as a template to start off from. She prepares a rubric consisting of three criteria, each of them having four levels with scores 3, 2, 1, 0. Hence the total score obtained from the rubric will be something between 9 and 0 points. When the teacher is happy with the preview, goes back to the assignment.
  • The list of submissions does not support the Quick grading mode now. The teacher clicks the "Grade" link at a submission row and is taken to /mod/assignment/submissions.php where the grading of that single submission happens.
  • Instead of the simple direct grade selector 0-20, the defined rubric is embedded now. The teacher clicks into the cells of the rubric grid to choose the most appropriate assessment of each criteria. Selected cells are highlighted. It is also possible to put a side plain-text remark for each criterion and the overall remark for the whole rubric.
Rubric embedded into the submission grading page


  • On each change of the rubric, it is saved via AJAX and the displayed grade is replaced with a new calculated value (so the teacher can see how the total calculated grade change on the fly). In this example, the finally displayed calculated grade value would be 11.1 of the maximum possible grade 20 (the selected cells give the total score 5 out of the 9 possible points and 5 / 9 * 20 = 11.1). There is also a very basic support for browsers with javascript disabled planned for the future. However, implementation details of it are not defined yet.
  • Using the link at the top right corner of the rubric, the rubric grid can be hidden to save space on the screen.
Collapsed rubric shows just the result of the grading


  • By clicking at "Save and show next" the teacher would continue with the next student. The calculated grade is pushed into the gradebook together with the information that the value was obtained from a particular rubric. The grader report is then able to generate a link to a standalone page (or some sort of pop-up like we have for in-built help maybe) that shows the filled rubric.

How it works

UML sequence diagram of the communication among browser, assignment module and rubric plugin


  • A: Teacher clicks the "Grade" link in the list of all submission to grade a particular submission
  • B: Assignment module realizes that an advanced grading method should be used (because there is an active method defined for the grading area "submission" in the module's context) so it adds an element into the form representing the rubric instead of the default simple grade/scale item selector. If the rubric was not defined yet (or it is being currently edited and it was not released for usage yet), the element displays a message and a link to the rubric editor.
  • C: If there is no grading_instances table record existing yet for this particular submission (itemid), one is created for the current user as the rater. If a previous instance is found, it is loaded.
  • D: The rubric plugin renders HTML code to be embedded into the form and requires the current page loads the needed YUI modules for the rubric.
  • E: The whole page including the form with embedded rubric is sent to the browser.
  • F: The teacher clicks a cell in the rubric grid. That triggers a function from the rubric YUI module.
  • G: AJAX request is sent to the rubric, informing the plugin about a change in the assessment. As a part of the AJAX request, the browser sends the information about the assignment grade value (maximum grade 20 in our case).
  • H: The rubric plugin saves the new data into its own table.
  • J: The rubric plugin loads how the whole rubric is filled now, calculates the current total score and converts it to the assignment grade value (points out of 20 in our case).
  • K: The rubric plugin stores the new total score as a raw grade between 0.00000 and 100.00000 into grading_instances.
  • L: AJAX response contains the updated grade for display purposes.
  • M: The teacher is informed visually about the successful save of the change and the displayed current grade is updated. The teacher continues with the grading (goto F until the whole rubric is filled and the teacher is happy with it).
  • N: The teacher clicks "Save and show next" and the main form is submitted to the assignment for processing.
  • O: The assignment asks the rubric for the most recent result of the grading, providing the grade range it wants to convert it to (points range 0 - 20 in our case).
  • P: The rubric module reads the current raw grade from grading_instances
  • Q: The rubric module converts the loaded raw grade into the required assignment grade (scale or points range)
  • R: The grade is provided back to the assignment
  • S: The assignment module process the whole form as it does not, having the final grade calculated by rubric available.
  • T: The assignment module pushes the new grade into the gradebook (as it does now). Gradebook API are improved so the caller is able to record the source of the value (probably into two new columns in the grade_grades table with the component name and instanceid). Using this information, the grader report must be later able to identify the source plugin and ask it for the URL of a page where the detailed grade analysis is displayed. Note that this part is actually independent on this project and can be developed and used separately. This way, quiz grades in the gradebook could lead to a page in quiz with the detailed analysis, for example.
  • U: The browser is redirected to the new page as requested by user (as it does now)

Highlights

This proposed solution has minimal impact on the current grading logic in modules. Advanced grading forms (like rubrics) are considered as helper tools that are supposed to produce the grade value, as if the marker used some spreadsheet application and then inserted the calculated value into the current grading forms. That is why they are to be implemented as a new subsystem and not part of the gradebook. Related changes in the gradebook are therefore minimal, too (only for the ability to link back to the grade value explanation). This way the project is doable for Moodle 2.2 release.

The grading forms are defined separately for each gradable area, they do not belong to a course or some higher context (as scales, outcomes or questions). So for example, each Assignment defines its own rubric. When the same grading form is to be re-used in another module, a new copy of its definition is created and eventually modified only there. Therefore the UI of the forms editor must support this cloning smoothly. It is believed that this feature would minimize the problems with modifying some form once it was already used (Once a form was used for real assessment, it can't be changed any more - otherwise the grades it produces wouldn't be comparable and the assessment would loose significant amount of objectivity).

Implementation plan

  • New feature called FEATURE_ADVANCED_GRADING to be registered in lib/moodlelib.php
  • New core subsystem called "grading" (core_grading in frankenstyle) to be added into get_core_subsystems(), located in $CFG->dirroot/grade/grading/
  • New plugin type "gradingform" to be added into get_plugin_types(), located in $CFG->dirroot/grade/grading/form/. Advanced grading methods are implemented as plugins of this type, having the frankenstyle name like "gradingform_rubric"
  • The files course/modedit.php, course/moodleform_mod.php are modified so that they add support for selecting the active grading method in module.
  • The class mod_assignment_grading_form to be modified to support grading form embedding
  • Rubric editor to be implemented
  • Rubric renderer and the submitted data handler to be implemented
  • Gradebook API and the table grade_grades to be extended so that each grade can store the source of the value in a way that allows to display a deeper analysis/overview/more information on the grade (independent on this project, can be used outside the scope of advanced grading).

Concept map

Concept map of the framework


Database schema

ER diagram of new tables


grading_areas

Identifies gradable areas where advanced grading can happen. For each area, the current active plugin can be set. Example: for the assignment in the use case above, there would be a record in this table with contextid set to the module instance's context, component set to the assignment subtype, areaname set to "submission" and activemethod set to "rubric" (as that is what the teacher selected in mod_form).

Field
Type
Default
Description
id bigint auto-numbered
contextid bigint The context of the gradable area, eg module instance context
component varchar (100) Frankenstyle name of the component holding this area, eg assignment_online
areaname varchar (100) The name of gradable area, eg submission
activemethod varchar (100) null The currently active grading method (plugin) that should be used for this area

grading_definitions

Each plugin can define one grading form in the given gradable area. That means for the given Assignment, there can be just one rubric defined. If there are more plugins available, the teacher can define one instance per gradable area for each of them and switch between them. This can be used for example to produce a quick initial feedback on the work in progress using one plugin and then switch to another method to grade the final products. This scenario is supported in the Workshop module, where the initial assessment can be done using a lite grading method like "Comments only" and then it can be replaced with heavyweight "Rubric" during the second grading iteration.

Field
Type
Default
Description
id bigint auto-numbered
areadid bigint The gradable area this form is defined for
method varchar (100) The name of the plugin that provides this form
name varchar (255) The title of the form that helps users to identify it
description text More detailed description of the form
descriptionformat int Format of the description field
status int 0 Status of the form definition, by default in the under-construction state
copiedfromid bigint The id of the original definition that this was initially copied from or null if it was from scratch
timecreated int The timestamp of when the form definition was created initially
usercreated int The ID of the user who created this definition and is considered as its owner for access control purposes
timemodified int The time stamp of when the form definition was modified recently
usermodified int The ID of the user who did the most recent modification
timecopied int 0 The timestamp of when the form definition was cloned last time


options text General field to be used by plugins as a general storage place for their own settings

grading_instances

For each item inside the gradable area, one instance of the defined form is created when the grading form is filled. It holds the information about the user who did the assessment. Note that we do not store the user who is being graded here intentionally. The framework is robust enough that it can be used for any item inside the gradable area. The item itself can be for example a team work - the interpretation and handling is done by modules.

Field
Type
Default
Description
id bigint auto-numbered
definitionid bigint The ID of the form definition this is instance of
raterid int The ID of the user who did the assessment
itemid int This identifies the graded item within the grabable area
rawgrade numeric (10, 5) The raw normalized grade 0.00000 - 100.00000 as a result of the most recent assessment
status int 0 The status of the assessment. By default the instance is under-assessment state
feedback text Overall feedback from the rater for the author of the graded item
feedbackformat int The format of the feedback field
timemodified int The timestamp of when the assessment was most recently modified

gradingform_rubric_criteria

Stores the rows of the rubric grid.

Field
Type
Default
Description
id bigint auto-numbered
definitionid bigint The ID of the form definition this criterion is part of.
sortorder int Defines the order of the criterion in the rubric.
description text The criterion description.
descriptionformat int The format of the description field

gradingform_rubric_levels

Stores the columns of the rubric grid.

Field
Type
Default
Description
id bigint auto-numbered
criterionid int The rubric criterion we are level of
score numeric (10,5) The score for this level
definition text null The optional text describing the level
definitionformat int The format of the definition field

gradingform_rubric_fillings

Stores the data of how the rubric is filled by a particular rater

Field
Type
Default
Description
id bigint auto-numbered
instanceid bigint The ID of the grading form instance
criterionid bigint The ID of the criterion (row) in the rubric
levelid bigint null If a particular level was selected during the assessment, we store its ID here
remark text Side note feedback regarding this particular criterion
remarkdformat int The format of the remark field

Grading method plugins

Various grading methods, each providing its own type of the grading form and the way it calculates the final grade, can be implemented. For the beginning, we plan to convert all current grading strategies from the Workshop module into advanced grading methods available for all other modules. Here is a quick summary of the planned methods.

All supported grading methods must have the following common characteristics:

  • They are producing a number between 0.00000 and 100.00000 as a final result of the grading (or null if the total score can't be calculated yet). The lower value corresponds the worst student's performance, the higher corresponds the best student's performance. Activity modules can then convert such a raw value into their grades, scales or use them for whatever purpose. Example: A rubric with the score between 0 and 10 would report the score 8 as a raw grade 80.00000.
  • They generally consist of a number of assessed criteria, each of them being again represented with a raw grade 0 - 100 or null. They can report this in a form of vector with the dimension n where n is the number of assessment criteria. This is required by Workshop module that needs this information to compare the "internal structure" of peer assessments and calculate the grade for assessment from it. Example: In case of rubric, each row of the grid is a dimension of the assessment.

Rubric

  • Rubric is represented as a grid.
  • Each row in the grid forms a dimension (criterion) of the assessment.
  • Each row can have various number of columns although usually rubrics have the constant number of columns in all rows (typically 4).
  • Each column represents a level for the given criterion and have a numerical score and description defined.
  • The total score is obtained as a sum of scores from all rows.

Notes and things to discuss

  • It seems that teachers round the world use rubric levels with scores like 4, 3, 2 and 1. It is not clear whether a "hidden" (implicit) score 0 can be used too for the worst observed performance or whether the rubric forces to choose from 4 to 1. If the minimum possible score per row was 1 then the student would get some score even for the worst assessment. Example: A rubric with five criteria (rows), each of them having levels 4, 3, 2, 1. The maximum possible score is 20, the minimum is 4. So the score 20 would be mapped to 100% and the score 4 to 0% raw grade.
  • The editor will allow to select the score for each cell individually. So it will be possible to have a rubric like:
   +---+---+---+---+
   | 4 | 3 | 2 | 1 |
   +---*---*---*---+
   |   1   |   0   |
   +---*---*---*---+
   | 3 | 2 | 1 | 0 |
   +---*---*---*---+
   |  8  |  3  | 1 |
   +-----+-----+---+

In this (very unusual but still possible) rubric, the minimum score is 2 and the maximum score is 16. Note the last row has much bigger impact on the total score than the others and can itself form up to one half of the total score.

Accumulative

  • Represented as a set of criteria, each of them can be graded either via a numerical score or using a scale.
  • Regardless the actual grading tool used for each criterion, it is always normalized first to a value 0 - 100 before the total score is calculated.
  • The total score is calculated as a weighted mean of normalized values from criteria. This way, a scale consisting of three items have the same influence on the total score as a grade of 80.
  • Each criterion can have its explicit weight defined.

Number of errors

  • In fact this can be considered as a simplified rubric where each criterion is assessed by exactly two levels - one for the absence of an error, one for the presence of error. Example: "The essay has at least 2000 words - Yes/No". Here "Yes" is for the absence of error and "No" is for the presence of an error.
  • The total score is obtained from the number of errors (negative responses) using a mapping table.

Other custom grading methods

Custom grading methods can be implemented as plugins that reflect the individual needs and/or habits.

See also