Advanced grading methods

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Advanced grading methods
Project state	Planning
Tracker issue	MDL-29108
Discussion	TODO
Assignee	David Mudrak

Moodle 2.2

Project goals

• Long-term goal: To be able to use advanced grading methods (like rubrics and other multi-criteria assessment forms) to assess all reasonable places (like assignment submissions, forum posts, wiki pages, database records etc.), not just the submissions in the workshop module.
• Short-term goal: To allow using rubric as a grading tool for the assignment module.

Project scope

This project does NOT change the gradebook internals or its API for obtaining and storing grades. It has also nothing to do with other gradebook issues like aggregating hidden grades and friends.

This project aims to:

• provide a way to define grading forms
• allow activity modules to use defined forms to calculate grades

Things that must be decided and designed carefully:

• export/import - how to easily publish forms definitions (XML + embedded files) and re-use them
• backup/restore - grading forms must be part of the gradebook structures, like grade categories and grade items are (XML structures similar to export/import)

Relevant tracker issues

• MDL-795 (44 votes, 27 watchers) - 'workshop-like' teacher assessment methods (rubrics, accumulative, etc) for assignments
• MDL-2255 (13 votes, 3 watchers) - Rubrics for all Activities
• CONTRIB-2058 - Ability to edit/delete rubrics (Assignment Rubric activity module)

Use case 1: Using rubric in the assignment

• When editing the Assignment settings, there is a new field in the "Grade" section (see callout 1 in the UI mockup below). The teacher can define whether the grade (value or scale) will be put directly by the marker (as it is now) or some advanced grading form will be used. She sets the grade for this Assignment to a value 20 and decides to use advanced grading strategy to calculate that grade. She picks "Rubric" from the list of available form types.

UI mockups of adding rubric into the Assignment

• At the assignment page, the Settings side block contains new node to define the rubric (callout 2). The teacher clicks at "Rubric editor" link.
• The editor allows the teacher to design the rubric from scratch, to import it from XML file or use some existing rubric at the site as a template to start off from. She prepares a rubric consisting of three criteria, each of them having four levels with scores 0, 1, 2 and 3. Hence the total score obtained from the rubric will be something from 0 to 12. When the teacher is happy with the preview, goes back to the assignment.
• The list of submissions does not offer the Quick grading mode now. The "Grade" link next to each submission leads to a standalone page /grade/form/grading.php where the grading happens.

Screen with the online assignment submission being graded with a rubric

• The screen displays the graded submission and the defined rubric. The teacher fills the rubric and clicks "Save and close". She is taken back to the list of submissions at /mod/assignment/submissions.php.
• There is a grade now for that submission displayed, as if the teacher chose it manually. Note the score from the rubric is converted to the selected grade for the assignment. So for example the maximum score 12 leads to the grade 20, score 3 would give grade 5 etc (comparable with how the total score for questions used in a quiz is converted to a grade in the course). The same would apply if the grade for the submission would be set to use a scale.
• The calculated grade is pushed into the gradebook. The grader report is able to generate a link to a page that shows the filled rubric.

How it works

I am currently working on a description of the flow - see the UML sequence diagram (work in progress). Just to quickly summarize now:

• The Assignment module must declare FEATURE_GRADING_FORMS in its lib.php so that standard_grading_coursemodule_elements() can put the new fields into mod_assignment_mod_form.
• A simple if-condition in /mod/assignment/submissions.php determines whether advanced grading methods should be used or not. If not, the module works the same as it does now. If grading methods are enabled, the link leads to a page /grade/form/grading.php passing the information about the graded submission via URL parameters contextid, component, grading area name and item id. In the case of assignment, the context is module instance's context, the component is the assignment subtype (written in frankenstyle), the grading area is "submission" and item id is the submission id.
• The page /grade/form/grading.php uses a callback back to the assignment to obtain information about the graded item (ie submission in our case). The callback function will return the HTML to print at the grading page and some other data so that the rading page may actually look as if it was part of the assignment module scope.
• As we know what grading method should be used in the given area, the appropriate plugin (here it is rubric) is called and it provides the grading form. If the rubric was not defined yet (or it is being currently edited and it was not released for usage yet), the page displays a message and a link to the rubric editor.
• When the teacher fills the rubric and submits it, the rubric plugin is asked to calculate a value representing the result of the assessment. All plugins return normalized value from 0.00000 to 100.00000 here. This value is sent back to the assignment module via another callback function so that it can handle it. In our scenario, the handler would convert the normalized score to the assignment grade (0 - 20 in this use case) and processes it in the same way as it does now. As a part of the response, the callback handler returns URL where the user should be taken to after finishing one assessment. In our case it would be the page with the list of all submissions.
• When modules push the grades into the gradebook, they will be able to record the source of the value (probably into two new columns in the grade_grades table with the component name and instanceid). Using this information, the grader report must be able to identify the source plugin and ask it for the URL of a page where the detailed grade analysis is displayed. Note that this part is actually independent on this project and can be developed and used separately. This way, quiz grades in the gradebook could lead to a page in quiz with the detailed analysis.

Highlights

This proposed solution has minimal impact on the current grading logic in modules. Advanced grading forms (like rubrics) are considered as helper tools that are supposed to produce the grade value, as if the marker used some spreadsheet application and then inserted the calculated value into the current grading forms. Changes in the gradebook are minimal (only for the ability to link back to the grade value explanation). This way the project is doable for Moodle 2.2 release.

Implementation plan

• New core plugin type called "Grading method" to be added into get_plugin_types() as 'grademethod' with the location in /grade/method/*
• All grading methods to be implemented as plugins - "Rubric", "Multi-scale" (aka Accumulative grading strategy in the Workshop), "Number of errors" etc.

Database schema

ER diagram of new tables

Core scope tables

grade_gradable_areas

identifies areas where advanced grading can happen. For each area, the current active plugin can be set. Example: for the assignment in the use case above, there would be a record in this table with contextid set to the module instance's context, component set to the assignment subtype, areaname set to "submission" and activemethod set to "rubric" (as that is what the teacher selected in mod_form).

grade_form_definitions

each plugin can define one grading form in the given gradable area. That means for the given Assignment, there can be just one rubric defined. If there are more plugins available, the teacher can define one instance per gradable area for each of them and switch between them. This can be used for example to produce a quick initial feedback on the work in progress using one plugin and then switch to another method to grade the final products. This scenario is supported in the Workshop module, where the initial assessment can be done using a lite grading method like "Comments only" and then it can be replaced with heavyweight "Rubric" during the second grading iteration.

grade_form_instances

for each item inside the gradable area, one instance of the defined form is created when the grading form is filled. It holds the information about the user who did the assessment. Note that we do not store the user who is being graded here intentionally. The framework is robust enough that it can be used for any item inside the gradable area. The item itself can be for example a team work - the interpretation and handling is done by modules.

Plugin scope tables

gradeform_rubric_criteria

stores the rows of the rubric grid

gradeform_rubric_levels

stores the columns of the rubric grid

gradeform_rubric_fillings

stores the data of how the rubric is filled by a particular rater

See also

• The description of rubric at Wikipedia
• Kathy Schrock's Guide for Educators - Examples of Assessment Rubrics
• RubiStar is a free tool to help teachers create quality rubrics
• LMS Instructure and especially their way of defining rubrics
• Rubrics Manual for JOULE by Moodle Rooms
• Gradebook_2.x_architecture to come in 2.3