Natural weighting

Natural Weighting
Project state	Specification
Tracker issue	https://tracker.moodle.org/browse/MDL-43378
Discussion	https://moodle.org/mod/forum/discuss.php?d=245442
Assignee	Moodle HQ

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Natural weighting is a modification to how grades are combined.

When 'Sum of Grades' aggregation is chosen (and natural weighting is active), there will be a weight column with form fields, looking like "Weighted mean" is in 2.7. By default, the weight column shows the natural weights given to all categories and items based on gradebook structure.

Weights can be overridden by modifying the values in the form fields.

A "Reset weights" button in each category resets the weights to their defaults.

Rationale

In grading, weight is the proportion of an item's value in relation to all the other items in its container (category or course).

Value is most often represented as 'points'.

Every grade has a weight.

There are three fundamental approaches to dealing with weighting in a gradebook

- Simple (or natural) weighting. When accurate points for categories and course totals are maintained, both Sum of grades and Simple weighted mean of grades accomplish exactly the same thing.

- Adjusted weighting. Users often want to adjust categories and even items to have a weight other than the natural weight, essentially ignoring the points entirely. An example would be, Quizzes=40%, Assignments=50%, Attendance=10%.

- Mean of grades. If I have a category made up of five quizzes all worth different points and I want a category percentage to just tell the average (or mean) of all the quiz results' percentage correct, I would choose this approach again, disregarding the points.

Simplifying the understanding of weighting reduces the aggregation methodology down to 'either/ or'

- either 'leave weights as is -- natural'

- or 'adjust to suit my needs -- adjusted weights'

This greatly simplifies the interface needed to present these options

- They don't need to select an aggregation type

- They don't need to know how each aggregation type performs

- The actual weight of every item and category is always displayed to them.

- If those are not the weights they want, they merely adjust them.

- Much easier for the common user to grasp.

This interface and calculations are expected to suit the needs of all except very edge cases (which can still be handled with formulas). If it turns out that this is enough for everyone then we can potentially remove quite a lot of aggregation code from the gradebook at a later stage, improving speed and readability.

How natural weights meets the requirements of the aggregation types:

Sum of grades

Leave all weightings as is, no change necessary.

Mean of grades

Set all weights equal to each other. Doesn't matter what the number is, as in legacy behavior Moodle will automatically adjust so the sum equals 100%. However, in this implementation the group wants to have a pop-up warning the user the weights don't add up to 100% and they will therefore be adjusted uniformly to equal 100%.

Weighted mean of grades

Adjust weights as desired. No change from current behavior.

Simple weighted mean of grades

Leave all weightings as is, no change necessary. Same as Sum of grades.

Mean of grades (with extra credits)

Extra credit has a natural weight that takes the total container (category/ course) weight above 100% (hence, 'extra'). Extra credit is available at all times no matter what scheme of weighting is chosen.

Highest

User would input the value '1' for 'Keep highest' in the category

Lowest

Not possible using natural weights, but it can be replaced by a calculation into a new item.

Median / Mode of grades

Not possible using natural weights, but it's doubtful anyone uses these anyway so perhaps this can be deprecated entirely.

Implementation stages

To be safe, we plan to implement this in stages:

STAGE 1) Implement natural weighting as part of the Sum of Grades aggregation type, with admin settings that allow them to disable all other aggregations if they choose. (Moodle 2.8)

STAGE 2) Once people have been using this for a while, explore the possibility of removing other aggregation code or making them into optional plugins. (Moodle 3.0 perhaps)

Interfaces

Category editing

Prototype from luther.edu:

In addition the weights should be editable form fields here, and there would be one button per category to "reset weights" back to a natural Sum of Grades.

Admin settings

New setting to "Force Natural Weighting" (on/off). This would default to OFF on upgraded sites and ON for new sites.

ON: this will hide the aggregation menu and all new categories will default to "Sum of grades" and natural weights. It will not affect existing gradebooks with other aggregations set (see upgrade script below for a conversion strategy).

OFF: aggregation menu is shown, but Sum of Grades will still contain new natural weighting UI and logic.

NOTE: I'd really like to see a third option, 'Optional' with the use case of admins wanting to introduce this new method to some teachers on a trial basis but not have it necessarily forced on all courses. We initially saw 2.9 being the point at which we would go down to two settings, 'On' or 'Off' with 3.0 full 'On'

Upgrade script

If a site is switching to forced natural weights, they will probably want to upgrade all existing categories.

This new optional upgrade script will examine all grade categories looking at aggregation methods. It should report on what exists and allow the admin to perform an automated upgrade as follows:

1. If GRADE_AGGREGATE_WEIGHTED_MEAN then:
   • Store value of 'aggregationcoef' fields to 'weight' fields in grade_items
   • Set 'weightoverride' field in grade_items
2. If ... then:

(We need exact strategies here for ALL aggregation types)

Database changes

Add three fields to grade_items

1. extracredit, tinyint, notnull, default: 0
2. weight, decimal(10,5), notnull, default: 0
3. weightoverride, tinyint, notnull, default: 0

Add one field to grade_grades

1. weight, decimal(10,5), notnull, default: 0

This fixes the use of aggregationcoef representing both extra credit and weight in 2.7.

Logic changes

Sum of Grades aggregation must be fixed to exclude hidden items and to exclude non-graded items

Determination of range (in Grader and User report) and Max grade (in Setup screen) must be fixed to exclude extra credit items

Natural weights must be saved and applied to the form.

Natural weights are calculated for the Setup screen, the Grader report, and the User report. The calculation for the User report is different than that used for the Setup screen and Grader report.

Weights should be stored in grade_items for the Setup screen and the Grader report and in the grade_grades table for the User report.

The Math

The sum of weights for all containers (categories and course) add up to 100% except in the case where extra credit is used in which case 100% is exceeded by the weight of the extra credit

Setup screen and Grader report For the purposes of the calculations below the total category or course points is its total maximum points - (all of the below)

1. extra credit items' points
2. hidden items' points (unless hidetotalsthatcontainhiddenitems = GRADE_REPORT_SHOW_REAL_TOTAL_IF_CONTAINS_HIDDEN)
3. all points associated with contained items or categories who've had their weights overridden

User report For the purposes of the calculations below the total category or course points is its total maximum points - (all of the below)

1. extra credit items' points
2. hidden items' points (unless hidetotalsthatcontainhiddenitems = GRADE_REPORT_SHOW_REAL_TOTAL_IF_CONTAINS_HIDDEN)
3. ungraded items' points (if aggregateonlygraded is on))
4. any items dropped through container's drop lowest or keep highest conditions
5. all points associated with contained items or categories who've had their weights overridden

A multiplicator is used when some natural weights have been manually adjusted = sum(all adjusted weights) / 100

The natural weight for each item contained in a category or course (including extra credit items) is 'item maximum points' / 'total category or course points' (in which the item is contained) * multiplicator (if any)

The natural weight for each category (including extra credit categories) is 'total category points' / 'total category or course points' (in which the category is contained) * multiplicator (if any)

Calculation examples

Example 1 simple sum

In this case you have three assignments: A (10 points), B (10 points) and C (10 points) plus a final quiz D (20 points), and you just want their scores added together (out of 50).

No action is required of the user regarding weights.

Example 2 simple mean

In this case you have three assignments: A (10 points), B (10 points) and C (10 points) plus a final quiz D (20 points), and you want the final grade to be the simple mean percentage of all these scores.

User should adjust all weights to be the same, i.e., 25%

Example 3 typical weighted categories

User has four categories whose category totals are not in relation to each other as the user desires. They want the 'Quizzes' to be worth 20%, the 'Exams' to be worth 50% (with the midterm worth 20% and the final worth 30%), 'Papers' worth 20%, and 'Attendance' worth 10%

User would edit the weights and

- place the value 20 on the Quizzes category line

- place the value 50 on the Exams category line

- place the value 20 on the midterm exam item line

- place the value 30 on the final exam item line

- place the value 20 on the Papers category line

- place the value 20 on the Attendance category (or item) line

Then the user would save their weight settings

Example 4 (complex)

Category 1 contains seven items

1. Item A = 20 points
2. Item B = 10 points is extra credit
3. Item C = 30 points
4. Item D = 40 points is extra credit
5. Item E = 50 points is hidden
6. Item F = 50 points is ungraded
7. Item G = 10 points whose natural weight has been adjusted to 50% of the category grade

In the Setup screen and the Grader report the following conditions will exist

- Category grademax = 120 (10 + 50 + 30 + 10 + 20)

- Remainder minus adjusted weights = 110 (50 + 30 + 10 + 20) the 10 points adjusted out represent 50% of the final category grade

- Multiplicator for each weight calculation to adjust for the 50% of weight already accounted for = .5 (50% / 100%)

- A weight = 8.33% (20 / 120 * .5)

- B weight = 4.16% (10 / 120 * .5)

- C weight = 12.5% (30 / 120 * .5)

- D weight = 33.33% (40 / 120) extra credit is not adjust, retains its value regardless of weights being overridden

- E weight = 0%

- F weight = 25% (50 / 120 * .5)

- G weight = 50% (adjusted)

In the User report the following conditions will exist

- Category grademax = 70 (10 + 30 + 10 + 20)

- Remainder minus adjusted weights = 60 (30 + 10 + 20) the 10 points adjusted out represent 50% of the final category grade

- Multiplicator for each weight calculation to adjust for the 50% of weight already accounted for = .5 (50% / 100%)

- A weight = 16.66% (20 / 60 * .5)

- B weight = 8.33% (10 / 60 * .5)

- C weight = 25% (30 / 60 * .5)

- D weight = 66.66% (40 / 60) extra credit is not adjust, retains its value regardless of weights being overridden

- E weight = 0%

- F weight = 0%

- G weight = 50% (adjusted)