Natural weighting

Jump to: navigation, search
Natural Weighting
Project state Specification
Tracker issue MDL-43378
Discussion https://moodle.org/mod/forum/discuss.php?d=263318
Assignee Moodle HQ

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.


Natural weighting is a modification to how grades are combined.

When 'Sum of Grades' aggregation is chosen (and natural weighting is active), there will be a weight column with form fields, looking like "Weighted mean" is in 2.7. By default, the weight column shows the natural weights given to all categories and items based on gradebook structure.

Weights can be overridden by modifying the values in the form fields.

A "Reset weights" button in each category resets the weights to their defaults.

A clear overview of what is proposed here can be seen in https://www.youtube.com/watch?v=Qw9xtl9pGac&feature=youtu.be from about 1:30:00 to about 1:38:00.


Rationale

In grading, weight is the proportion of an item's value in relation to all the other items in its container (category or course).
Value is most often represented as 'points'.
Every grade has a weight.
There are three fundamental approaches to dealing with weighting in a gradebook
- Simple (or natural) weighting, maximum points earnable for each item are appropriately in relation to one another. Example - If you have two items in a category, one worth 10 points and the other worth 30, then item 1 is 25% (10/40) and item 2 is 75% (30/40).
- Adjusted weighting. Users often want to adjust categories and even items to have a weight other than the natural weight, essentially ignoring the points entirely. An example would be, Quizzes=40%, Assignments=50%, Attendance=10%.
- Mean of grades. If I have a category made up of five quizzes all worth different points and I want a category percentage to just tell the average (or mean) of all the quiz results' percentage correct, I would choose this approach again, disregarding the points.
Simplifying the understanding of weighting reduces the aggregation methodology down to 'either/ or'
- either 'leave weights as is -- natural'
- or 'adjust to suit my needs -- adjusted weights'
This greatly simplifies the interface needed to present these options
- They don't need to select an aggregation type
- They don't need to know how each aggregation type performs
- The actual weight of every item and category is always displayed to them.
- If those are not the weights they want, they merely adjust them.
- Much easier for the common user to grasp.

This interface and calculations are expected to suit the needs of all except very edge cases (which can still be handled with formulas). If it turns out that this is enough for everyone then we can potentially remove quite a lot of aggregation code from the gradebook at a later stage, improving speed and readability.

How natural weights meets the requirements of the aggregation types

Sum of grades
Moodle determines the "natural weights" for this by default based on point values.
Mean of grades
Set all weights equal to each other. Doesn't matter what the number is, as in legacy behavior Moodle will automatically adjust so the sum equals 100%. However, in this implementation the group wants to have a pop-up warning the user the weights don't add up to 100% and they will therefore be adjusted uniformly to equal 100%.
Weighted mean of grades
Adjust weights as desired. No change from current behavior.
Simple weighted mean of grades
Leave all weightings as is, no change necessary. Same as Sum of grades.
Mean of grades (with extra credits)
Extra credit has a natural weight that takes the total container (category/ course) weight above 100% (hence, 'extra'). Extra credit is available at all times no matter what scheme of weighting is chosen.
Highest 
User would input the value '1' for 'Keep highest' in the category
Lowest 
Of doubtful use. Not possible using natural weights, but it can also be done by a calculation into a new item.
Median / Mode of grades
Not possible using natural weights, but it's doubtful anyone uses these anyway so perhaps this can be deprecated entirely.

Implementation stages

To be safe, we plan to implement this in stages:

STAGE 1) Implement natural weighting as part of the Sum of Grades aggregation type, with admin settings that allow them to disable all other aggregations if they choose. (Moodle 2.8)

STAGE 2) Once people have been using this for a while, explore the possibility of removing other aggregation code or making them into optional plugins. (Moodle 3.0 perhaps)


Interfaces

Category editing

Prototype from luther.edu: NaturalWeights.png

In addition the weights should be editable form fields here, and there would be one button per category to "reset weights" back to a natural Sum of Grades.

Admin settings

New setting to "Force Natural Weighting" (on/off). This would default to OFF on upgraded sites and ON for new sites.

ON: this will hide the aggregation menu and all new categories will default to "Sum of grades" and natural weights. It will not affect existing gradebooks with other aggregations set (see upgrade script below for a conversion strategy).

OFF: aggregation menu is shown, but Sum of Grades will still contain new natural weighting UI and logic.

Upgrade script

If a site is switching to forced natural weights, they will probably want to upgrade all existing categories.

This new optional upgrade script will examine all grade categories looking at aggregation methods. It should report on what exists and allow the admin to perform an automated upgrade as follows:

  1. If GRADE_AGGREGATE_WEIGHTED_MEAN then:
    • Store value of 'aggregationcoef' fields to 'weight' fields in grade_items
    • Set 'weightoverride' field in grade_items
  2. If ... then:

(We need exact strategies here for ALL aggregation types)

Database changes

Add three fields to grade_items
  1. extracredit, tinyint, notnull, default: 0
  2. weight, decimal(10,5), notnull, default: 0
  3. weightoverride, tinyint, notnull, default: 0
Add one field to grade_grades
  1. weight, decimal(10,5), notnull, default: 0
This fixes the use of aggregationcoef representing both extra credit and weight in 2.7.

Unit test changes

Functional test changes

Logic changes

Sum of Grades aggregation must be fixed to exclude hidden items and to exclude non-graded items

Determination of range (in Grader and User report) and Max grade (in Setup screen) must be fixed to exclude extra credit items

Natural weights must be saved and applied to the form.

Natural weights are calculated for the Setup screen, the Grader report, and the User report. The calculation for the User report is different than that used for the Setup screen and Grader report.

Weights should be stored in grade_items for the Setup screen and the Grader report and in the grade_grades table for the User report.

Functional Logic

  1. calculate up each container’s (category or course items) adjusted weight percentage and grademax associated with the items adjusted (need this for weighting, so could be done here or after the next step)
  2. calculate points (must be first, you cannot calculate weights until you have the finalgrade and grademax for categories). For each gtree->levels starting with highest (items) to lowest (course) for both finalgrade and grademax
    1. include hidden only if hidetotalsifcontainhiddenitems is set to GRADE_REPORT_SHOW_REAL_TOTAL_IF_CONTAINS_HIDDEN
    2. ignore ungraded items if aggregateonlygraded is set, does not apply when calculating total points for Setup (Categories and items) screen
    3. include extra credit items in finalgrade calculations only, not grademax
    4. ignore items that have been dropped from drop low or keep high
    5. accumulate all other finalgrades and grademaxes into parent category finalgrade and grademaxes
  3. calculate weights (must be done before calculating contributions). For each gtree->levels starting with highest (items) to lowest (course)
    1. weight is always percentage grade * weight... in this case PERCENTAGE:[item grademax / (container grademax - adjusted grademax for the container from step 1)] * WEIGHT:[(100 - adjusted weight for the container from step 1)]
  4. Adjust weights in a container to always equal 100
  5. calculate contributions (needed in order to accurately drop lowest, keep highest. Also needed to provide the most percentages). For each gtree->levels starting with highest (items) to lowest (course)
    1. contribution is always the item’s percentage * the item’s weight * the container’s weight… so (finalgrade / grademax) * item weight * container weight
  6. limit categories - drop lowest, keep highest (need to have contributions in order to accurately drop low and keep high)
    1. remove contribs
    2. remove finalgrades from parent finalgrade
    3. remove maxgrade from parent maxgrade
  7. during the process of limiting categories the points for containers is adjusted so we don’t need to recalculate points
  8. recalculate weights since we’ve removed items by dropping
  9. Adjust weights in a container to always equal 100
  10. recalc contributions since we’ve adjusted weights because of dropped items

The Math

The sum of weights for all containers (categories and course) add up to 100% except in the case where extra credit is used in which case 100% is exceeded by the weight of the extra credit

Setup screen and Grader report For the purposes of the calculations below the total category or course points is its total maximum points - (all of the below)

  1. extra credit items' points
  2. hidden items' points (unless hidetotalsthatcontainhiddenitems = GRADE_REPORT_SHOW_REAL_TOTAL_IF_CONTAINS_HIDDEN)
  3. all points associated with contained items or categories who've had their weights overridden

User report For the purposes of the calculations below the total category or course points is its total maximum points - (all of the below)

  1. extra credit items' points
  2. hidden items' points (unless hidetotalsthatcontainhiddenitems = GRADE_REPORT_SHOW_REAL_TOTAL_IF_CONTAINS_HIDDEN)
  3. ungraded items' points (if aggregateonlygraded is on))
  4. any items dropped through container's drop lowest or keep highest conditions
  5. all points associated with contained items or categories who've had their weights overridden

A multiplicator is used when some natural weights have been manually adjusted = sum(all adjusted weights) / 100

The natural weight for each item contained in a category or course (including extra credit items) is 'item maximum points' / 'total category or course points' (in which the item is contained) * multiplicator (if any)

The natural weight for each category (including extra credit categories) is 'total category points' / 'total category or course points' (in which the category is contained) * multiplicator (if any)

Calculation examples

Example 1 simple sum

In this case you have three assignments: A (10 points), B (10 points) and C (10 points) plus a final quiz D (20 points), and you just want their scores added together (out of 50).

No action is required of the user regarding weights.

Example 2 simple mean

In this case you have three assignments: A (10 points), B (10 points) and C (10 points) plus a final quiz D (20 points), and you want the final grade to be the simple mean percentage of all these scores.

User should adjust all weights to be the same, i.e., 25%

Example 3 typical weighted categories

User has multiple categories whose category totals are not in relation to each other as the user desires. They want the 'Quizzes' to be worth 20%, the 'Exams' to be worth 50% (with the midterm worth 20% and the final worth 30%), 'Papers' worth 20%, and 'Attendance' worth 10%

User would edit the weights and
- place the value 20 on the Quizzes category line
- place the value 50 on the Exams category line
- place the value 20 on the midterm exam item line
- place the value 30 on the final exam item line
- place the value 20 on the Papers category line
- place the value 10 on the Attendance category (or item) line
Then the user would save their weight settings

Example 4 (complex)

Category 1 contains seven items

  1. Item A = 20 points
  2. Item B = 10 points is extra credit
  3. Item C = 30 points
  4. Item D = 40 points is extra credit
  5. Item E = 50 points is hidden
  6. Item F = 50 points is ungraded
  7. Item G = 10 points whose natural weight has been adjusted to 50% of the category grade
In the Setup screen and the Grader report the following conditions will exist
- Category grademax = 120 (10 + 50 + 30 + 10 + 20)
- Remainder minus adjusted weights = 110 (50 + 30 + 10 + 20) the 10 points adjusted out represent 50% of the final category grade
- Multiplicator for each weight calculation to adjust for the 50% of weight already accounted for = .5 (50% / 100%)
- A weight = 8.33% (20 / 120 * .5)
- B weight = 4.16% (10 / 120 * .5)
- C weight = 12.5% (30 / 120 * .5)
- D weight = 33.33% (40 / 120) extra credit is not adjust, retains its value regardless of weights being overridden
- E weight = 0%
- F weight = 25% (50 / 120 * .5)
- G weight = 50% (adjusted)
In the User report the following conditions will exist
- Category grademax = 70 (10 + 30 + 10 + 20)
- Remainder minus adjusted weights = 60 (30 + 10 + 20) the 10 points adjusted out represent 50% of the final category grade
- Multiplicator for each weight calculation to adjust for the 50% of weight already accounted for = .5 (50% / 100%)
- A weight = 16.66% (20 / 60 * .5)
- B weight = 8.33% (10 / 60 * .5)
- C weight = 25% (30 / 60 * .5)
- D weight = 66.66% (40 / 60) extra credit is not adjust, retains its value regardless of weights being overridden
- E weight = 0%
- F weight = 0%
- G weight = 50% (adjusted)

See also

- Specifications for Natural weighting (Google Doc) from the Moodle Gradebook Working Group

- How to fix grade book issues