Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Outcomes Technical Specification

From MoodleDocs

Introduction

Please be familiar with the function specification as outlined in the Outcomes Specification. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

In addition, some features and use cases are known but not yet being implemented. Please see Specifically Excluded Use Cases. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

Backwards compatibility breaks

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

  1. Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
  2. Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
  3. Outcomes will no longer appear in the grade book.
  4. Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

Settings

Site

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

Course

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

Activity

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

Advanced grading

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

Questions

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

Data structure

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

Outcomes ERD

outcome_sets

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve. Instructors can create personal outcome sets that can be used in their courses.

Field Type Description
id int(10), unique index Primary key
userid int(10), indexed, default null Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
idnumber varchar(255), unique index GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
name varchar(255) The name of the outcome set
description text, default null, optional The plain text description of the outcome.
provider varchar(255), default null, optional Provider information regarding the outcome set. EG: Could be set to ASN.
revision varchar(255), default null, optional Version information regarding the outcome set. This will typically be a date or year, EG: 2011
region varchar(255), default null, optional Any region information. Example: In the US this could be a state name.
deleted int(1), default 0 If the outcome set has been deleted.
timemodified int(10) Time modified.
timecreated int(10) Time created.

outcome

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

Field Type Description
id int(10), unique index Primary key
outcomesetid int(10), indexed Foreign key to outcome_sets table.
parentid int(10), indexed, default null Self referencing key. Identifies the parent outcome in the hierarchy.
idnumber varchar(255), unique index GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
name varchar(255) The name of the outcome.
docnum varchar(255), default null, optional Field for tracking additional information on the outcome. EG: 2.3.A
description text, default null, optional The plain text description of the outcome.
assessable int(1) If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
deleted int(1), default 0 If the outcome has been deleted.
sortorder int(10) The sort order. This is 0-X for all outcomes that belong to the same outcome set.
timemodified int(10) Time modified.
timecreated int(10) Time created.

outcome_metadata

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

Field Type Description
id int(10), unique index Primary key
outcomeid int(10), indexed Foreign key to outcome table.
name varchar(255), indexed The name of the meta data.
value varchar(1333), indexed The value of the meta data.

outcome_areas

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

Field Type Description
id int(10), unique index Primary key
component varchar(255), indexed The name of the component.
area varchar(255), default null, indexed, optional The name of the area. Used when a component has multiple items to associate to outcomes.
itemid int(10), indexed The item ID.
deleted int(1), default 0 Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.

outcome_area_outcomes

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

Field Type Description
id int(10), unique index Primary key
outcomeid int(10), unique index Foreign key to outcome table.
outcomeareaid int(10), unique index Foreign key to outcome_areas table.

outcome_used_areas

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

Field Type Description
id int(10), unique index Primary key
cmid int(10), indexed Foreign key to course_modules table.
outcomeareaid int(10), indexed Foreign key to outcome_areas table.

outcome_used_sets

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

Field Type Description
id int(10), unique index Primary key
courseid int(10), indexed Foreign key to course table.
outcomesetid int(10), indexed Foreign key to outcome_sets table.
filter text, default null Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.

outcome_attempts

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

Field Type Description
id int(10), unique index Primary key
outcomeusedareaid int(10), indexed Foreign key to outcome_used_areas table.
userid int(10), indexed Foreign key to user table.
percentgrade float(10,5) Percent grade earned on this attempt.
mingrade float(10,5) The minimum possible grade.
maxgrade float(10,5) The maximum possible grade.
rawgrade float(10,5) The points earned, should be between the minimum and maximum grades.
timecreated int(10) Time created.

outcome_marks

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

Field Type Description
id int(10), unique index Primary key
courseid int(10), indexed Foreign key to course table.
outcomeid int(10), indexed Foreign key to outcome table.
userid int(10), indexed Foreign key to user table. This is the user who is receiving the mark.
graderid int(10), indexed Foreign key to user table. This is the user is doing the marking.
result int(2) 0 is not pass, 1 is pass, null is that no decision has been made yet.
timemodified int(10) Time modified.
timecreated int(10) Time created.

outcome_marks_history

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

Field Type Description
id int(10), unique index Primary key
outcomemarkid int(10), indexed Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
courseid int(10), indexed Foreign key to course table.
outcomeid int(10), indexed Foreign key to outcome table.
userid int(10), indexed Foreign key to user table. This is the user who is receiving the mark.
graderid int(10), indexed Foreign key to user table. This is the user is doing the marking.
result int(2) 0 is not pass, 1 is pass, null is that no decision has been made yet.
timecreated int(10) Time created.

outcome_awards

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

Field Type Description
id int(10), unique index Primary key
outcomeid int(10), indexed Foreign key to outcome table.
userid int(10), indexed Foreign key to user table.
timecreated int(10) Time created.

Architecture

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

Models and repository mappers:

  • outcome: Represents a single outcome
  • outcome set: Represents a group of outcomes
  • outcome content: An outcome association to content.
  • outcome attempt: An attempt made against an outcome.
  • outcome mark: A teacher marking an outcome against a user.
  • outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

Collections:

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

  • outcomes
  • outcome attempts

New mform elements:

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

API for plugins

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

  • A way to notify the outcome system that a content item is or is not being used within a course.
    • Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
  • Record an attempt made against a content item.
    • Examples: A question has been attempted or an assignment submission has been graded.
  • A way to retrieve associated outcomes.
  • A way to update, delete or otherwise modify the associated outcomes.
  • A way to pick outcome associations (EG: what the mform element would do).
  • A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
  • API to cleanup tables on course deletion.

Supported plugin type plugins

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

  • Given an outcome association, please provide a link to the mapped content.
    • For an activity, this would be a link to the module.
    • For a grading form, a link to the grading instance.
    • For a question, a link to the question instance in the question bank.
  • What are all of the content items that have not been mapped?
    • For activities, this would return instances that are not mapped.
    • For grading form, this would return sub-plugin specific information, like criteria.
    • For questions, this would return questions from the course question bank that are not mapped.
  • May assist with automatic cleanup when the content is deleted.
  • Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

  • mod
  • qtype
  • gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

Import and export plugins

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from Achievement Standards Network (ASN) (Format Specification) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

Recommendation engine plugins

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

Reporting

Site

The following reports:

  • Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
  • Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

Course

The following reports:

  • Outcomes Coverage report - shows outcomes that have been mapped in this course.
  • Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
  • Outcomes Performance report - shows aggregated performance on outcomes in the course.
  • Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
  • Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

User profile

At the very least, there will be a user site report where the user can see earned outcomes.

Grading

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

Logging

We want to audit most actions in the Moodle log. These actions include:

  • CRUD type actions.
  • Any association editing. EG: associating an outcome with a course, content item, etc.
  • Outcome marking.
  • User attempts against outcomes.

Role capabilities

Description Capability
Map outcomes to content moodle/outcome:contentmap
Map outcome sets to courses moodle/outcome:coursemap
Administering outcomes (Editing, importing, etc) moodle/outcome:edit
Each report will have a capability to view them. report/REPORTNAME:view
Marking outcomes by graders moodle/grade:edit

Backup and restore

The workflow for backup and restore may go like:

  • Same site, then just map to existing outcomes via the identifier.
  • Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

  • Outcomes would plug into activities in order to backup outcome associations and attempts.
  • Outcomes would plug into questions in order to backup outcome associations and attempts.
  • Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

Migration of old outcomes

Please first review Migration and Technical Issues. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.