Outcomes Administrator Specification

Description

The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

Outcomes Administration Screen

The screen, Outcomes Administration, is accessed from the following path: Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.

Page Elements

• Title: Outcomes Administration
• Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
• Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
• Checkbox: Display Individual Course Outcomes. This will show all outcome sets in the system, even ones added by individual instructors to their course directly.
• Data Grid: Displays all of the current outcome sets that are included in the system.
  • Full Name – Name of the standards. Links to the editor for these outcomes.
  • Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
  • Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
  • Reports – Links to the reports for coverage and performance for these standards.

Add New Outcome Set / Edit Outcome Set

This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.

Page Elements

• General Fieldset
  • Name – Textual name of the outcome set.
  • Unique ID – The short name or logical identifier of the outcome set.
  • Globally Available - Default is true. This is a system level Outcome Set, rather than an individual course or instructor outcome set.
  • Author - If it is not global, then there must be an author who created it for a course, and that would be the owner.
• Outcomes Fieldset
  • Add Outcome Link – Pops up the modal for adding an outcome
  • Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
    • Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

Upload Outcome Set

Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.

Page Elements

• General Fieldset
  • Name – Textual name of the outcome set.
  • Unique ID – The short name or logical identifier of the outcome set.
• File Fieldset
  • This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

XML File Upload

If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

Page Elements

This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

CSV File Upload

If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

Page Elements

Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

Add Course / Edit Course

After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.

• Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

Achievement Configuration

NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

Streak Threshold

(More research needed here) We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

At Risk Threshold

This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

Reporting

Add notes here.

Coverage Report (Administrator)

This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

Page Elements

• Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
• Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
• Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
• Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report

• ID – Logical ID of the Outcome
• Outcome – Text of the outcome
• Course – If Course is selected, the name of the course.
• Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
• Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
• Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.

Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

Performance Report (Administrator)

The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

Page Elements

• Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
• Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report

• ID – Logical ID of the Outcome
• Outcome – Text of the outcome
• Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
• Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
• Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

See Also

Outcomes Specification