MoodleDocs - User contributions [en]

Outcomes Instructor Specification

2013-08-15T17:38:47Z

Phillmiller: /* Instructor Student Completion Marking */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

(Deferred from PHase I)

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

[[File:Coverage Report - Callouts.png]]

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.
**Configure - This link launches a modal window which allows you to configure the selected Recommendation engine.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of recommendation engine, this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

*Action Buttons
**Apply to All - Applies Completed recommendations to all of the items where the recommendation suggests it.
**Save - Saves the Completion marking to the student record.

== Recommendation Configuration: Percentage Based ==
For the initial launch, we will be building a recommendation engine based on the percentage correct on a number of items. There are basically three outcomes:
**Complete - There is enough data and the student has cleared the threshold
**At Risk - There is enough data, and the student is below the At Risk Threshold
**Not enough data - There is either no data or not enough graded attempts

[[File:Percentage.png]]

*Proficiency Criteria
**Minimum number of graded data points - This defines how many points must have been completed prior to the engine making a recommendation.
**Minimum Percentage - This is the average percentage on the available data points completed by the student. This is a combination of graded quiz questions, rubric rows, or graded activities mapped against this outcome. Anything at this or above it will be shown a recommendation of complete.

*At Risk Criteria
**Minimum number of graded data points - This defines how many points must have been completed prior to the engine making a recommendation.
**Threshold Percentage - This is the average percentage on the available data points completed by the student. This is a combination of graded quiz questions, rubric rows, or graded activities mapped against this outcome. Anything below this threshold will be shown as At Risk.

==See Also ==
*[[Outcomes_Specification]]

Outcomes Administrator Specification

2013-08-15T17:37:21Z

Phillmiller: /* Achievement Configuration */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Checkbox: Display Individual Course Outcomes. This will show all outcome sets in the system, even ones added by individual instructors to their course directly. (Phase II - Leave out of Phase I)
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
**Globally Available - Default is true. This is a system level Outcome Set, rather than an individual course or instructor outcome set. (Phase II - Not available in Phase I)
**Author - If it is not global, then there must be an author who created it for a course, and that would be the owner. (Phase II - Not available in Phase I)
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

NOTE: Likely will be removed.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

File:Percentage.png

2013-08-13T15:28:33Z

Phillmiller:

Outcomes Instructor Specification

2013-08-13T15:28:10Z

Phillmiller:

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

(Deferred from PHase I)

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

[[File:Coverage Report - Callouts.png]]

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.
**Configure - This link launches a modal window which allows you to configure the selected Recommendation engine.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

*Action Buttons
**Apply to All - Applies Completed recommendations to all of the items where the recommendation suggests it.
**Save - Saves the Completion marking to the student record.

== Recommendation Configuration: Percentage Based ==
For the initial launch, we will be building a recommendation engine based on the percentage correct on a number of items. There are basically three outcomes:
**Complete - There is enough data and the student has cleared the threshold
**At Risk - There is enough data, and the student is below the At Risk Threshold
**Not enough data - There is either no data or not enough graded attempts

[[File:Percentage.png]]

*Proficiency Criteria
**Minimum number of graded data points - This defines how many points must have been completed prior to the engine making a recommendation.
**Minimum Percentage - This is the average percentage on the available data points completed by the student. This is a combination of graded quiz questions, rubric rows, or graded activities mapped against this outcome. Anything at this or above it will be shown a recommendation of complete.

*At Risk Criteria
**Minimum number of graded data points - This defines how many points must have been completed prior to the engine making a recommendation.
**Threshold Percentage - This is the average percentage on the available data points completed by the student. This is a combination of graded quiz questions, rubric rows, or graded activities mapped against this outcome. Anything below this threshold will be shown as At Risk.

==See Also ==
*[[Outcomes_Specification]]

File:Instructor Outcome Marking.png

2013-08-13T14:59:27Z

Phillmiller: Phillmiller uploaded a new version of "File:Instructor Outcome Marking.png"

Outcomes Instructor Specification

2013-08-13T14:53:28Z

Phillmiller: /* Instructor Student Completion Marking */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

(Deferred from PHase I)

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

[[File:Coverage Report - Callouts.png]]

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.
**Configure - This link launches a modal window which allows you to configure the selected Recommendation engine.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

*Action Buttons
**Apply to All - Applies Completed recommendations to all of the items where the recommendation suggests it.
**Save - Saves the Completion marking to the student record.

==See Also ==
*[[Outcomes_Specification]]

File:Outcomes Performance Report - Course.png

2013-08-13T14:50:57Z

Phillmiller: Phillmiller uploaded a new version of "File:Outcomes Performance Report - Course.png"

File:Question.png

2013-04-05T17:35:03Z

Phillmiller: uploaded a new version of "File:Question.png"

File:Activity - Mapped.png

2013-04-05T17:34:00Z

Phillmiller: uploaded a new version of "File:Activity - Mapped.png"

File:Activity - Empty.png

2013-04-05T17:33:39Z

Phillmiller: uploaded a new version of "File:Activity - Empty.png"

Outcomes Instructor Specification

2013-04-05T17:33:17Z

Phillmiller: /* Resource / Activity Mapping */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

(Deferred from PHase I)

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

[[File:Coverage Report - Callouts.png]]

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

Outcomes Instructor Specification

2013-04-05T17:32:54Z

Phillmiller: /* Course Settings and Course Outcomes */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

(Deferred from PHase I)

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

[[File:Coverage Report - Callouts.png]]

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Edit Course - Mapped.png

2013-04-05T17:22:56Z

Phillmiller: uploaded a new version of "File:Edit Course - Mapped.png"

File:Select Outcomes Sets.png

2013-04-05T17:20:46Z

Phillmiller: uploaded a new version of "File:Select Outcomes Sets.png"

File:Coverage Report - Callouts.png

2013-04-05T16:32:40Z

Phillmiller:

Outcomes Instructor Specification

2013-04-05T16:32:18Z

Phillmiller: /* Instructor Coverage Reporting */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

[[File:Coverage Report - Callouts.png]]

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

Outcomes Administrator Specification

2013-04-05T16:19:22Z

Phillmiller: /* Page Elements */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Checkbox: Display Individual Course Outcomes. This will show all outcome sets in the system, even ones added by individual instructors to their course directly. (Phase II - Leave out of Phase I)
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
**Globally Available - Default is true. This is a system level Outcome Set, rather than an individual course or instructor outcome set. (Phase II - Not available in Phase I)
**Author - If it is not global, then there must be an author who created it for a course, and that would be the owner. (Phase II - Not available in Phase I)
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

Outcomes Administrator Specification

2013-04-05T16:18:45Z

Phillmiller: /* Outcomes Administration Screen */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Checkbox: Display Individual Course Outcomes. This will show all outcome sets in the system, even ones added by individual instructors to their course directly. (Phase II - Leave out of Phase I)
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
**Globally Available - Default is true. This is a system level Outcome Set, rather than an individual course or instructor outcome set.
**Author - If it is not global, then there must be an author who created it for a course, and that would be the owner.
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

Outcomes Migration and Technical Issues

2013-04-05T16:09:22Z

Phillmiller: /* Introduction */

== Introduction ==
Because the Outcomes feature already exists, and because some clients are using it, we need to build a relatively elegant way to maintain data and to preserve the work that was already completed in previous versions. At Moodlerooms, our simple analysis of roughly 900 clients shows that only about 50 clients are using it, so roughly 5.5%. Of those that are using it, only 20 clients (2% have more than 500 graded outcomes).

NOTE: Moodlerooms will actually be running the two outcomes systems side by side for one of its releases to ease the transition.

== Migration Goals ==
* Preserve and convert all outcomes from previous versions into outcomes in the new system
* Preserve all mappings of outcomes to courses
* Preserve all mappings of outcomes to activities
* Elegantly remove the dependency on Scales from the mapping of outcomes.
* Maintain historically proficiency data by converting scales to proficient or not proficient.
* Create a unique identifier that will be unique outside of the Moodle Instance, so that Outcomes can be integrated with other platforms (Academic Benchmarks, xpLor, etc.)

==Outcome Migration==
* All migrated site wide outcomes will be placed into an outcome set called "Upgraded Site Outcomes"
* All course specific outcomes are migrated to a "Upgraded X Outcomes" set where X is the course shortname. So, one outcome set per course that has course specific outcomes.
*Users can later move the individual outcomes into new outcome sets that get created
*On outcome migration, we will store a reference to the old outcome and the old scale

==Scale Migration==
Migrating away from scales will be the most difficult part of this, and we are going to have slight data loss, although the output will actually be more meaningful. Here's the proposed plan:
*Prior to the upgrade, we will provide a way for administrators to go through the Scales that are tied to Outcomes, an define a threshold that means proficiency
*On upgrade, we will create new records for the proficiency tables that show whether or not the user is proficient based on their score on the scale
*We remove the scale as the outcome is upgraded.
*We need to store a link to the old outcome and to the old scale that was used, in order to deal with any migration problems.

== See Also ==
* [[Outcomes Specification]]

Outcomes Specification Change Log

2013-03-14T13:36:55Z

Phillmiller:

{| class="nicetable"
|-
! Issue
! Status
! Resolution/Description
! Date
|-
| Exporting Outcomes
| Closed
| Added use case. Updated Administrator spec to have export link.
| 2/16/13
|-
| Levels of Outcomes
| Closed
| Outcome Sets will be flat, Outcomes will be nested. Each Outcome can be configured as “assessable”, which determines whether the Outcome can be assigned to content and marked as achieved.
| 2/23/13
|-
| Should Outcome Text Allow HTML
| Closed
| Answer: No (They get displayed in too many places where HTML might not be good)
| 2/15/13
|-
| Restrict Outcomes by Category or Cohort
| Closed
| Answer: No.
| 2/15/13
|-
| Should outcome marking allow feedback?
| Closed
| Answer: No. (Potentially in future versions, but we have too many commenting places now as it is).
| 2/15/13
|-
| Should we allow tagging of outcomes/outcome sets for reporting?
| Closed
| Answer: No. (It would be nice, but we are cutting for scope. We should not design to where this could not be added in the future)
| 2/15/13
|-
| Review Academic Benchmarks XML
| Closed
| Sample format file received.
| 2/15/13
|-
| Migration of outcomes to a single outcome set.
| Closed
| Instead of migrating old outcomes to a single "Upgraded Outcomes" set, we migrate all site wide outcomes to "Upgraded Site Outcomes" and all course specific outcomes are migrated to a "Upgraded SHORTNAME Outcomes" where SHORTNAME is course shortname.
| 2/20/13
|-
| Add Use Case for Instructor adding Course Outcomes
| Closed
| Added a use case to the Instructor Use Cases section
| 2/24/13
|-
| Determine if we want to display outcomes to students ala feedback on quizzes.
| Open
|
|
|-
| Change the reports to be based off of “Completion” rather than “Resource Views”
| Closed
| Updated all reporting interfaces to be based on completion, rather than resource views.
| 2/24/13
|-
| Remove Outcome Set ShortName field as it is duplicative
| Closed
| Removed
| 3/14/13
|-
| On Outcome Creation, add Educational Level and Subject
| Closed
| Added
| 3/14/13
|-
| Collapse Outcome Sets, Outcomes, and Folders and allow them to be mapped, add a flag called assessable.
| Closed
| Collapsed these down and handled the multi level outcomes
| 3/14/13
|-
| Rename Outcome ID to Id Number, user editable, maps to Doc Num in AB and to Statement Notation in ASN
| Closed
| An alphanumeric notation or ID code as defined by the promulgating body to identify the statement.
|
|-
| Redo select Outcome Sets to make it flat, with optional filters for educational level and subject
| Open
|
|
|-
| Does Outcomes live in the Settings block? Gradebook or on their own?
| Open
|
|
|-
| How do we handle scale based items?
| Closed
| Added Scale Based Items to the Performance reports and a call out for scale based items.
| 3/14/13
|-
| Update Reports to show questions and rubric items under an assignment and quiz
| Closed
| Added
| 3/14
|}

Outcomes Specification Change Log

2013-03-14T13:36:25Z

Phillmiller:

{| class="nicetable"
|-
! Issue
! Status
! Resolution/Description
! Date
|-
| Exporting Outcomes
| Closed
| Added use case. Updated Administrator spec to have export link.
| 2/16/13
|-
| Levels of Outcomes
| Closed
| Outcome Sets will be flat, Outcomes will be nested. Each Outcome can be configured as “assessable”, which determines whether the Outcome can be assigned to content and marked as achieved.
| 2/23/13
|-
| Should Outcome Text Allow HTML
| Closed
| Answer: No (They get displayed in too many places where HTML might not be good)
| 2/15/13
|-
| Restrict Outcomes by Category or Cohort
| Closed
| Answer: No.
| 2/15/13
|-
| Should outcome marking allow feedback?
| Closed
| Answer: No. (Potentially in future versions, but we have too many commenting places now as it is).
| 2/15/13
|-
| Should we allow tagging of outcomes/outcome sets for reporting?
| Closed
| Answer: No. (It would be nice, but we are cutting for scope. We should not design to where this could not be added in the future)
| 2/15/13
|-
| Review Academic Benchmarks XML
| Closed
| Sample format file received.
| 2/15/13
|-
| Migration of outcomes to a single outcome set.
| Closed
| Instead of migrating old outcomes to a single "Upgraded Outcomes" set, we migrate all site wide outcomes to "Upgraded Site Outcomes" and all course specific outcomes are migrated to a "Upgraded SHORTNAME Outcomes" where SHORTNAME is course shortname.
| 2/20/13
|-
| Add Use Case for Instructor adding Course Outcomes
| Closed
| Added a use case to the Instructor Use Cases section
| 2/24/13
|-
| Determine if we want to display outcomes to students ala feedback on quizzes.
| Open
|
|
|-
| Change the reports to be based off of “Completion” rather than “Resource Views”
| Closed
| Updated all reporting interfaces to be based on completion, rather than resource views.
| 2/24/13
|-
| Remove Outcome Set ShortName field as it is duplicative
| Closed
| Removed
| 3/14/13
|-
| On Outcome Creation, add Educational Level and Subject
| Closed
| Added
| 3/14/13
|-
| Collapse Outcome Sets, Outcomes, and Folders and allow them to be mapped, add a flag called assessable.
| Closed
| 3/14/13
| Collapsed these down and handled the multi level outcomes
|-
| Rename Outcome ID to Id Number, user editable, maps to Doc Num in AB and to Statement Notation in ASN
| Closed
| An alphanumeric notation or ID code as defined by the promulgating body to identify the statement.
|
|-
| Redo select Outcome Sets to make it flat, with optional filters for educational level and subject
| Open
|
|
|-
| Does Outcomes live in the Settings block? Gradebook or on their own?
| Open
|
|
|-
| How do we handle scale based items?
| Closed
| Added Scale Based Items to the Performance reports and a call out for scale based items.
| 3/14/13
|-
| Update Reports to show questions and rubric items under an assignment and quiz
| Closed
| Added
| 3/14
|}

Outcomes Specification Change Log

2013-03-14T13:22:09Z

Phillmiller:

{| class="nicetable"
|-
! Issue
! Status
! Resolution/Description
! Date
|-
| Exporting Outcomes
| Closed
| Added use case. Updated Administrator spec to have export link.
| 2/16/13
|-
| Levels of Outcomes
| Closed
| Outcome Sets will be flat, Outcomes will be nested. Each Outcome can be configured as “assessable”, which determines whether the Outcome can be assigned to content and marked as achieved.
| 2/23/13
|-
| Should Outcome Text Allow HTML
| Closed
| Answer: No (They get displayed in too many places where HTML might not be good)
| 2/15/13
|-
| Restrict Outcomes by Category or Cohort
| Closed
| Answer: No.
| 2/15/13
|-
| Should outcome marking allow feedback?
| Closed
| Answer: No. (Potentially in future versions, but we have too many commenting places now as it is).
| 2/15/13
|-
| Should we allow tagging of outcomes/outcome sets for reporting?
| Closed
| Answer: No. (It would be nice, but we are cutting for scope. We should not design to where this could not be added in the future)
| 2/15/13
|-
| Review Academic Benchmarks XML
| Closed
| Sample format file received.
| 2/15/13
|-
| Migration of outcomes to a single outcome set.
| Closed
| Instead of migrating old outcomes to a single "Upgraded Outcomes" set, we migrate all site wide outcomes to "Upgraded Site Outcomes" and all course specific outcomes are migrated to a "Upgraded SHORTNAME Outcomes" where SHORTNAME is course shortname.
| 2/20/13
|-
| Add Use Case for Instructor adding Course Outcomes
| Closed
| Added a use case to the Instructor Use Cases section
| 2/24/13
|-
| Determine if we want to display outcomes to students ala feedback on quizzes.
| Open
|
|
|-
| Change the reports to be based off of “Completion” rather than “Resource Views”
| Closed
| Updated all reporting interfaces to be based on completion, rather than resource views.
| 2/24/13
|-
| Remove Outcome Set ShortName field as it is duplicative
| Closed
| Removed
| 3/14/13
|-
| On Outcome Creation, add Educational Level and Subject
| Closed
| 3/14/13
|
|-
| Collapse Outcome Sets, Outcomes, and Folders and allow them to be mapped, add a flag called assessable.
| Open
|
|
|-
| Rename Outcome ID to Id Number, user editable, maps to Doc Num in AB and to Statement Notation in ASN
| Open
| An alphanumeric notation or ID code as defined by the promulgating body to identify the statement.
|
|-
| Redo select Outcome Sets to make it flat, with optional filters for educational level and subject
| Open
|
|
|-
| Add Educational Level and Subject to the Outcome Set
| Open
|
|
|-
| Does Outcomes live in the Settings block? Gradebook or on their own?
| Open
|
|
|-
| How do we handle scale based items?
| Open
|
|
|-
| Update Reports to show questions and rubric items under an assignment and quiz
| Open
|
|
|}

File:Coverage Report - Course.png

2013-03-14T13:19:23Z

Phillmiller: uploaded a new version of "File:Coverage Report - Course.png"

Outcomes Instructor Specification

2013-03-14T13:16:24Z

Phillmiller: /* Instructor Student Completion Marking */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - If there are any scale graded resources that are mapped against this outcome, this link appears. Otherwise, N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Instructor Outcome Marking.png

2013-03-14T13:15:30Z

Phillmiller: uploaded a new version of "File:Instructor Outcome Marking.png"

Outcomes Instructor Specification

2013-03-14T13:13:07Z

Phillmiller: /* Instructor Student Performance Reporting */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items - For any scale graded resources that are mapped against this outcome, creates a link.. If there are no scale graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Individual Performance Report.png

2013-03-14T13:12:13Z

Phillmiller: uploaded a new version of "File:Individual Performance Report.png"

File:Activity and Grade Callouts.png

2013-03-14T13:09:49Z

Phillmiller: uploaded a new version of "File:Activity and Grade Callouts.png"

Outcomes Instructor Specification

2013-03-14T13:09:16Z

Phillmiller: /* Instructor Course Performance Reporting */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Scale Items – For any scale graded resources that are mapped against this outcome, this is the average grade of the students. If there are no scale activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Outcomes Performance Report - Course.png

2013-03-14T13:08:11Z

Phillmiller: uploaded a new version of "File:Outcomes Performance Report - Course.png"

File:Outcomes Performance Report - Administrator.png

2013-03-14T13:07:11Z

Phillmiller: uploaded a new version of "File:Outcomes Performance Report - Administrator.png"

Outcomes Instructor Specification

2013-03-13T17:14:55Z

Phillmiller: /* Empty State */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

== Empty State ==
[[File: Edit Course - Empty - Course Level.png]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Edit Course - Empty - Course Level.png

2013-03-13T17:14:29Z

Phillmiller:

Outcomes Instructor Specification

2013-03-13T17:14:04Z

Phillmiller:

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Course Settings and Course Outcomes ==
If instructors have the capabilities, they can map their course against outcomes that have already been added to the system, or they can create their own outcomes that are specific to the course. This all happens in the Course Settings Page.

== Empty State ==
[[File: Edit Course - Empty - Course Level]]

== Mapped State ==

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

Outcomes Capabilities and Roles

2013-03-13T17:06:23Z

Phillmiller:

The following will be capabilities and the roles which should have these capabilities selected by default:

{| class="nicetable"
|-
! Capability
! Description
! Roles with Capability Enabled by Default
|-
| Outcomes: Administration - Manage Outcomes
| Controls the ability to Add, Delete, Upload and Edit Outcomes from the Administration Screen
| System Administrator
|-
| Outcomes: Administration - Outcomes Reporting
| Controls the ability to view the Outcomes Reports in the Outcomes Administration Area
| System Administrator
|-
| Outcomes: Administration - Achievement Configuration
| Ability to configure how students are marked as having achieved an outcome and the recommendation engines.
| System Administrator
|-
| Outcomes: Course Mapping
| Controls the ability to map a course against an outcome set
| System Administrator, Manager, Course Creator, Teacher
|-
| Outcomes: Activity and Resource Mapping
| Controls the ability to map an activity, resource, or item against an outcome
| System Administrator, Manager, Course Creator, Teacher
|-
| Outcomes: Course Reporting
| Controls the ability to view the coverage and performance reports for mapped outcomes.
| System Administrator, Manager, Course Creator, Teacher
|-
| Outcomes: Add Course Outcomes
| Grants the ability to add outcomes at the course level that will be tied directly to that course/user.
| System Administrator, Manager, Course Creator, Teacher

|}

== See Also ==
* [[Outcomes Specification]]
* [[Outcomes Administrator Specification]]

File:Add-Edit Outcome Set.png

2013-03-13T17:04:06Z

Phillmiller: uploaded a new version of "File:Add-Edit Outcome Set.png"

Outcomes Administrator Specification

2013-03-13T17:02:51Z

Phillmiller: /* Page Elements */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Checkbox: Display Individual Course Outcomes. This will show all outcome sets in the system, even ones added by individual instructors to their course directly.
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
**Globally Available - Default is true. This is a system level Outcome Set, rather than an individual course or instructor outcome set.
**Author - If it is not global, then there must be an author who created it for a course, and that would be the owner.
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

File:Add-Edit Outcome Set.png

2013-03-13T17:00:50Z

Phillmiller: uploaded a new version of "File:Add-Edit Outcome Set.png"

Outcomes Administrator Specification

2013-03-13T16:58:06Z

Phillmiller: /* Page Elements */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Checkbox: Display Individual Course Outcomes. This will show all outcome sets in the system, even ones added by individual instructors to their course directly.
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

File:Outcomes Administration.png

2013-03-13T16:57:07Z

Phillmiller: uploaded a new version of "File:Outcomes Administration.png"

Outcomes Instructor Specification

2013-03-10T16:46:21Z

Phillmiller: /* Instructor Student Completion Marking */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to refine to other Outcomes.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The percentage of completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Recommendation – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Instructor Outcome Marking.png

2013-03-10T16:44:26Z

Phillmiller: uploaded a new version of "File:Instructor Outcome Marking.png"

Outcomes Instructor Specification

2013-03-10T16:30:08Z

Phillmiller: /* Instructor Student Performance Reporting */

== Description ==
Instructors are where Outcomes really get implemented. Instructors have the responsibility, in an outcomes driven system, to map content against outcomes, to monitor student progress, and to respond when students appear to have a problem.

== Resource / Activity Mapping ==
The most basic task that is required to make outcome driven learning work is the mapping of specific content items, resources, activities, and quiz questions, against outcomes. This takes place on the Activity creation/editing screen.

=== Empty State - Course Not Mapped ===

[[File:Activity - Empty.png]]

If the course has not been mapped against outcomes, it appears in this empty state. The user can click on Select Outcome Sets to choose outcomes sets to map against.

=== Empty State - Course Mapped, Activity Not Mapped ===

[[File:Activity - Course Mapped, not Activity.png]]

When a course is mapped against Outcome Sets, those sets appear in the Activity mapping screen. The user can select additional Outcome Sets, or they can click on the individual Outcome Sets to select specific outcomes to map against.

=== Outcome Mapping Screen ===

[[File:Outcome Mapper.png]]

When a user decides to map individual outcomes against a resource, this screen allows them to do so. They simply click on an outcome, which, once selected is highlighted. Once they click “OK” the outcomes are added back to the main resource screen, as shown below.

=== Mapped State - Course Mapped, Activity Mapped ===
[[File:Activity - Mapped.png]]

== Quiz Question Mapping ==
Mapping of quiz questions will happen identically to the way that regular content items are mapped, but on the quiz creation screen.

[[File:Question.png]]

== Rubric Mapping ==
Mappings against Rubric Rows will happen similar to content items, but will happen in the context of the criterion. This should be true of other advanced grading methodologies as well, with a framework for other grading methods to interact with outcomes.

[[File:Rubric.png]]

=== Add Outcome ===
This will launch the add outcome screen, allowing the user to select the outcome that maps against this particular criterion.

= Instructor Reporting =
One of the key drivers of Outcome driven design of courses is to allow instructors to see how their students are performing against outcomes and to provide accountability data on the instruction.

== Instructor Coverage Reporting ==
The coverage report shows all outcome sets that have been mapped against this course, lists out all outcomes within a given outcome set, and shows any resources, quiz questions, or activities that are mapped against those particular outcomes. The goal is to allow instructors to quickly see if they have covered all of the outcomes with content, and how deep that coverage is for each outcome.

[[File:Coverage Report - Course.png]]

*Filters:
**Outcome Set – Shows all outcome sets that are mapped against this course.

*Report:
Lists off ALL outcomes within an outcome set, including ones that have nothing mapped.
**ID: the logical identifier of the outcome.
**Outcome: The text of the outcome
**Resources: The number of resources mapped (on hover or click, pops up list)
**Gradeable Activities: # of gradeable activities mapped (on hover or click, pops up list)
**Questions: # of quiz questions mapped (on hover or click, pops up list)

Pagination and the ability to export are also included.

== Instructor Unmapped Content Report ==
This report shows all unmapped activities, resources, and questions. It gives the instructor the ability to jump directly to the item to map it, if necessary.
[[File:Unmapped Report.png]]

== Instructor Course Performance Reporting ==
Performance reporting is more complex, because it has to deal with various types of resources, and their relative importance. There are three types of reporting we want to achieve:
*Resource activity reporting: have there been interactions with resources that are mapped to outcomes.
*Completion Reporting: for instructors that use completion, we should report about the percentage completion for those items.
*Grade Reporting: What are the average grades for gradeable items that are mapped to outcomes.

[[File:Outcomes Performance Report - Course.png]]

*Filters:
**Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
**Group – Default is “All Groups.” Filters the performance report down to users in a specific group
*Report
**ID – Logical ID of the Outcome
**Outcome – Text of the outcome
**Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
**Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

The links for Completion and Grade Reports launch modal windows with the detailed information:

[[File:Activity and Grade Callouts.png]]

== Instructor Student Performance Reporting ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc.

[[File:Individual Performance Report.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**Filter by Outcome - Allows you to further refine the list of outcomes
**User – Roster of students
**Next Link – Takes me to the next Student

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Completion – The completion of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource completion. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.

== Instructor Student Completion Marking ==
On an individual student basis, we can actual display all of the standards with all associated activities, performance, interactions, etc. The Instructor can then mark them as complete based on the recommendations and data.

[[File:Instructor Outcome Marking.png]]

*Filters:
**Outcome Set – List of all outcome sets mapped to this course.
**User – Roster of students
**Next Link – Takes me to the next Student
**Recommendation - This changes which engine is used to provide the instructor with a recommendation for marking completion of outcomes. Initially, this will probably have None, Percentage Correct, and Streak.

*Report: This is a list of all outcomes in this outcome set, and the student’s associated performance.
**ID – Logical ID of the Outcome
**Outcome – Text of the Outcome
**Resource Views – The number of views of resources that are mapped against this outcome. Hyperlinked to a full log report of those resource views. If none, it is 0. If there are no resources, it is N/A.
**Average Grade - For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
**Recent Activity – This is the last assessable activity or streak. If they haven’t had any assessable activity against this standard, it is blank.
**Status – Based on this schools configuration of streaks and “At Risk” this will display either a check for “Mastery”, a warning sign for “At Risk”, and blank for any other state.
**Complete - Checkbox where instructor can mark whether or not they believe the student has completed this outcome, based on the percentage, etc.

==See Also ==
*[[Outcomes_Specification]]

File:Individual Performance Report.png

2013-03-10T16:28:17Z

Phillmiller: uploaded a new version of "File:Individual Performance Report.png"

File:Activity and Grade Callouts.png

2013-03-10T16:21:50Z

Phillmiller: uploaded a new version of "File:Activity and Grade Callouts.png"

Outcomes Administrator Specification

2013-03-10T16:14:56Z

Phillmiller: /* Page Elements */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

File:Upload Outcome Set.png

2013-03-10T16:14:36Z

Phillmiller: uploaded a new version of "File:Upload Outcome Set.png"

Outcomes Administrator Specification

2013-03-10T16:13:26Z

Phillmiller: /* Add New Outcome Set / Edit Outcome Set */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Tree View of Outcome Set – Default state is empty. As outcomes are added, it expands into a nested tree view.
***Outcome – An outcome can be nested, and each will have Add Edit Delete options that are visible on hover. On delete, there will be a warning.

In the pop-up for adding a new outcome, you have ID Number, Subject, and Education Level. Additionally, "Assessable" is a flag that allows it to be mapped and reported against. If Assessable is not checked, it can't be mapped at lower levels, and is largely used for organizational purposes.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Short Name – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]

File:Add-Edit Outcome Set.png

2013-03-10T16:10:38Z

Phillmiller: uploaded a new version of "File:Add-Edit Outcome Set.png"

Outcomes Administrator Specification

2013-03-10T16:10:20Z

Phillmiller: /* Page Elements */

== Description ==
The first step in building an outcomes based system inside of Moodle is to handle the Administration of standards. This includes the importing of standards from various sources and the mapping of standards to individual courses.

== Outcomes Administration Screen ==
The screen, Outcomes Administration, is accessed from the following path:
Settings > Site Administration > Grades > Outcomes

This screen lists off all outcome sets that have been added to the system, as well as providing access to the courses that are mapped to the standards and reports on the standards.
[[File:Outcomes Administration.png]]

=== Page Elements ===
*Title: Outcomes Administration
*Link: Add New Outcome Set – Links to the page where you can manually add a new Outcome Set.
*Link: Import Outcome Set – Links to the page where you can import XML, CSV, etc. Outcome Sets
*Data Grid: Displays all of the current outcome sets that are included in the system.
**Full Name – Name of the standards. Links to the editor for these outcomes.
**Mapped Courses – The number of courses that have been mapped against these outcomes. Links to a modal listing off the courses mapped to these outcome sets.
**Edit/Delete/Export – Takes you to the editor. On Delete, you are warned before deleting. Deleting is a “Soft Delete” so that the data stays in the database and student performance data is not lost. Export creates an xml file (or JSON file) of the outcomes.
**Reports – Links to the reports for coverage and performance for these standards.

== Add New Outcome Set / Edit Outcome Set ==
This screen allows you to create or modify an outcome set directly. It is used both to create outcome sets, and to update outcome sets that have been imported to clean up any problems with the import.
[[File:Add-Edit Outcome Set.png]]

=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Unique ID – The short name or logical identifier of the outcome set.
*Outcomes Fieldset
**Add Outcome Link – Pops up the modal for adding an outcome
**Add Folder – Pops up the modal for adding a folder
**Tree View of Outcome Set – Default state is empty. As outcomes and folders are added, it expands into a nested tree view.
***Folder – Folders have the options “Add Folder”, “Add Outcome” and “Delete”. “Add Folder” and “Add Outcome” create those elements as sub items. Delete warns before deleting, and deletes all nested outcomes.
***Outcome – An outcome is a leaf node in the tree. Only options are to edit and delete.

Delete Note – If outcomes have been deleted which have content mapped against the outcome that has been deleted, we should both warn and do a soft delete.

== Upload Outcome Set ==
Many schools will receive their outcomes from a third party body in XML or CSV format. This series of screens allows those items to be imported into the system.
[[File:Upload Outcome Set.png]]
=== Page Elements ===
*General Fieldset
**Name – Textual name of the outcome set.
**Short Name – The short name or logical identifier of the outcome set.
*File Fieldset
**This is a standard Moodle file upload area, where a CSV or XML File can be uploaded.

When the file is uploaded, it is validated that it is in a proper format and user is warned if it is not.

== XML File Upload ==
If the file that is uploaded is an XML file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:XML File.png]]
=== Page Elements ===
This is designed to allow a user to pick the element in the XML file that they can map against, or they can specify specific XPath. The Fields area allows you to choose available elements from the XML in a dropdown menu. The XPath allows you to specify specific XPath to get that item.

==CSV File Upload==
If the file that is uploaded is a CSV file, the screen displayed allows you to import that file by mapping the appropriate fields.

[[File:CSV File.png]]
===Page Elements ===
Dropdowns specify which column you want to map against that field (Column1, Column2). The checkbox for Column Headers means that the import process will ignore the first row.

==Add Course / Edit Course ==
After the outcome sets have been created or imported, individual courses need to be mapped to the outcome sets, so that the outcomes become available to individual instructors to map against content and assessment questions.

This will be achieved by updating the course creation screen to allow mapping. Additionally, this should be updated in the API, so that integrations against the SIS systems, etc. can be updated to make this an automatic process.

[[File:Edit Course - Empty.png]]

Empty State: If the course has not yet been mapped against outcome sets, the only change is to add a new Field Set "Outcomes", in which the course can be mapped against outcomes.

[[File:Edit Course - Mapped.png]]

Mapped State: If the course has already been mapped against outcome sets, those outcome sets appear, and the user has the option to delete them, or to map against additional Outcome Sets.

On delete, the user is warned that if they remove this mapping, it will remove mappings for content items, and could impact student outcome reporting.
*Note: We have to find a soft delete methodology that keeps data around for reporting and provisioning.

[[File:Select Outcomes Sets.png]]

== Achievement Configuration ==
NOTE: This area still under construction and discussion.

Based on some research that is out there (Khan Academy, etc.), we would like to add the (optional) capabilities to do automatic completion and at risk notifications, based on some crude, but highly accurate statistics. (More research needed here). Essentially, we would set up a streak or an at risk metric that flags this based on granular performance (either on a quiz question or being graded on a specific rubric row).

NOTE: This could become a pluggable system, where certain data (last 5 questions, current grade, etc.) are passed to a framework. This would allow people in the community to build their own proficiency models.

Sample of what this might look like:

[[File:Streaks.png]]

=== Streak Threshold ===
(More research needed here)
We will define a “streak” of the number of granular assessable items (quiz questions or scored rubric rows mapped to one outcome) that mark competency. The only configurable item here is what counts as “correct”. Default would be 100%, but could be adjusted downward.

=== At Risk Threshold ===
This can be a bit more flexible, but we would mark someone as “at risk” for an outcome if they have gotten a low enough grade on a graded activity mapped against an outcome, or if they have attempted X # of questions with a % below a certain threshold. (See individual performance report for more details).

= Reporting =
Add notes here.

== Coverage Report (Administrator) ==
This report shows which standards have some content mapped against them, both in aggregate and broken down by course.

[[File:Coverage Report - Administrators.png]]

=== Page Elements ===
*Outcome Set – Lists off the various outcome sets and allows selection, it is required to select an outcome set before running this report.
*Category – The category for course filtering. When category selected, it filters course list to courses within that category, mapped to this outcome set. Default is Any Category.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set. Furthermore, on selection of a course category, it is further refined.
*Group by Course Checkbox – If All Mapped Courses is selected, you can optionally choose to have the data show up in aggregate or grouped by course. If it is grouped by course, an additional column appears in the row for Course, and there is a row for every standard for every course. Data is then grouped into courses. Default is not checked. If Course dropdown is NOT “All Mapped Courses”, then this is disabled.

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Course – If Course is selected, the name of the course.
*Resources – Number of resources mapped against that outcome, linked to a detail report or modal that lists those resources.
*Gradeable Activities – Number of gradeable activities mapped against that outcome, linked to a detail report or modal that lists those activities.
*Resources – Number of Quiz Questions mapped against that outcome, linked to a detail report or modal that lists those quiz questions.
Note: All outcomes in this outcome set are displayed even if NO items are mapped against it.

== Performance Report (Administrator) ==
The performance report is used to show how courses are performing against particular outcomes. The report attempts to provide as much data about student interaction and performance with resources, activities, and quiz questions that are mapped against outcomes as possible.

NOTE: Currently, this depends on resource view logs, which Moodlerooms handles through its data collectors. This could be complex until a new logging framework is introduced in Moodle.

[[File:Outcomes Performance Report - Administrator.png]]

=== Page Elements ===
*Outcome Set: Lists off the various outcome sets and allows selection. It is required to select an outcome before running the report.
*Course – Default is “All”, which will run the report against all courses that are mapped against this Outcome Set. Upon Selection of Outcome Set, the dropdown is populated with courses that are mapped against this Outcome Set

Report
*ID – Logical ID of the Outcome
*Outcome – Text of the outcome
*Completion Percentage – This is the percentage completion by the students in courses who have access to the mapped activities. If there are no resources with completion turned on, it will read N/A.
*Average Grade – For any graded resources that are mapped against this outcome, this is the average grade of the students. If there are no graded activities, it will read N/A.
*Associated Activities – List of the mapped activities/resources. If the list is more than 3, then it will display ellipses (…) and you can hover or click to see the full list. On hover, it displays the individual item in the list with its own statistics.

== See Also ==
[[Outcomes Specification]]