2.7 Outcomes scope

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

Purpose

This document aims to describe the enhancements to the Moodle outcomes system that will be coming to Moodle core soon. This is not a description of the outcome system. It is a description of the gap between the documents and the current code.

Code can be found in the Moodle outcomes repository, within the 40230-27 branch.

The following can be used to add this repository as a remote to any standard Moodle:

# Add the remote
git remote add moodle-outcomes https://bitbucket.org/samhemelryk/moodle-outcomes && git remote set-url --push moodle-outcomes git@bitbucket.org:samhemelryk/moodle-outcomes.git
# Fetch from the newly added moodle-outcomes repository
git fetch moodle-outcomes
# Checkout a local branch that will track the outcome code as it develops.
git checkout -b 40230-27 moodle-outcomes/40230-27

Sources

The items listed below have come from two sources:

1. Items that exist in the 2.6 specification (https://docs.moodle.org/dev/Outcomes_Specification) and related documents but which do not exist in the code. The specification has been compared against the code at https://github.com/andyjdavis/moodle/commits/MDL-40230_outcomes.
2. Feedback received on MDL-40230.

Issues By User Type

Admin

https://docs.moodle.org/dev/Outcomes_Administrator_Use_Cases

https://docs.moodle.org/dev/Outcomes_Administrator_Specification

Import Outcome Set

1. Academic Benchmark - http://www.academicbenchmarks.com/ | http://www.academicbenchmarks.com/component/content/article/42/246-sampleccssxml. This appears to be done.
2. ASN - ie http://asn.jesandco.org/resources/ASNJurisdiction/AK. This appears to be done.
3. General Moodle Format - This is the XML format that we export.
4. CSV - Described in the spec https://docs.moodle.org/dev/Outcomes_Administrator_Specification#CSV_File_Upload but does not appear to exist in the code. We can export the various outcome reports as csv but not the outcome set itself.

There is a general purpose XML import described in the spec which does not appear to exist in the code. https://docs.moodle.org/dev/Outcomes_Administrator_Specification#XML_File_Upload

Export Outcome Set

User can export as XML. This is the "General Moodle Format" which could possibly do with renaming. Do we want to support any other formats?

Administrator Mapping Outcomes to a Course

The user maps outcome sets to the course on the course settings page. There is a page at course administration > Outcomes which you use to access information about the existing mappings of outcomes within the course. Should the mapping of outcome sets to the course be done there too?

Administrator Deleting Outcomes (Soft Delete)

Deleted outcomes still exist in the database. However, I don't think this work is entirely complete. Deleting an individual outcome causes it to disappear from the UI everywhere including currently mapped activity settings and the Activity and performance page.

Administrator wants to import K-12 State Standards and Associate with the appropriate courses

https://docs.moodle.org/dev/Outcomes_Administrator_Use_Cases#Administrator_wants_to_import_K-12_State_Standards_and_Associate_with_the_appropriate_courses Is this the purpose of the subject and year drop downs when the user is mapping an outcome set to a course?

Administrator wants to include a new version of a K-12 State Standards Package

Functionality to allow updating of outcome sets does not appear to exist.

Administrator configures criteria for how a student is marked as having achieved competence on an outcome

Does not appear to exist. This may have been decided against. https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Achievement_Configuration

Administrator Generates Report to Show that Courses Cover All Required Outcomes

There is a coverage report for a single course. There is no report that allows the user to select an outcome set and see its coverage across courses. Links are shown in the spec but don't exist in the code.

https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Outcomes_Administration_Screen

https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Coverage_Report_.28Administrator.29

Administrator Generates Report to Show the Performance of Various Courses Against Outcomes

An Activity and performance report exists for a single course. There is no report that allows the user to select an outcome set and see student performance across courses. Links are shown in the spec but don't exist in the code.

https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Outcomes_Administration_Screen

https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Performance_Report_.28Administrator.29

Teacher

https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases

https://docs.moodle.org/dev/Outcomes_Instructor_Specification

Instructor Browses Available Outcomes and Maps Course to Appropriate Outcomes

The user can select outcomes on both the activity setting page and on the rubric. The two sets of outcomes do not appear to be related.

The rubric could be limited to the outcomes in the activity settings however a rubric can be used on multiple assignments. Perhaps the outcomes section of the activity settings should be disabled as soon as "rubric" is selected.

Instructor Maps Quiz Questions Against Standards

The user can associate outcomes with either a quiz or individual questions. Should there be a relationship between the outcomes associated with the quiz and the questions? Should the outcomes for a quiz simply be the aggregate of the outcomes for the individual questions?

Instructor Uses Other Advanced Grading Methods With Outcomes

"The frameworks should make it possible for Advanced Grading should allow outcomes to flow through, if the developer for that advanced grading chooses to implement it."

Does this exist? Investigation required to determine if the advanced grading API has been implemented.

Instructor is Required to Show Student Performance Against Outcomes

https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases#Instructor_is_Required_to_Show_Student_Performance_Against_Outcomes

This does not exist as described. Currently there is the "Activity and performance" report which allows the user to view performance for all outcomes by group. They can use the "Completion marking" page to view an individual student's performance against all outcomes. There does not seem to be anywhere where you can view the performance of all students against a single outcome.

K-12 Instructor in U.S. has to Provide Data of Student Performance Against State Standards

All of the outcomes pages allow exporting of the displayed table. This is presumably adequate but confirmation of this would be great.

Backup / Restore Common Cartridge with Outcome ID

https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases#Backup_.2F_Restore_Common_Cartridge_with_Outcome_ID

Does this exist?

K-12 Instructor wants to use Moodle Sections as a Lesson Plan (Optional)

https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases#K-12_Instructor_wants_to_use_Moodle_Sections_as_a_Lesson_Plan_.28Optional.29

This does not currently exist. A page showing the used outcomes grouped by course section could be handy.

Completion marking

https://docs.moodle.org/dev/Outcomes_Instructor_Specification#Instructor_Course_Performance_Reporting

The completion marking page shown in the specific appears to have more columns than the current version in the code. Is there more work to be done here?

Outcomes Activity and Performance by Student

https://docs.moodle.org/dev/Outcomes_Instructor_Specification#Instructor_Student_Performance_Reporting

This closely resembles the completion marking page. Is there more work to do or does the completion marking page fill this role?

Recommendation Configuration

https://docs.moodle.org/dev/Outcomes_Instructor_Specification#Recommendation_Configuration:_Percentage_Based

This does not appear to exist.

Student

https://docs.moodle.org/dev/Outcomes_Student_Use_Cases

https://docs.moodle.org/dev/Outcomes_Student_Specification

Student Begins Course, and Views All Outcomes Required for this Course

https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Student_Begins_Course.2C_and_Views_All_Outcomes_Required_for_this_Course

https://docs.moodle.org/dev/Outcomes_Student_Specification#Course_Outcomes_-_Student_View

This does not appear to exist.

Student Views Assignment Rubric and Associated Outcomes

https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Student_Views_Assignment_Rubric_and_Associated_Outcomes

The student can view the rubric which contains rows copied from the outcomes. There is no link to anywhere that might provide more information about the outcome. Should there be?

Student Successfully Masters Outcomes in a Course

https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Student_Successfully_Masters_Outcomes_in_a_Course

Does not appear to exist. "Student Begins Course, and Views All Outcomes Required for this Course" will partially resolve this although additional work will be required if outcomes data is to be displayed on student profiles. https://docs.moodle.org/dev/Outcomes_Student_Specification#Profile_Page_-_Overview

Graduating Student Required to Provide Evidence of Skills

https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Graduating_Student_Required_to_Provide_Evidence_of_Skills

Does not appear to exist. This is another item that requires outcomes data to be displayed on student profiles.

Outcomes API to send Performance Data to Portfolio System

https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Outcomes_API_to_send_Performance_Data_to_Portfolio_System

Does not appear to exist.

Open Questions

Badges

How does the outcomes system connect/relate to the badge system? If an instructor wanted achieving an outcome to result in a badge being awarded, how would they do that?

To Do

This is a summary of the tasks that come out of the above issues.

Pre-Coding Tasks

What about upgrades? We don't particularly want to run two outcome systems at the same time. Removing the old one would require us to have an upgrade path. There have been some notes made at https://docs.moodle.org/dev/Migration_and_Technical_Issues

Review and clarify export and import options.

Figure out how badges and outcomes should work together. Is this as simple as adding a new criteria for badges?

Figure out how rubric assignments should handle outcomes. https://docs.moodle.org/dev/2.7_Outcomes_rubric_outcomes

Figure out how quizzes/questions should handle outcomes. This should probably (but may not necessarily) work the same way as rubric assignments.

Why do you consider this is a necessary pre-coding task? There was a great discussion when Moodlerooms were developing their spec, which fed into their spec. Then they implemented that. What do you think is wrong with what was done?--Tim Hunt (talk) 13:14, 19 November 2013 (WST)

The issue is that both questions and quizzes can be mapped to outcomes but the two sets of mappings are independent. I can associate the quiz activity itself with a given set of outcomes while the quiz is composed entirely of questions that are mapped to a different set of outcomes. This is not necessarily wrong but we need to clarify whether the outcomes from the questions trump the outcomes from the quiz, whether the quiz should even be able to have it's own set of outcomes, whether we effectively union them all together and if there is some way to help the user understand this relationship.--Andrew Davis (talk) 15:19, 19 November 2013 (WST)

Right, so you don't understand this bit of the system. Discuss it with the folks from Moodlerooms, or ask me some time, and we will explain.--Tim Hunt (talk) 17:11, 20 November 2013 (WST)

Well, OK, I will try to explain briefly now. The point is that questions can be used in many different quizzes. And a question might be capable of assessing several different outcomes. In a particular quiz, some of those outcomes might be relevant, and others not. The quiz is designed to test certain things, and not others, so we indicate which outcomes that are declared by the questions are relevant to this quiz. For example, suppose we have questions Q1, testing outcome A, Q2 testing B & C, Q3 testing A & D, and Q4 testing B. We might assemble those into a quiz that does a reasonable job of measuring A & B. It does not give enough significant data to report meaningfully on C & D, so we don't want to track those outcomes for this quiz. However, Q2 & Q3 might later be reused in another quiz which is all about C & D.--Tim Hunt (talk) 17:11, 20 November 2013 (WST)

Missing or Incomplete Components

Soft deletion of outcomes

Updating of imported outcomes

Reports showing:

1. Outcome coverage across courses
2. Outcome performance across courses
3. Performance of all students against a single outcome
4. Outcomes used, grouped by course section
5. All outcomes for a course (for student)

Backup / Restore Common Cartridge with Outcome ID

Recommendation configuration

Ability to display completed outcomes on profile

Ability to send performance data to an external portfolio