2.7 Outcomes scope: Difference between revisions
Andrew Davis (talk | contribs) (→Admin) |
(Note about plan not to migrate this page to the new developer resources. See template for more info.) |
||
(27 intermediate revisions by 5 users not shown) | |||
Line 1: | Line 1: | ||
{{Template:WillNotMigrate}} | |||
=Purpose= | =Purpose= | ||
This document aims to describe the enhancements to the Moodle outcomes system that will be | This document aims to describe the enhancements to the Moodle outcomes system that will be coming to Moodle core soon. This is not a description of the outcome system. It is a description of the gap between the documents and the current code. | ||
Code can be found in the [https://bitbucket.org/samhemelryk/moodle-outcomes Moodle outcomes repository], within the [https://bitbucket.org/samhemelryk/moodle-outcomes/branch/40230-27 40230-27] branch. | |||
The following can be used to add this repository as a remote to any standard Moodle: | |||
<syntaxhighlight lang="bash"> | |||
# Add the remote | |||
git remote add moodle-outcomes https://bitbucket.org/samhemelryk/moodle-outcomes && git remote set-url --push moodle-outcomes git@bitbucket.org:samhemelryk/moodle-outcomes.git | |||
# Fetch from the newly added moodle-outcomes repository | |||
git fetch moodle-outcomes | |||
# Checkout a local branch that will track the outcome code as it develops. | |||
git checkout -b 40230-27 moodle-outcomes/40230-27 | |||
</syntaxhighlight> | |||
=Sources= | =Sources= | ||
Line 8: | Line 21: | ||
#Feedback received on MDL-40230. | #Feedback received on MDL-40230. | ||
= | =Issues By User Type= | ||
==Admin== | ==Admin== | ||
https://docs.moodle.org/dev/Outcomes_Administrator_Use_Cases | https://docs.moodle.org/dev/Outcomes_Administrator_Use_Cases | ||
Line 16: | Line 29: | ||
'''Import Outcome Set''' | '''Import Outcome Set''' | ||
#Academic Benchmark | #Academic Benchmark - http://www.academicbenchmarks.com/ | http://www.academicbenchmarks.com/component/content/article/42/246-sampleccssxml. This appears to be done. | ||
#ASN | #ASN - ie http://asn.jesandco.org/resources/ASNJurisdiction/AK. This appears to be done. | ||
#General Moodle Format | #General Moodle Format - This is the XML format that we export. | ||
#CSV | #CSV - Described in the spec https://docs.moodle.org/dev/Outcomes_Administrator_Specification#CSV_File_Upload but does not appear to exist in the code. We can export the various outcome reports as csv but not the outcome set itself. | ||
There is XML import described in the spec which does not appear to exist in the code. https://docs.moodle.org/dev/Outcomes_Administrator_Specification#XML_File_Upload | There is a general purpose XML import described in the spec which does not appear to exist in the code. https://docs.moodle.org/dev/Outcomes_Administrator_Specification#XML_File_Upload | ||
'''Export Outcome Set''' | '''Export Outcome Set''' | ||
User can export as XML. | User can export as XML. This is the "General Moodle Format" which could possibly do with renaming. Do we want to support any other formats? | ||
'''Administrator Mapping Outcomes to a Course''' | '''Administrator Mapping Outcomes to a Course''' | ||
The user maps outcome sets to the course on the course settings page. | |||
There is a page at course administration > Outcomes which you use to access information about the | There is a page at course administration > Outcomes which you use to access information about the existing mappings of outcomes within the course. Should the mapping of outcome sets to the course be done there too? | ||
'''Administrator Deleting Outcomes (Soft Delete)''' | '''Administrator Deleting Outcomes (Soft Delete)''' | ||
Deleted outcomes still exist in the database. | Deleted outcomes still exist in the database. However, I don't think this work is entirely complete. Deleting an individual outcome causes it to disappear from the UI everywhere including currently mapped activity settings and the Activity and performance page. | ||
However, I don't think this is entirely | |||
'''Administrator wants to import K-12 State Standards and Associate with the appropriate courses''' | '''Administrator wants to import K-12 State Standards and Associate with the appropriate courses''' | ||
Line 48: | Line 56: | ||
'''Administrator wants to include a new version of a K-12 State Standards Package''' | '''Administrator wants to include a new version of a K-12 State Standards Package''' | ||
Functionality to allow updating of outcome sets does not appear to exist. | |||
'''Administrator configures criteria for how a student is marked as having achieved competence on an outcome''' | '''Administrator configures criteria for how a student is marked as having achieved competence on an outcome''' | ||
Line 69: | Line 77: | ||
https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Performance_Report_.28Administrator.29 | https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Performance_Report_.28Administrator.29 | ||
==Teacher== | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Specification | |||
'''Instructor Browses Available Outcomes and Maps Course to Appropriate Outcomes ''' | |||
The user can select outcomes on both the activity setting page and on the rubric. The two sets of outcomes do not appear to be related. | |||
The rubric could be limited to the outcomes in the activity settings however a rubric can be used on multiple assignments. Perhaps the outcomes section of the activity settings should be disabled as soon as "rubric" is selected. | |||
'''Instructor Maps Quiz Questions Against Standards''' | |||
The user can associate outcomes with either a quiz or individual questions. Should there be a relationship between the outcomes associated with the quiz and the questions? Should the outcomes for a quiz simply be the aggregate of the outcomes for the individual questions? | |||
'''Instructor Uses Other Advanced Grading Methods With Outcomes''' | |||
"The frameworks should make it possible for Advanced Grading should allow outcomes to flow through, if the developer for that advanced grading chooses to implement it." | |||
Does this exist? Investigation required to determine if the advanced grading API has been implemented. | |||
'''Instructor is Required to Show Student Performance Against Outcomes''' | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases#Instructor_is_Required_to_Show_Student_Performance_Against_Outcomes | |||
This does not exist as described. Currently there is the "Activity and performance" report which allows the user to view performance for all outcomes by group. They can use the "Completion marking" page to view an individual student's performance against all outcomes. There does not seem to be anywhere where you can view the performance of all students against a single outcome. | |||
'''K-12 Instructor in U.S. has to Provide Data of Student Performance Against State Standards''' | |||
All of the outcomes pages allow exporting of the displayed table. This is presumably adequate but confirmation of this would be great. | |||
'''Backup / Restore Common Cartridge with Outcome ID''' | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases#Backup_.2F_Restore_Common_Cartridge_with_Outcome_ID | |||
Does this exist? | |||
'''K-12 Instructor wants to use Moodle Sections as a Lesson Plan (Optional)''' | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases#K-12_Instructor_wants_to_use_Moodle_Sections_as_a_Lesson_Plan_.28Optional.29 | |||
This does not currently exist. A page showing the used outcomes grouped by course section could be handy. | |||
'''Completion marking''' | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Specification#Instructor_Course_Performance_Reporting | |||
The completion marking page shown in the specific appears to have more columns than the current version in the code. Is there more work to be done here? | |||
'''Outcomes Activity and Performance by Student''' | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Specification#Instructor_Student_Performance_Reporting | |||
This closely resembles the completion marking page. Is there more work to do or does the completion marking page fill this role? | |||
'''Recommendation Configuration''' | |||
https://docs.moodle.org/dev/Outcomes_Instructor_Specification#Recommendation_Configuration:_Percentage_Based | |||
This does not appear to exist. | |||
==Student== | |||
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases | |||
https://docs.moodle.org/dev/Outcomes_Student_Specification | |||
'''Student Begins Course, and Views All Outcomes Required for this Course''' | |||
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Student_Begins_Course.2C_and_Views_All_Outcomes_Required_for_this_Course | |||
https://docs.moodle.org/dev/Outcomes_Student_Specification#Course_Outcomes_-_Student_View | |||
This does not appear to exist. | |||
'''Student Views Assignment Rubric and Associated Outcomes''' | |||
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Student_Views_Assignment_Rubric_and_Associated_Outcomes | |||
The student can view the rubric which contains rows copied from the outcomes. There is no link to anywhere that might provide more information about the outcome. Should there be? | |||
'''Student Successfully Masters Outcomes in a Course''' | |||
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Student_Successfully_Masters_Outcomes_in_a_Course | |||
Does not appear to exist. "Student Begins Course, and Views All Outcomes Required for this Course" will partially resolve this although additional work will be required if outcomes data is to be displayed on student profiles. https://docs.moodle.org/dev/Outcomes_Student_Specification#Profile_Page_-_Overview | |||
'''Graduating Student Required to Provide Evidence of Skills''' | |||
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Graduating_Student_Required_to_Provide_Evidence_of_Skills | |||
Does not appear to exist. This is another item that requires outcomes data to be displayed on student profiles. | |||
'''Outcomes API to send Performance Data to Portfolio System''' | |||
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases#Outcomes_API_to_send_Performance_Data_to_Portfolio_System | |||
Does not appear to exist. | |||
=Open Questions= | |||
==Badges== | |||
How does the outcomes system connect/relate to the badge system? If an instructor wanted achieving an outcome to result in a badge being awarded, how would they do that? | |||
=To Do= | |||
This is a summary of the tasks that come out of the above issues. | |||
==Pre-Coding Tasks== | |||
What about upgrades? We don't particularly want to run two outcome systems at the same time. Removing the old one would require us to have an upgrade path. There have been some notes made at https://docs.moodle.org/dev/Migration_and_Technical_Issues | |||
[https://tracker.moodle.org/browse/MDL-40230?focusedCommentId=257982&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-257982 Review and clarify export and import options.] | |||
Figure out how badges and outcomes should work together. Is this as simple as adding a new criteria for badges? | |||
Figure out how rubric assignments should handle outcomes. https://docs.moodle.org/dev/2.7_Outcomes_rubric_outcomes | |||
Figure out how quizzes/questions should handle outcomes. This should probably (but may not necessarily) work the same way as rubric assignments. | |||
:''Why do you consider this is a necessary pre-coding task? There was a great discussion when Moodlerooms were developing their spec, which fed into their spec. Then they implemented that. What do you think is wrong with what was done?''--[[User:Tim Hunt|Tim Hunt]] ([[User talk:Tim Hunt|talk]]) 13:14, 19 November 2013 (WST) | |||
::''The issue is that both questions and quizzes can be mapped to outcomes but the two sets of mappings are independent. I can associate the quiz activity itself with a given set of outcomes while the quiz is composed entirely of questions that are mapped to a different set of outcomes. This is not necessarily wrong but we need to clarify whether the outcomes from the questions trump the outcomes from the quiz, whether the quiz should even be able to have it's own set of outcomes, whether we effectively union them all together and if there is some way to help the user understand this relationship.''--[[User:Andrew Davis|Andrew Davis]] ([[User talk:Andrew Davis|talk]]) 15:19, 19 November 2013 (WST) | |||
:::''Right, so you don't understand this bit of the system. Discuss it with the folks from Moodlerooms, or ask me some time, and we will explain.''--[[User:Tim Hunt|Tim Hunt]] ([[User talk:Tim Hunt|talk]]) 17:11, 20 November 2013 (WST) | |||
:::''Well, OK, I will try to explain briefly now. The point is that questions can be used in many different quizzes. And a question might be capable of assessing several different outcomes. In a particular quiz, some of those outcomes might be relevant, and others not. The quiz is designed to test certain things, and not others, so we indicate which outcomes that are declared by the questions are relevant to this quiz. For example, suppose we have questions Q1, testing outcome A, Q2 testing B & C, Q3 testing A & D, and Q4 testing B. We might assemble those into a quiz that does a reasonable job of measuring A & B. It does not give enough significant data to report meaningfully on C & D, so we don't want to track those outcomes for this quiz. However, Q2 & Q3 might later be reused in another quiz which is all about C & D.--[[User:Tim Hunt|Tim Hunt]] ([[User talk:Tim Hunt|talk]]) 17:11, 20 November 2013 (WST) | |||
==Missing or Incomplete Components== | |||
Soft deletion of outcomes | |||
Updating of imported outcomes | |||
Reports showing: | |||
#Outcome coverage across courses | |||
#Outcome performance across courses | |||
#Performance of all students against a single outcome | |||
#Outcomes used, grouped by course section | |||
#All outcomes for a course (for student) | |||
Backup / Restore Common Cartridge with Outcome ID | |||
Recommendation configuration | |||
Ability to display completed outcomes on profile | |||
Ability to send performance data to an external portfolio |
Latest revision as of 11:36, 25 June 2022
Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable. |
Purpose
This document aims to describe the enhancements to the Moodle outcomes system that will be coming to Moodle core soon. This is not a description of the outcome system. It is a description of the gap between the documents and the current code.
Code can be found in the Moodle outcomes repository, within the 40230-27 branch.
The following can be used to add this repository as a remote to any standard Moodle:
# Add the remote
git remote add moodle-outcomes https://bitbucket.org/samhemelryk/moodle-outcomes && git remote set-url --push moodle-outcomes git@bitbucket.org:samhemelryk/moodle-outcomes.git
# Fetch from the newly added moodle-outcomes repository
git fetch moodle-outcomes
# Checkout a local branch that will track the outcome code as it develops.
git checkout -b 40230-27 moodle-outcomes/40230-27
Sources
The items listed below have come from two sources:
- Items that exist in the 2.6 specification (https://docs.moodle.org/dev/Outcomes_Specification) and related documents but which do not exist in the code. The specification has been compared against the code at https://github.com/andyjdavis/moodle/commits/MDL-40230_outcomes.
- Feedback received on MDL-40230.
Issues By User Type
Admin
https://docs.moodle.org/dev/Outcomes_Administrator_Use_Cases
https://docs.moodle.org/dev/Outcomes_Administrator_Specification
Import Outcome Set
- Academic Benchmark - http://www.academicbenchmarks.com/ | http://www.academicbenchmarks.com/component/content/article/42/246-sampleccssxml. This appears to be done.
- ASN - ie http://asn.jesandco.org/resources/ASNJurisdiction/AK. This appears to be done.
- General Moodle Format - This is the XML format that we export.
- CSV - Described in the spec https://docs.moodle.org/dev/Outcomes_Administrator_Specification#CSV_File_Upload but does not appear to exist in the code. We can export the various outcome reports as csv but not the outcome set itself.
There is a general purpose XML import described in the spec which does not appear to exist in the code. https://docs.moodle.org/dev/Outcomes_Administrator_Specification#XML_File_Upload
Export Outcome Set
User can export as XML. This is the "General Moodle Format" which could possibly do with renaming. Do we want to support any other formats?
Administrator Mapping Outcomes to a Course
The user maps outcome sets to the course on the course settings page. There is a page at course administration > Outcomes which you use to access information about the existing mappings of outcomes within the course. Should the mapping of outcome sets to the course be done there too?
Administrator Deleting Outcomes (Soft Delete)
Deleted outcomes still exist in the database. However, I don't think this work is entirely complete. Deleting an individual outcome causes it to disappear from the UI everywhere including currently mapped activity settings and the Activity and performance page.
Administrator wants to import K-12 State Standards and Associate with the appropriate courses
https://docs.moodle.org/dev/Outcomes_Administrator_Use_Cases#Administrator_wants_to_import_K-12_State_Standards_and_Associate_with_the_appropriate_courses Is this the purpose of the subject and year drop downs when the user is mapping an outcome set to a course?
Administrator wants to include a new version of a K-12 State Standards Package
Functionality to allow updating of outcome sets does not appear to exist.
Administrator configures criteria for how a student is marked as having achieved competence on an outcome
Does not appear to exist. This may have been decided against. https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Achievement_Configuration
Administrator Generates Report to Show that Courses Cover All Required Outcomes
There is a coverage report for a single course. There is no report that allows the user to select an outcome set and see its coverage across courses. Links are shown in the spec but don't exist in the code.
https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Outcomes_Administration_Screen
https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Coverage_Report_.28Administrator.29
Administrator Generates Report to Show the Performance of Various Courses Against Outcomes
An Activity and performance report exists for a single course. There is no report that allows the user to select an outcome set and see student performance across courses. Links are shown in the spec but don't exist in the code.
https://docs.moodle.org/dev/Outcomes_Administrator_Specification#Outcomes_Administration_Screen
Teacher
https://docs.moodle.org/dev/Outcomes_Instructor_Use_Cases
https://docs.moodle.org/dev/Outcomes_Instructor_Specification
Instructor Browses Available Outcomes and Maps Course to Appropriate Outcomes
The user can select outcomes on both the activity setting page and on the rubric. The two sets of outcomes do not appear to be related.
The rubric could be limited to the outcomes in the activity settings however a rubric can be used on multiple assignments. Perhaps the outcomes section of the activity settings should be disabled as soon as "rubric" is selected.
Instructor Maps Quiz Questions Against Standards
The user can associate outcomes with either a quiz or individual questions. Should there be a relationship between the outcomes associated with the quiz and the questions? Should the outcomes for a quiz simply be the aggregate of the outcomes for the individual questions?
Instructor Uses Other Advanced Grading Methods With Outcomes
"The frameworks should make it possible for Advanced Grading should allow outcomes to flow through, if the developer for that advanced grading chooses to implement it."
Does this exist? Investigation required to determine if the advanced grading API has been implemented.
Instructor is Required to Show Student Performance Against Outcomes
This does not exist as described. Currently there is the "Activity and performance" report which allows the user to view performance for all outcomes by group. They can use the "Completion marking" page to view an individual student's performance against all outcomes. There does not seem to be anywhere where you can view the performance of all students against a single outcome.
K-12 Instructor in U.S. has to Provide Data of Student Performance Against State Standards
All of the outcomes pages allow exporting of the displayed table. This is presumably adequate but confirmation of this would be great.
Backup / Restore Common Cartridge with Outcome ID
Does this exist?
K-12 Instructor wants to use Moodle Sections as a Lesson Plan (Optional)
This does not currently exist. A page showing the used outcomes grouped by course section could be handy.
Completion marking
The completion marking page shown in the specific appears to have more columns than the current version in the code. Is there more work to be done here?
Outcomes Activity and Performance by Student
This closely resembles the completion marking page. Is there more work to do or does the completion marking page fill this role?
Recommendation Configuration
This does not appear to exist.
Student
https://docs.moodle.org/dev/Outcomes_Student_Use_Cases
https://docs.moodle.org/dev/Outcomes_Student_Specification
Student Begins Course, and Views All Outcomes Required for this Course
https://docs.moodle.org/dev/Outcomes_Student_Specification#Course_Outcomes_-_Student_View
This does not appear to exist.
Student Views Assignment Rubric and Associated Outcomes
The student can view the rubric which contains rows copied from the outcomes. There is no link to anywhere that might provide more information about the outcome. Should there be?
Student Successfully Masters Outcomes in a Course
Does not appear to exist. "Student Begins Course, and Views All Outcomes Required for this Course" will partially resolve this although additional work will be required if outcomes data is to be displayed on student profiles. https://docs.moodle.org/dev/Outcomes_Student_Specification#Profile_Page_-_Overview
Graduating Student Required to Provide Evidence of Skills
Does not appear to exist. This is another item that requires outcomes data to be displayed on student profiles.
Outcomes API to send Performance Data to Portfolio System
Does not appear to exist.
Open Questions
Badges
How does the outcomes system connect/relate to the badge system? If an instructor wanted achieving an outcome to result in a badge being awarded, how would they do that?
To Do
This is a summary of the tasks that come out of the above issues.
Pre-Coding Tasks
What about upgrades? We don't particularly want to run two outcome systems at the same time. Removing the old one would require us to have an upgrade path. There have been some notes made at https://docs.moodle.org/dev/Migration_and_Technical_Issues
Review and clarify export and import options.
Figure out how badges and outcomes should work together. Is this as simple as adding a new criteria for badges?
Figure out how rubric assignments should handle outcomes. https://docs.moodle.org/dev/2.7_Outcomes_rubric_outcomes
Figure out how quizzes/questions should handle outcomes. This should probably (but may not necessarily) work the same way as rubric assignments.
- Why do you consider this is a necessary pre-coding task? There was a great discussion when Moodlerooms were developing their spec, which fed into their spec. Then they implemented that. What do you think is wrong with what was done?--Tim Hunt (talk) 13:14, 19 November 2013 (WST)
- The issue is that both questions and quizzes can be mapped to outcomes but the two sets of mappings are independent. I can associate the quiz activity itself with a given set of outcomes while the quiz is composed entirely of questions that are mapped to a different set of outcomes. This is not necessarily wrong but we need to clarify whether the outcomes from the questions trump the outcomes from the quiz, whether the quiz should even be able to have it's own set of outcomes, whether we effectively union them all together and if there is some way to help the user understand this relationship.--Andrew Davis (talk) 15:19, 19 November 2013 (WST)
- Well, OK, I will try to explain briefly now. The point is that questions can be used in many different quizzes. And a question might be capable of assessing several different outcomes. In a particular quiz, some of those outcomes might be relevant, and others not. The quiz is designed to test certain things, and not others, so we indicate which outcomes that are declared by the questions are relevant to this quiz. For example, suppose we have questions Q1, testing outcome A, Q2 testing B & C, Q3 testing A & D, and Q4 testing B. We might assemble those into a quiz that does a reasonable job of measuring A & B. It does not give enough significant data to report meaningfully on C & D, so we don't want to track those outcomes for this quiz. However, Q2 & Q3 might later be reused in another quiz which is all about C & D.--Tim Hunt (talk) 17:11, 20 November 2013 (WST)
Missing or Incomplete Components
Soft deletion of outcomes
Updating of imported outcomes
Reports showing:
- Outcome coverage across courses
- Outcome performance across courses
- Performance of all students against a single outcome
- Outcomes used, grouped by course section
- All outcomes for a course (for student)
Backup / Restore Common Cartridge with Outcome ID
Recommendation configuration
Ability to display completed outcomes on profile
Ability to send performance data to an external portfolio