MoodleDocs - User contributions [en]

External tool

2014-05-16T19:54:33Z

Bushido: /* Addition to the add activities menu */

This page documents the External tool plugin in Moodle and its features.

== LTI Source Plugins ==

The External tool supports sub plugins that live under the ''mod/lti/source'' directory with the plugin type of ''ltisource''. The purpose of a LTI source plugin is to extend the functionality of the External Tool activity.

The LTI source plugins support the following standard plugin features:

* Language file: ''mod/lti/source/pluginname/lang/en/ltisource_pluginname.php''
* [[Upgrade_API| Plugin upgrade API]] under the ''mod/lti/source/pluginname/db'' directory.

=== Extending the LTI protocol ===

The most powerful extension that the LTI source plugins can perform is to extend the LTI services provided by the External Tool activity. All incoming requests are handled by ''mod/lti/service.php'' and all standard LTI requests are automatically handled. But, if the request is non-standard, then a LTI source plugin can have an opportunity to handle the request.

The request is routed to the LTI source plugin by using the message type. For example, let's say this request was sent to the External Tool:

<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<imsx_POXEnvelopeRequest xmlns="http://www.imsglobal.org/services/ltiv1p1/xsd/imsoms_v1p0">
<imsx_POXHeader>
<imsx_POXRequestHeaderInfo>
<imsx_version>V1.0</imsx_version>
<imsx_messageIdentifier>999998123</imsx_messageIdentifier>
</imsx_POXRequestHeaderInfo>
</imsx_POXHeader>
<imsx_POXBody>
<myCustomRequest>
<myRecord>
<sourcedGUID>
<sourcedId>3124567</sourcedId>
</sourcedGUID>

</myRecord>
</myCustomRequest>
</imsx_POXBody>
</imsx_POXEnvelopeRequest>
</code>

In the above request, the message type is '''myCustomRequest'''. Since this is not a standard request, a LTI source plugin can handle it. We can do this by adding a LTI source plugin and for this example, we will call it ''widget''. Then, in our ''ltisource_widget'' plugin, we must define a specific function In order to handle the request. The function name takes on the form of '''component'''_'''messageType'''. So, for our above request example we would create ''mod/lti/source/widget/lib.php'' and add this function to the file:

<code php>
/**
* Handle a custom widget request.
*
* @param object $data The LTI request details
*/
function ltisource_widget_myCustomRequest($data) {
$data->body; // The raw LTI request XML body.
$data->xml; // A SimpleXMLElement of the XML body.
$data->messageid; // The value of the <imsx_messageIdentifier> element in the request.
$data->messagetype; // The message type.
$data->consumerkey; // OAuth consumer key.
$data->sharedsecret; // The shared secret used to verify the request body.

// Do your custom work here.

// Throw exceptions on error, they will be sent back appropriately.

// When done, echo out your response XML.
$responsexml = lti_get_response_xml(
'success',
'Widget handled',
$data->messageid,
$data->messagetype
);

echo $responsexml->asXML();
}
</code>

'''Word of warning:''' If more than one LTI source plugins implement a function for handling the ''myCustomRequest'' message, then an exception will be thrown. So, when coming up with names for your message types, try to keep them unique and perhaps prefix them with something unique about your service.

And that's about it! With this extension point, you can build deep integrations with external services all while communicating through the LTI protocol.

=== Addition to the add activities menu ===

Overall, the following has a narrow use case, but it might be expanded in the future.

A LTI source plugin can add itself to the add activities menu by implementing a '''component'''_get_types function in its lib.php file. So, for example, if our LTI source plugin was ''widget'', then in ''mod/lti/source/widget/lib.php'' we would add the following function:

<code php>
function ltisource_widget_get_types() {
$types = array();
$types[] = (object) array(
'modclass' => MOD_CLASS_ACTIVITY,
'type' => 'lti&type=widget',
'typestr' => get_string('addwidget', 'ltisource_widget'),
'help' => get_string('addwidget_help', 'ltisource_widget'),
);
return $types;
}
</code>

But, this on its own doesn't actually do much at the moment. You must also add another callback that gets called from the External Tool add instance form. The callback takes on the form of '''component'''_add_instance_hook. So, also add this function to ''mod/lti/source/widget/lib.php'':

<code php>
function ltisource_widget_add_instance_hook() {
// Do custom work here.
}
</code>

At this point, the most common use case is to redirect to an external site to browse ''widget-like'' content. Then the '''Extending the LTI protocol''' section can be used to create ''widget-like'' content in Moodle.

External tool

2014-05-16T18:55:56Z

Bushido:

This page documents the External tool plugin in Moodle and its features.

== LTI Source Plugins ==

The External tool supports sub plugins that live under the ''mod/lti/source'' directory with the plugin type of ''ltisource''. The purpose of a LTI source plugin is to extend the functionality of the External Tool activity.

The LTI source plugins support the following standard plugin features:

* Language file: ''mod/lti/source/pluginname/lang/en/ltisource_pluginname.php''
* [[Upgrade_API| Plugin upgrade API]] under the ''mod/lti/source/pluginname/db'' directory.

=== Extending the LTI protocol ===

The most powerful extension that the LTI source plugins can perform is to extend the LTI services provided by the External Tool activity. All incoming requests are handled by ''mod/lti/service.php'' and all standard LTI requests are automatically handled. But, if the request is non-standard, then a LTI source plugin can have an opportunity to handle the request.

The request is routed to the LTI source plugin by using the message type. For example, let's say this request was sent to the External Tool:

<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<imsx_POXEnvelopeRequest xmlns="http://www.imsglobal.org/services/ltiv1p1/xsd/imsoms_v1p0">
<imsx_POXHeader>
<imsx_POXRequestHeaderInfo>
<imsx_version>V1.0</imsx_version>
<imsx_messageIdentifier>999998123</imsx_messageIdentifier>
</imsx_POXRequestHeaderInfo>
</imsx_POXHeader>
<imsx_POXBody>
<myCustomRequest>
<myRecord>
<sourcedGUID>
<sourcedId>3124567</sourcedId>
</sourcedGUID>

</myRecord>
</myCustomRequest>
</imsx_POXBody>
</imsx_POXEnvelopeRequest>
</code>

In the above request, the message type is '''myCustomRequest'''. Since this is not a standard request, a LTI source plugin can handle it. We can do this by adding a LTI source plugin and for this example, we will call it ''widget''. Then, in our ''ltisource_widget'' plugin, we must define a specific function In order to handle the request. The function name takes on the form of '''component'''_'''messageType'''. So, for our above request example we would create ''mod/lti/source/widget/lib.php'' and add this function to the file:

<code php>
/**
* Handle a custom widget request.
*
* @param object $data The LTI request details
*/
function ltisource_widget_myCustomRequest($data) {
$data->body; // The raw LTI request XML body.
$data->xml; // A SimpleXMLElement of the XML body.
$data->messageid; // The value of the <imsx_messageIdentifier> element in the request.
$data->messagetype; // The message type.
$data->consumerkey; // OAuth consumer key.
$data->sharedsecret; // The shared secret used to verify the request body.

// Do your custom work here.

// Throw exceptions on error, they will be sent back appropriately.

// When done, echo out your response XML.
$responsexml = lti_get_response_xml(
'success',
'Widget handled',
$data->messageid,
$data->messagetype
);

echo $responsexml->asXML();
}
</code>

'''Word of warning:''' If more than one LTI source plugins implement a function for handling the ''myCustomRequest'' message, then an exception will be thrown. So, when coming up with names for your message types, try to keep them unique and perhaps prefix them with something unique about your service.

And that's about it! With this extension point, you can build deep integrations with external services all while communicating through the LTI protocol.

=== Addition to the add activities menu ===

Overall, the following has a narrow use case, but it might be expanded in the future.

A LTI source plugin can add itself to the add activities menu by implementing a '''component'''_get_types function in its lib.php file. So, for example, if our LTI source plugin was ''widget'', then in ''mod/lti/source/widget/lib.php'' we would add the following function:

<code php>
function ltisource_widget_get_types() {
$types = array();
$types[] = (object) array(
'modclass' => MOD_CLASS_ACTIVITY,
'type' => 'lti&type=widget',
'typestr' => get_string('addwidget', 'ltisource_widget'),
'help' => get_string('addwidget_help', 'ltisource_widget'),
);
return $types;
}
</code>

But, this on its own doesn't actually do much at the moment. You must also add another callback that gets called from the External Tool add instance form. The callback takes on the form of '''component'''_add_instance_hook. So, also add this function to ''mod/lti/source/widget/lib.php'':

<code php>
function ltisource_widget_add_instance_hook() {
// Do custom work here.
}
</code>

At this point, the most common use case is to redirect to an external site to browse ''widget'' like content. Then the '''Extending the LTI protocol''' section can be used to create ''widget'' content in Moodle.

External tool

2014-05-16T18:52:58Z

Bushido: /* LTI Source Plugins */

This page documents the External tool plugin in Moodle and its features.

== LTI Source Plugins ==

The External tool supports sub plugins that live under the ''mod/lti/source'' directory with the plugin type of ''ltisource''. The purpose of a LTI source plugin is to extend the functionality of the External Tool activity.

The LTI source plugins support the following standard plugin features:

* Language file: ''mod/lti/source/pluginname/lang/en/ltisource_pluginname.php''
* [[Upgrade_API| Plugin upgrade API]] under the ''mod/lti/source/pluginname/db'' directory.

=== Extending the LTI protocol ===

The most powerful extension that the LTI source plugins can perform is to extend the LTI services provided by the External Tool activity. All incoming requests are handled by ''mod/lti/service.php'' and all standard LTI requests are automatically handled. But, if the request is non-standard, then a LTI source plugin can have an opportunity to handle the request.

The request is routed to the LTI source plugin by using the message type. For example, let's say this request was sent to the External Tool:

<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<imsx_POXEnvelopeRequest xmlns="http://www.imsglobal.org/services/ltiv1p1/xsd/imsoms_v1p0">
<imsx_POXHeader>
<imsx_POXRequestHeaderInfo>
<imsx_version>V1.0</imsx_version>
<imsx_messageIdentifier>999998123</imsx_messageIdentifier>
</imsx_POXRequestHeaderInfo>
</imsx_POXHeader>
<imsx_POXBody>
<widgetCustomRequest>
<widgetRecord>
<sourcedGUID>
<sourcedId>3124567</sourcedId>
</sourcedGUID>

</widgetRecord>
</widgetCustomRequest>
</imsx_POXBody>
</imsx_POXEnvelopeRequest>
</code>

In the above request, the message type is '''widgetCustomRequest'''. Since this is not a standard request, a LTI source plugin can handle it. We can do this by adding a LTI source plugin and for this example, we will call it ''widget''. Then, in our ''ltisource_widget'' plugin, we must define a specific function In order to handle the request. The function name takes on the form of '''component'''_'''messageType'''. So, for our above request example we would create ''mod/lti/source/widget/lib.php'' and add this function to the file:

<code php>
/**
* Handle a custom widget request.
*
* @param object $data The LTI request details
*/
function ltisource_widget_widgetCustomRequest($data) {
$data->body; // The raw LTI request XML body.
$data->xml; // A SimpleXMLElement of the XML body.
$data->messageid; // The value of the <imsx_messageIdentifier> element in the request.
$data->messagetype; // The message type.
$data->consumerkey; // OAuth consumer key.
$data->sharedsecret; // The shared secret used to verify the request body.

// Do your custom work here.

// Throw exceptions on error, they will be sent back appropriately.

// When done, echo out your response XML.
$responsexml = lti_get_response_xml(
'success',
'Widget handled',
$data->messageid,
$data->messagetype
);

echo $responsexml->asXML();
}
</code>

'''Word of warning:''' If more than one LTI source plugins implement a function for handling the ''widgetCustomRequest'' message, then an exception will be thrown. So, when coming up with names for your message types, try to keep them unique and perhaps prefix them with something unique about your service.

And that's about it! With this extension point, you can build deep integrations with external services all while communicating through the LTI protocol.

=== Addition to the add activities menu ===

Overall, the following has a narrow use case, but it might be expanded in the future.

A LTI source plugin can add itself to the add activities menu by implementing a '''component'''_get_types function in its lib.php file. So, for example, if our LTI source plugin was ''widget'', then in ''mod/lti/source/widget/lib.php'' we would add the following function:

<code php>
function ltisource_widget_get_types() {
$types = array();
$types[] = (object) array(
'modclass' => MOD_CLASS_ACTIVITY,
'type' => 'lti&type=widget',
'typestr' => get_string('addwidget', 'ltisource_widget'),
'help' => get_string('addwidget_help', 'ltisource_widget'),
);
return $types;
}
</code>

But, this on its own doesn't actually do much at the moment. You must also add another callback that gets called from the External Tool add instance form. The callback takes on the form of '''component'''_add_instance_hook. So, also add this function to ''mod/lti/source/widget/lib.php'':

<code php>
function ltisource_widget_add_instance_hook() {
// Do custom work here.
}
</code>

At this point, the most common use case is to redirect to an external site to browse ''widget'' like content. Then the '''Extending the LTI protocol''' section can be used to create ''widget'' content in Moodle.

External tool

2014-05-16T17:59:36Z

Bushido: /* Addition to the add activities menu */

This page documents the External tool plugin in Moodle and its features.

== LTI Source Plugins ==

The External tool supports sub plugins that live under the ''mod/lti/source'' directory with the plugin type of ''ltisource''. The purpose of a LTI source plugin is to extend the functionality of the External Tool activity. Those extensions are discussed below.

=== Extending the LTI protocol ===

The most powerful extension that the LTI source plugins can perform is to extend the LTI services provided by the External Tool activity. All incoming requests are handled by ''mod/lti/service.php'' and all standard LTI requests are automatically handled. But, if the request is non-standard, then a LTI source plugin can have an opportunity to handle the request.

The request is routed to the LTI source plugin by using the message type. For example, let's say this request was sent to the External Tool:

<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<imsx_POXEnvelopeRequest xmlns="http://www.imsglobal.org/services/ltiv1p1/xsd/imsoms_v1p0">
<imsx_POXHeader>
<imsx_POXRequestHeaderInfo>
<imsx_version>V1.0</imsx_version>
<imsx_messageIdentifier>999998123</imsx_messageIdentifier>
</imsx_POXRequestHeaderInfo>
</imsx_POXHeader>
<imsx_POXBody>
<widgetCustomRequest>
<widgetRecord>
<sourcedGUID>
<sourcedId>3124567</sourcedId>
</sourcedGUID>

</widgetRecord>
</widgetCustomRequest>
</imsx_POXBody>
</imsx_POXEnvelopeRequest>
</code>

In the above request, the message type is '''widgetCustomRequest'''. Since this is not a standard request, a LTI source plugin can handle it. We can do this by adding a LTI source plugin and for this example, we will call it ''widget''. Then, in our ''ltisource_widget'' plugin, we must define a specific function In order to handle the request. The function name takes on the form of '''component'''_'''messageType'''. So, for our above request example we would create ''mod/lti/source/widget/lib.php'' and add this function to the file:

<code php>
/**
* Handle a custom widget request.
*
* @param object $data The LTI request details
*/
function ltisource_widget_widgetCustomRequest($data) {
$data->body; // The raw LTI request XML body.
$data->xml; // A SimpleXMLElement of the XML body.
$data->messageid; // The value of the <imsx_messageIdentifier> element in the request.
$data->messagetype; // The message type.
$data->consumerkey; // OAuth consumer key.
$data->sharedsecret; // The shared secret used to verify the request body.

// Do your custom work here.

// Throw exceptions on error, they will be sent back appropriately.

// When done, echo out your response XML.
$responsexml = lti_get_response_xml(
'success',
'Widget handled',
$data->messageid,
$data->messagetype
);

echo $responsexml->asXML();
}
</code>

'''Word of warning:''' If more than one LTI source plugins implement a function for handling the ''widgetCustomRequest'' message, then an exception will be thrown. So, when coming up with names for your message types, try to keep them unique and perhaps prefix them with something unique about your service.

And that's about it! With this extension point, you can build deep integrations with external services all while communicating through the LTI protocol.

=== Addition to the add activities menu ===

Overall, the following has a narrow use case, but it might be expanded in the future.

A LTI source plugin can add itself to the add activities menu by implementing a '''component'''_get_types function in its lib.php file. So, for example, if our LTI source plugin was ''widget'', then in ''mod/lti/source/widget/lib.php'' we would add the following function:

<code php>
function ltisource_widget_get_types() {
$types = array();
$types[] = (object) array(
'modclass' => MOD_CLASS_ACTIVITY,
'type' => 'lti&type=widget',
'typestr' => get_string('addwidget', 'ltisource_widget'),
'help' => get_string('addwidget_help', 'ltisource_widget'),
);
return $types;
}
</code>

But, this on its own doesn't actually do much at the moment. You must also add another callback that gets called from the External Tool add instance form. The callback takes on the form of '''component'''_add_instance_hook. So, also add this function to ''mod/lti/source/widget/lib.php'':

<code php>
function ltisource_widget_add_instance_hook() {
// Do custom work here.
}
</code>

At this point, the most common use case is to redirect to an external site to browse ''widget'' like content. Then the '''Extending the LTI protocol''' section can be used to create ''widget'' content in Moodle.

External tool

2014-05-16T17:45:16Z

Bushido:

This page documents the External tool plugin in Moodle and its features.

== LTI Source Plugins ==

The External tool supports sub plugins that live under the ''mod/lti/source'' directory with the plugin type of ''ltisource''. The purpose of a LTI source plugin is to extend the functionality of the External Tool activity. Those extensions are discussed below.

=== Extending the LTI protocol ===

The most powerful extension that the LTI source plugins can perform is to extend the LTI services provided by the External Tool activity. All incoming requests are handled by ''mod/lti/service.php'' and all standard LTI requests are automatically handled. But, if the request is non-standard, then a LTI source plugin can have an opportunity to handle the request.

The request is routed to the LTI source plugin by using the message type. For example, let's say this request was sent to the External Tool:

<code xml>
<?xml version="1.0" encoding="UTF-8"?>
<imsx_POXEnvelopeRequest xmlns="http://www.imsglobal.org/services/ltiv1p1/xsd/imsoms_v1p0">
<imsx_POXHeader>
<imsx_POXRequestHeaderInfo>
<imsx_version>V1.0</imsx_version>
<imsx_messageIdentifier>999998123</imsx_messageIdentifier>
</imsx_POXRequestHeaderInfo>
</imsx_POXHeader>
<imsx_POXBody>
<widgetCustomRequest>
<widgetRecord>
<sourcedGUID>
<sourcedId>3124567</sourcedId>
</sourcedGUID>

</widgetRecord>
</widgetCustomRequest>
</imsx_POXBody>
</imsx_POXEnvelopeRequest>
</code>

In the above request, the message type is '''widgetCustomRequest'''. Since this is not a standard request, a LTI source plugin can handle it. We can do this by adding a LTI source plugin and for this example, we will call it ''widget''. Then, in our ''ltisource_widget'' plugin, we must define a specific function In order to handle the request. The function name takes on the form of '''component'''_'''messageType'''. So, for our above request example we would create ''mod/lti/source/widget/lib.php'' and add this function to the file:

<code php>
/**
* Handle a custom widget request.
*
* @param object $data The LTI request details
*/
function ltisource_widget_widgetCustomRequest($data) {
$data->body; // The raw LTI request XML body.
$data->xml; // A SimpleXMLElement of the XML body.
$data->messageid; // The value of the <imsx_messageIdentifier> element in the request.
$data->messagetype; // The message type.
$data->consumerkey; // OAuth consumer key.
$data->sharedsecret; // The shared secret used to verify the request body.

// Do your custom work here.

// Throw exceptions on error, they will be sent back appropriately.

// When done, echo out your response XML.
$responsexml = lti_get_response_xml(
'success',
'Widget handled',
$data->messageid,
$data->messagetype
);

echo $responsexml->asXML();
}
</code>

'''Word of warning:''' If more than one LTI source plugins implement a function for handling the ''widgetCustomRequest'' message, then an exception will be thrown. So, when coming up with names for your message types, try to keep them unique and perhaps prefix them with something unique about your service.

And that's about it! With this extension point, you can build deep integrations with external services all while communicating through the LTI protocol.

=== Addition to the add activities menu ===

Overall, the following has a narrow use case, but it might be expanded in the future.

A LTI source plugin can add itself to the [[Course_homepage#Activities_and_resources|add activities menu]]

External tool

2014-05-16T17:38:56Z

Bushido:

External tool

2014-05-16T17:23:29Z

Bushido: /* Extending the LTI protocol */

External tool

2014-05-16T17:16:21Z

Bushido: /* LTI Source Plugins */

External tool

2014-05-16T17:15:32Z

Bushido: /* LTI Source Plugins */

External tool

2014-05-16T17:12:46Z

Bushido: Created page with "This page documents the External tool plugin in Moodle and its features. == LTI Source Plugins == The External tool supports sub plugins that live under the ''mod/lti/sourc..."

Outcomes Technical Specification

2013-05-12T17:36:11Z

Bushido: /* Change History */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| action
| int(3)
| The action taken against the outcome_mark record: 1 is create, 2 is update and 3 is delete.
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Attempts ==

Addition information about outcome attempt handling.

* If Advanced Grading Method used then remove attempts if submission is removed
* If directly associated to activity then remove attempts if submission is removed and multiple graded submissions are allowed (i.e. Quiz, Lesson, Scorm) OR if grade removed from gradebook for other activities
* Whenever the all outcomes are removed from the object, it would remove the associated attempts.
* If a question is in use and is edited to now include outcomes, then outcome attempts are not generated unless the quiz is re-graded.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

* Removed outcome.name field and outcome.description field is now required --[[User:Mark Nielsen|Mark Nielsen]] 01:36, 13 May 2013 (WST)
* Added outcome_marks_history.action field to keep track of the action performed to generate the history --[[User:Mark Nielsen|Mark Nielsen]] 01:36, 13 May 2013 (WST)
* Added Architecture > Attempts section --[[User:Mark Nielsen|Mark Nielsen]] 08:01, 4 May 2013 (WST)
* Schema changes --[[User:Mark Nielsen|Mark Nielsen]] 07:57, 4 May 2013 (WST)
** Removed outcome_sets.userid
** Removed outcome_areas.deleted
** Added outcome_attempts.itemid
** Added outcome_attempts.timemodified

Outcomes Technical Specification

2013-05-12T17:35:00Z

Bushido: /* outcome_marks_history */ Added outcome_marks_history.action field

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| action
| int(3)
| The action taken against the outcome_mark record: 1 is create, 2 is update and 3 is delete.
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Attempts ==

Addition information about outcome attempt handling.

* If Advanced Grading Method used then remove attempts if submission is removed
* If directly associated to activity then remove attempts if submission is removed and multiple graded submissions are allowed (i.e. Quiz, Lesson, Scorm) OR if grade removed from gradebook for other activities
* Whenever the all outcomes are removed from the object, it would remove the associated attempts.
* If a question is in use and is edited to now include outcomes, then outcome attempts are not generated unless the quiz is re-graded.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

* Added Architecture > Attempts section --[[User:Mark Nielsen|Mark Nielsen]] 08:01, 4 May 2013 (WST)
* Schema changes --[[User:Mark Nielsen|Mark Nielsen]] 07:57, 4 May 2013 (WST)
** Removed outcome_sets.userid
** Removed outcome_areas.deleted
** Added outcome_attempts.itemid
** Added outcome_attempts.timemodified

Outcomes Technical Specification

2013-05-12T17:32:47Z

Bushido: /* outcome */ Removed outcome.name and outcome.description is now required

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Attempts ==

Addition information about outcome attempt handling.

* If Advanced Grading Method used then remove attempts if submission is removed
* If directly associated to activity then remove attempts if submission is removed and multiple graded submissions are allowed (i.e. Quiz, Lesson, Scorm) OR if grade removed from gradebook for other activities
* Whenever the all outcomes are removed from the object, it would remove the associated attempts.
* If a question is in use and is edited to now include outcomes, then outcome attempts are not generated unless the quiz is re-graded.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

* Added Architecture > Attempts section --[[User:Mark Nielsen|Mark Nielsen]] 08:01, 4 May 2013 (WST)
* Schema changes --[[User:Mark Nielsen|Mark Nielsen]] 07:57, 4 May 2013 (WST)
** Removed outcome_sets.userid
** Removed outcome_areas.deleted
** Added outcome_attempts.itemid
** Added outcome_attempts.timemodified

File:Outcome ERD.jpg

2013-05-12T17:31:15Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": Added outcome_marks_history.action field Removed outcome.name field

Outcomes ERD

Outcomes Technical Specification

2013-05-04T00:01:09Z

Bushido: /* Change History */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Attempts ==

Addition information about outcome attempt handling.

* If Advanced Grading Method used then remove attempts if submission is removed
* If directly associated to activity then remove attempts if submission is removed and multiple graded submissions are allowed (i.e. Quiz, Lesson, Scorm) OR if grade removed from gradebook for other activities
* Whenever the all outcomes are removed from the object, it would remove the associated attempts.
* If a question is in use and is edited to now include outcomes, then outcome attempts are not generated unless the quiz is re-graded.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

* Added Architecture > Attempts section --[[User:Mark Nielsen|Mark Nielsen]] 08:01, 4 May 2013 (WST)
* Schema changes --[[User:Mark Nielsen|Mark Nielsen]] 07:57, 4 May 2013 (WST)
** Removed outcome_sets.userid
** Removed outcome_areas.deleted
** Added outcome_attempts.itemid
** Added outcome_attempts.timemodified

Outcomes Technical Specification

2013-05-04T00:00:23Z

Bushido: /* Architecture */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Attempts ==

Addition information about outcome attempt handling.

* If Advanced Grading Method used then remove attempts if submission is removed
* If directly associated to activity then remove attempts if submission is removed and multiple graded submissions are allowed (i.e. Quiz, Lesson, Scorm) OR if grade removed from gradebook for other activities
* Whenever the all outcomes are removed from the object, it would remove the associated attempts.
* If a question is in use and is edited to now include outcomes, then outcome attempts are not generated unless the quiz is re-graded.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

* Schema changes --[[User:Mark Nielsen|Mark Nielsen]] 07:57, 4 May 2013 (WST)
** Removed outcome_sets.userid
** Removed outcome_areas.deleted
** Added outcome_attempts.itemid
** Added outcome_attempts.timemodified

Outcomes Technical Specification

2013-05-03T23:57:19Z

Bushido: /* Change History */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

* Schema changes --[[User:Mark Nielsen|Mark Nielsen]] 07:57, 4 May 2013 (WST)
** Removed outcome_sets.userid
** Removed outcome_areas.deleted
** Added outcome_attempts.itemid
** Added outcome_attempts.timemodified

Outcomes Technical Specification

2013-05-03T23:55:32Z

Bushido: /* outcome_attempts */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| itemid
| int(10), indexed
| Used to identify the attempt for updates. For example, might be a quiz attempt record ID.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

Outcomes Technical Specification

2013-05-03T23:53:43Z

Bushido:

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

= Change History =

File:Outcome ERD.jpg

2013-05-03T23:52:34Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": Sigh...

Outcomes ERD

File:Outcome ERD.jpg

2013-05-03T23:42:56Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": Smaller image (hopefully)

Outcomes ERD

File:Outcome ERD.jpg

2013-05-03T23:36:30Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": Removed outcome_sets.userid Removed outcome_areas.deleted Added outcome_attempts.itemid Added outcome_attempts.timemodified

Outcomes ERD

Outcomes Technical Specification

2013-04-03T23:43:35Z

Bushido: /* outcome_area_outcomes */ Removed deleted field, doesn't make sense - just re-link the outcome and there will be no data loss

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

File:Outcome ERD.jpg

2013-04-03T23:38:46Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": Reverted to version as of 22:28, 8 March 2013

Outcomes ERD

Outcomes Technical Specification

2013-04-03T23:38:23Z

Bushido: /* outcome_sets */ Undo personal outcome sets

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-04-03T23:35:47Z

Bushido: /* Role capabilities */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapoutcomes
|-
| Map outcome sets to courses
| moodle/outcome:mapoutcomesets
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-19T17:14:13Z

Bushido: /* Role capabilities */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapcontent
|-
| Map outcome sets to courses
| moodle/outcome:mapcourse
|-
| Edit outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-19T17:13:54Z

Bushido: /* Role capabilities */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapcontent
|-
| Map outcome sets to courses
| moodle/outcome:mapcourse
|-
| Editing of outcomes
| moodle/outcome:edit
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Specification Change Log

2013-03-19T17:13:21Z

Bushido:

{| class="nicetable"
|-
! Issue
! Status
! Resolution/Description
! Date
|-
| Exporting Outcomes
| Closed
| Added use case. Updated Administrator spec to have export link.
| 2/16/13
|-
| Levels of Outcomes
| Closed
| Outcome Sets will be flat, Outcomes will be nested. Each Outcome can be configured as “assessable”, which determines whether the Outcome can be assigned to content and marked as achieved.
| 2/23/13
|-
| Should Outcome Text Allow HTML
| Closed
| Answer: No (They get displayed in too many places where HTML might not be good)
| 2/15/13
|-
| Restrict Outcomes by Category or Cohort
| Closed
| Answer: No.
| 2/15/13
|-
| Should outcome marking allow feedback?
| Closed
| Answer: No. (Potentially in future versions, but we have too many commenting places now as it is).
| 2/15/13
|-
| Should we allow tagging of outcomes/outcome sets for reporting?
| Closed
| Answer: No. (It would be nice, but we are cutting for scope. We should not design to where this could not be added in the future)
| 2/15/13
|-
| Review Academic Benchmarks XML
| Closed
| Sample format file received.
| 2/15/13
|-
| Migration of outcomes to a single outcome set.
| Closed
| Instead of migrating old outcomes to a single "Upgraded Outcomes" set, we migrate all site wide outcomes to "Upgraded Site Outcomes" and all course specific outcomes are migrated to a "Upgraded SHORTNAME Outcomes" where SHORTNAME is course shortname.
| 2/20/13
|-
| Add Use Case for Instructor adding Course Outcomes
| Closed
| Added a use case to the Instructor Use Cases section
| 2/24/13
|-
| Determine if we want to display outcomes to students ala feedback on quizzes.
| Open
|
|
|-
| Change the reports to be based off of “Completion” rather than “Resource Views”
| Closed
| Updated all reporting interfaces to be based on completion, rather than resource views.
| 2/24/13
|-
| Remove Outcome Set ShortName field as it is duplicative
| Closed
| Removed
| 3/14/13
|-
| On Outcome Creation, add Educational Level and Subject
| Closed
| Added
| 3/14/13
|-
| Collapse Outcome Sets, Outcomes, and Folders and allow them to be mapped, add a flag called assessable.
| Closed
| Collapsed these down and handled the multi level outcomes
| 3/14/13
|-
| Rename Outcome ID to Id Number, user editable, maps to Doc Num in AB and to Statement Notation in ASN
| Closed
| An alphanumeric notation or ID code as defined by the promulgating body to identify the statement.
|
|-
| Redo select Outcome Sets to make it flat, with optional filters for educational level and subject
| Open
|
|
|-
| Does Outcomes live in the Settings block? Gradebook or on their own?
| Open
|
|
|-
| How do we handle scale based items?
| Closed
| Added Scale Based Items to the Performance reports and a call out for scale based items.
| 3/14/13
|-
| Update Reports to show questions and rubric items under an assignment and quiz
| Closed
| Added
| 3/14
|-
| Update roles and capabilities: remove course creator from all capabilities except for mapping outcome sets to courses. Update capabilities to be editing teacher instead of teacher. Introduce new capabilities for import/export.
| Closed
| Added/Updated
| 3/19
|}

Outcomes Capabilities and Roles

2013-03-19T17:08:38Z

Bushido:

The following will be capabilities and the roles which should have these capabilities selected by default:

{| class="nicetable"
|-
! Capability
! Description
! Roles with Capability Enabled by Default
|-
| Outcomes: Administration - Manage Outcomes
| Controls the ability to Add, Delete, Upload and Edit Outcomes from the Administration Screen
| System Administrator
|-
| Outcomes: Administration - Outcomes Reporting
| Controls the ability to view the Outcomes Reports in the Outcomes Administration Area
| System Administrator
|-
| Outcomes: Administration - Achievement Configuration
| Ability to configure how students are marked as having achieved an outcome and the recommendation engines.
| System Administrator
|-
| Outcomes: Course Mapping
| Controls the ability to map a course against an outcome set
| System Administrator, Manager, Course Creator, Editing Teacher
|-
| Outcomes: Activity and Resource Mapping
| Controls the ability to map an activity, resource, or item against an outcome
| System Administrator, Manager, Editing Teacher
|-
| Outcomes: Course Reporting
| Controls the ability to view the coverage and performance reports for mapped outcomes.
| System Administrator, Manager, Editing Teacher
|-
| Outcomes: Add Course Outcomes
| Grants the ability to add outcomes at the course level that will be tied directly to that course/user.
| System Administrator, Manager, Editing Teacher

|}

== See Also ==
* [[Outcomes Specification]]
* [[Outcomes Administrator Specification]]

Outcomes Technical Specification

2013-03-19T16:45:01Z

Bushido: /* Role capabilities */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapcontent
|-
| Map outcome sets to courses
| moodle/outcome:mapcourse
|-
| Administering outcomes (Editing site outcomes, etc)
| moodle/outcome:admin
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-19T16:35:20Z

Bushido: /* Role capabilities */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:mapcontent
|-
| Map outcome sets to courses
| moodle/outcome:mapcourse
|-
| Administering outcomes (Editing site outcomes, etc)
| moodle/outcome:admin
|-
| Manage personal outcomes
| moodle/outcome:my
|-
| Export outcomes
| moodle/outcome:export
|-
| Import outcomes
| moodle/outcome:import
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-18T21:21:02Z

Bushido: /* outcome_areas */ Reduced field size so an unique index on component, area and itemid could be created

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(100), indexed
| The name of the component.
|-
| area
| varchar(50), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-18T18:26:43Z

Bushido: /* Data structure */ Moving outcome_areas.deleted to outcome_area_outcomes

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-18T16:47:17Z

Bushido: /* Backwards compatibility breaks */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

These changes will be documented in the relevant change log files.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-18T16:38:58Z

Bushido: /* Data structure */ Fixing default on foreign key fields

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default null
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets, outcome

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed, default null
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

File:Outcome ERD.jpg

2013-03-13T17:33:44Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": Attempting to improve quality

Outcomes ERD

File:Outcome ERD.jpg

2013-03-13T17:31:53Z

Bushido: uploaded a new version of "File:Outcome ERD.jpg": outcome_sets can now optionally link to a user record.

Outcomes ERD

Outcomes Technical Specification

2013-03-13T17:17:51Z

Bushido: /* outcome_sets */ Personal outcome sets

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: user

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve]. Instructors can create personal outcome sets that can be used in their courses.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| userid
| int(10), indexed, default 0
| Foreign key to user table. If set to a user's ID, then that user owns the outcome set and no one else can use or edit it besides admins. This allows for instructors to create their own outcome sets for their courses.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-08T22:29:58Z

Bushido: /* Data structure */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[File:Outcome_ERD.jpg|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

File:Outcome ERD.jpg

2013-03-08T22:28:35Z

Bushido: Outcomes ERD

Outcomes ERD

Outcomes Technical Specification

2013-03-08T22:26:47Z

Bushido: /* Data structure */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

This diagram provides an overview of how the tables relate to each other and to core tables. Please see the below sections for details about each table.

[[Image:Outcome_ERD.png|frame|center|Outcomes ERD]]

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-07T21:03:40Z

Bushido: /* Course */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate multiple outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-07T20:59:58Z

Bushido: Renaming tables

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_sets ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_sets

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_metadata ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_areas ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_area_outcomes table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_area_outcomes ==

Relations: outcome, outcome_areas

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomeareaid
| int(10), unique index
| Foreign key to outcome_areas table.
|}

== outcome_used_areas ==

Relations: outcome_areas, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomeareaid
| int(10), indexed
| Foreign key to outcome_areas table.
|}

== outcome_used_sets ==

Relations: outcome_sets, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_sets table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempts ==

Relations: outcome_used_areas, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeusedareaid
| int(10), indexed
| Foreign key to outcome_used_areas table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_marks_history ==

Relations: course, user, outcome, outcome_marks

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_marks table. Though the record could be missing from outcome_marks.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_awards ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-06T22:11:02Z

Bushido: /* outcome_attempt */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_set ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_set

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_meta ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_content ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_content_map table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_content_map ==

Relations: outcome, outcome_content

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomecontentid
| int(10), unique index
| Foreign key to outcome_content table.
|}

== outcome_mod_content_map ==

Relations: outcome_content, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|}

== outcome_set_course_map ==

Relations: outcome_set, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempt ==

Relations: outcome_mod_content_map, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemodcontentid
| int(10), indexed
| Foreign key to outcome_mod_content_map table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark_history ==

Relations: course, user, outcome, outcome_mark

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_mark table. Though the record could be missing from outcome_mark.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_award ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-06T22:08:17Z

Bushido: /* outcome_attempt */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_set ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_set

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_meta ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_content ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_content_map table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_content_map ==

Relations: outcome, outcome_content

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomecontentid
| int(10), unique index
| Foreign key to outcome_content table.
|}

== outcome_mod_content_map ==

Relations: outcome_content, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|}

== outcome_set_course_map ==

Relations: outcome_set, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempt ==

Relations: outcome_mod_content_map, user

Purpose: Track a user attempt against a content item within a particular activity. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemodcontentid
| int(10), indexed
| Foreign key to outcome_content table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark_history ==

Relations: course, user, outcome, outcome_mark

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_mark table. Though the record could be missing from outcome_mark.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_award ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-06T22:06:40Z

Bushido:

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_set ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_set

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_meta ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_content ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_content_map table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_content_map ==

Relations: outcome, outcome_content

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomecontentid
| int(10), unique index
| Foreign key to outcome_content table.
|}

== outcome_mod_content_map ==

Relations: outcome_content, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|}

== outcome_set_course_map ==

Relations: outcome_set, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempt ==

Relations: outcome_content, user

Purpose: Track a user attempt against a content item. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark_history ==

Relations: course, user, outcome, outcome_mark

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_mark table. Though the record could be missing from outcome_mark.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_award ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The outcome API, outcome administrative settings, outcome associations and other UIs that drives outcomes will be located inside of the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under outcome/import/ and outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-06T22:03:00Z

Bushido: /* Questions */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() before the Tags section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_set ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_set

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_meta ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_content ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_content_map table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_content_map ==

Relations: outcome, outcome_content

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomecontentid
| int(10), unique index
| Foreign key to outcome_content table.
|}

== outcome_mod_content_map ==

Relations: outcome_content, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|}

== outcome_set_course_map ==

Relations: outcome_set, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempt ==

Relations: outcome_content, user

Purpose: Track a user attempt against a content item. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark_history ==

Relations: course, user, outcome, outcome_mark

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_mark table. Though the record could be missing from outcome_mark.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_award ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The API that drives outcomes could be located inside of lib/outcome/ directory. Any UI like the administrative settings, outcome associations, etc could be located inside of a top level outcome/ directory. Alternatively, the administrative UI could live under the admin/outcome/ directory while all non-administrative UIs could live under the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under lib/outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under lib/outcome/import/ and lib/outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under lib/outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-03-06T22:02:28Z

Bushido: /* Activity */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically. In the flow of the activity's form, the outcome mapping options would appear below the grades section. Do note though that if an activity does not support grades, it can still be mapped to outcomes.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() after the General section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_set ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_set

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_meta ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_content ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_content_map table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_content_map ==

Relations: outcome, outcome_content

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomecontentid
| int(10), unique index
| Foreign key to outcome_content table.
|}

== outcome_mod_content_map ==

Relations: outcome_content, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|}

== outcome_set_course_map ==

Relations: outcome_set, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempt ==

Relations: outcome_content, user

Purpose: Track a user attempt against a content item. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark_history ==

Relations: course, user, outcome, outcome_mark

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_mark table. Though the record could be missing from outcome_mark.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_award ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The API that drives outcomes could be located inside of lib/outcome/ directory. Any UI like the administrative settings, outcome associations, etc could be located inside of a top level outcome/ directory. Alternatively, the administrative UI could live under the admin/outcome/ directory while all non-administrative UIs could live under the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under lib/outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under lib/outcome/import/ and lib/outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under lib/outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.

Outcomes Technical Specification

2013-02-25T23:45:55Z

Bushido: /* outcome_attempt */

= Introduction =

Please be familiar with the function specification as outlined in the [[Outcomes Specification]]. The functional specification will provide details about the concepts and goals of this project. In particular, the following sections provide the overview of the system by user role:

* [[Outcomes Administrator Specification]]
* [[Outcomes Instructor Specification]]
* [[Outcomes Student Specification]]

In addition, some features and use cases are known but not yet being implemented. Please see [[Specifically Excluded Use Cases]]. Specifically, this initial version will not be attempting to map an existing outcome set to a new version of the outcome set on import.

This document attempts to capture how the functional specification would be implemented in Moodle.

= Backwards compatibility breaks =

First and foremost, it is important to identify key differences between the current outcome system and this proposal as there are some philosophical shifts as well as breaking of existing code.

# Outcomes will now only be created at the site level and then mapped to courses and content. Outcomes will no longer be created at the course level.
# Anything using the current outcome API or tables will have to be updated to use the new outcomes when appropriate.
# Outcomes will no longer appear in the grade book.
# Outcomes will no longer support scales. It is just a simple pass or fail.

= Settings =

== Site ==

At the site level, there will be an area to do all of the configuration and setup of outcome sets. The proposed location is under Settings > Site Administration > Grades > Outcomes.

In addition to managing outcomes, there will be site settings page for general outcome settings. For example, a setting to toggle if outcome idnumbers are editable or not.

== Course ==

When editing a course, there will be a section to associate outcome sets to the course. The section will appear below the course General settings.

This association is used to determine the relevant outcomes for the course for reporting and as a way to pre-filter outcome pickers when associating outcomes to content.

== Activity ==

When editing an activity, there will be a section to associate individual outcomes to the activity. This section will be provided by the moodleform_mod::standard_coursemodule_elements() callback so that all activities will be able to be associated to outcomes automatically.

Graded activities will have to add additional integration points with outcomes in order to capture user attempts.

== Advanced grading ==

Rubrics and Marking guides will be able to associate an outcome for a criteria. API will be provided to support this, but each grading method will have to integrate with outcomes where it makes sense.

== Questions ==

When editing a question, there will be a section to associate individual outcomes to the question. The section can be added to question_edit_form::definition() after the General section.

This association UI might behave a little differently based on the context in Moodle. If the question is being authored at the site level, then the outcome association is more free form, meaning one can select any outcome from any outcome set. If the question is being authored at the course level, then the outcome association defaults to those outcome sets that are associated to the course.

= Data structure =

== outcome_set ==

Relations: none

Purpose: An outcome set represents a collection of outcomes and perhaps some meta data regarding the revision or applicable region. Outcome sets are mapped to courses, which makes the outcomes within the set available to be mapped to course content. An example outcome set would be [http://asn.jesandco.org/resources/D2403043 Health Education Content Standards for California Public Schools, Kindergarten Through Grade Twelve].

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome set
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| provider
| varchar(255), default null, optional
| Provider information regarding the outcome set. EG: Could be set to ASN.
|-
| revision
| varchar(255), default null, optional
| Version information regarding the outcome set. This will typically be a date or year, EG: 2011
|-
| region
| varchar(255), default null, optional
| Any region information. Example: In the US this could be a state name.
|-
| deleted
| int(1), default 0
| If the outcome set has been deleted.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome ==

Relations: outcome_set

Purpose: Stores information regarding an individual outcome. Outcomes can be stored in a hierarchy, but the hierarchy is visual only. This means that if an outcome is earned, then only that outcome is earned, not it and all of its children. Also, if all child outcomes are earned, then parent outcomes are not also automatically earned. In addition, sometimes outcomes are not assessable, but are only there for extra context in the hierarchy. This identifies that the outcome cannot be mapped to content, nor can a student earn it.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| parentid
| int(10), indexed
| Self referencing key. Identifies the parent outcome in the hierarchy.
|-
| idnumber
| varchar(255), unique index
| GUID that is auto-generated if missing. Generally on import, these are provided. This would be used to identify the outcome set during import or course restore.
|-
| name
| varchar(255)
| The name of the outcome.
|-
| docnum
| varchar(255), default null, optional
| Field for tracking additional information on the outcome. EG: 2.3.A
|-
| description
| text, default null, optional
| The plain text description of the outcome.
|-
| assessable
| int(1)
| If set to 1, then the outcome is assessable and can be mapped to content. If set to 0, then the outcome is not assessable and cannot be mapped to content.
|-
| deleted
| int(1), default 0
| If the outcome has been deleted.
|-
| sortorder
| int(10)
| The sort order. This is 0-X for all outcomes that belong to the same outcome set.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_meta ==

Relations: outcome

Purpose: Represents additional meta information regarding outcomes. Only support for “Education Level” (K, 1, … 12, etc.) and “Subjects” (e.g. Math, English, etc.) will be available out of the box, but additional data points can be added later (which would need to be added to the UI). One outcome could have many meta values. For example: an outcome could be associated to many educational levels and subjects.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| name
| varchar(255), indexed
| The name of the meta data.
|-
| value
| varchar(1333), indexed
| The value of the meta data.
|}

== outcome_content ==

Relations: dynamic with core tables

Purpose: This represents that a content item has been mapped to an outcome or to multiple outcomes. Check the outcome_content_map table for the specific outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| component
| varchar(255), indexed
| The name of the component.
|-
| area
| varchar(255), default null, indexed, optional
| The name of the area. Used when a component has multiple items to associate to outcomes.
|-
| itemid
| int(10), indexed
| The item ID.
|-
| deleted
| int(1), default 0
| Soft delete if the outcome was disassociated from the content. Note, if the content is actually deleted, then this association will be deleted.
|}

== outcome_content_map ==

Relations: outcome, outcome_content

Purpose: This represents the one to many relationship of a single content item mapped to many outcomes.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), unique index
| Foreign key to outcome table.
|-
| outcomecontentid
| int(10), unique index
| Foreign key to outcome_content table.
|}

== outcome_mod_content_map ==

Relations: outcome_content, course_modules

Purpose: Associates an outcome content item with an activity. For example: a question is created in the site question bank and outcomes are associated to the question. Once that question is added to a Quiz then that association is stored in this table so the outcome system knows that the content item and its outcomes are in use.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| cmid
| int(10), indexed
| Foreign key to course_modules table.
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|}

== outcome_set_course_map ==

Relations: outcome_set, course

Purpose: Associates an outcome set with the course. Filtering of the outcome set is optionally allowed since outcome sets can span multiple educational levels and even subjects. Once an outcome set is mapped to the course, then the outcomes in the outcome set are available to be mapped to course content.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomesetid
| int(10), indexed
| Foreign key to outcome_set table.
|-
| filter
| text, default null
| Some outcome sets span multiple education levels and subjects. So, when mapping an outcome set to a course, it may make sense to restrict the available outcomes to a specific education level(s) and/or subject(s). This information would be serialized into this field.
|}

== outcome_attempt ==

Relations: outcome_content, user

Purpose: Track a user attempt against a content item. The idea here is to aggregate attempt information so that the recommendation engines (detailed below) have an easily accessible data set to process. This does not mean however that recommendation engines are only limited to this data as one could mine for additional data where needed.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomecontentid
| int(10), indexed
| Foreign key to outcome_content table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| percentgrade
| float(10,5)
| Percent grade earned on this attempt.
|-
| mingrade
| float(10,5)
| The minimum possible grade.
|-
| maxgrade
| float(10,5)
| The maximum possible grade.
|-
| rawgrade
| float(10,5)
| The points earned, should be between the minimum and maximum grades.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark ==

Relations: course, user, outcome

Purpose: Graders will manually mark the student as having Met or Not Met an outcome. The recommendation engines will make a recommendation to the grader based on user attempts against content that has been mapped to the outcome.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timemodified
| int(10)
| Time modified.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_mark_history ==

Relations: course, user, outcome, outcome_mark

Purpose: Keep track of past markings.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomemarkid
| int(10), indexed
| Foreign key to outcome_mark table. Though the record could be missing from outcome_mark.
|-
| courseid
| int(10), indexed
| Foreign key to course table.
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table. This is the user who is receiving the mark.
|-
| graderid
| int(10), indexed
| Foreign key to user table. This is the user is doing the marking.
|-
| result
| int(2)
| 0 is not pass, 1 is pass, null is that no decision has been made yet.
|-
| timecreated
| int(10)
| Time created.
|}

== outcome_award ==

Relations: outcome, user

Purpose: This preserves any outcomes earned by a student, regardless if the course is deleted or not.

{| class="nicetable"
|-
! Field
! Type
! Description
|-
| id
| int(10), unique index
| Primary key
|-
| outcomeid
| int(10), indexed
| Foreign key to outcome table.
|-
| userid
| int(10), indexed
| Foreign key to user table.
|-
| timecreated
| int(10)
| Time created.
|}

= Architecture =

The API that drives outcomes could be located inside of lib/outcome/ directory. Any UI like the administrative settings, outcome associations, etc could be located inside of a top level outcome/ directory. Alternatively, the administrative UI could live under the admin/outcome/ directory while all non-administrative UIs could live under the top level outcome/ directory.

'''Models and repository mappers:'''

* outcome: Represents a single outcome
* outcome set: Represents a group of outcomes
* outcome content: An outcome association to content.
* outcome attempt: An attempt made against an outcome.
* outcome mark: A teacher marking an outcome against a user.
* outcome award: An awarded outcome at the site level.

A repository mapper would include ways to store the model and ways to retrieve them in a re-usable and organized manner.

'''Collections:'''

A group of models that you may take action on or filter. Example of a third party library that helps to manage collections: https://github.com/doctrine/collections

Collections may or may not be used, but some possible collections would be the following:

* outcomes
* outcome attempts

'''New mform elements:'''

For most plugins, the process of associating an outcome would be done through a Moodle form. Proposing adding a new Moodle form element that can be added to a form to provide the necessary UI. An API for validating and saving the new mform element will be provided. In addition, the mform element will just be a wrapper around a well formed API that could be used on it’s own when other use cases present themselves.

'''API for plugins'''

Beyond the mform element, plugins may interact with outcomes in order to provide a richer integration. An API will be provided to assist with the following (not an exhaustive list):

* A way to notify the outcome system that a content item is or is not being used within a course.
** Example: When a question is authored at the site level and then is actually used within a course, it must inform the outcome system that it is in use. This could potentially be done via events or just direct API calls at the appropriate times. Same goes for when a question is no longer used within a course.
* Record an attempt made against a content item.
** Examples: A question has been attempted or an assignment submission has been graded.
* A way to retrieve associated outcomes.
* A way to update, delete or otherwise modify the associated outcomes.
* A way to pick outcome associations (EG: what the mform element would do).
* A method of syncing up attempts against a content item. This would be used when associating an outcome on content that has already been graded/attempted.
* API to cleanup tables on course deletion.

== Supported plugin type plugins ==

This may sound a little odd, but basically the point of these plugins is to add support for a plugin type to the outcome system. These types would assist in interpreting outcome content associations. Examples:

* Given an outcome association, please provide a link to the mapped content.
** For an activity, this would be a link to the module.
** For a grading form, a link to the grading instance.
** For a question, a link to the question instance in the question bank.
* What are all of the content items that have not been mapped?
** For activities, this would return instances that are not mapped.
** For grading form, this would return sub-plugin specific information, like criteria.
** For questions, this would return questions from the course question bank that are not mapped.
* May assist with automatic cleanup when the content is deleted.
* Also other meta data could be derived, like plugin type name.

This plugin would live under lib/outcome/support/ and would initially ship with support of the following plugin types:

* mod
* qtype
* gradingform: note that this integration would be deeper in that each gradingform plugin would need to implement something in order to help interpret associations. The reason being is that outcomes are being associated to plugin specific content, instead of something generic like the grading instance.

== Import and export plugins ==

The outcome system would support the exporting of outcome sets. The first export plugin would be a very simple JSON export that is specific to Moodle itself. This would be used for moving custom outcomes from one Moodle site to another.

The outcome system would support importing outcome sets. Initially, there would be support for importing outcomes from [http://asn.jesandco.org Achievement Standards Network (ASN)] ([http://standards.jesandco.org/wiki/ASN_Application_Profile Format Specification]) and importing the JSON export from above.

Both of the outcomes import and export processes would be a plugin based, so other formats could be supported by creating new plugins. The import and export plugins would live under lib/outcome/import/ and lib/outcome/export/.

== Recommendation engine plugins ==

The outcome system would provide a way to recommend to a teacher whether a student should be marked as completing the outcome or not. These plugins should have full Moodle plugin support because they may need to listen to events, run crons, have DB tables, global settings, etc. These plugins would live under lib/outcome/engine/. Recommendations would be generated on demand and stored in MUC.

Initially, the only plugin would be a percentage based recommendation engine. It would be configurable for a specific threshold that would determine if an attempt against an outcome should be considered correct or incorrect. It would then process the outcome attempts to determine the recommendation.

Recommendation engines should also be enabled or disabled at the site level. This would require a “Manage recommendation engines” section in the admin site settings.

= Reporting =

== Site ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped to content in the site.
* Outcomes Performance report - shows performance on outcomes at the site level. Optionally broken down by course.

== Course ==

The following reports:

* Outcomes Coverage report - shows outcomes that have been mapped in this course.
* Outcomes: Unmapped Content report - shows content that is not mapped to an outcome.
* Outcomes Performance report - shows aggregated performance on outcomes in the course.
* Outcomes Student Performance report - shows student performance on outcomes in the course for the teacher.
* Outcomes Personal Performance report - shows student performance on outcomes in the course for the student.

== User profile ==

At the very least, there will be a user site report where the user can see earned outcomes.

= Grading =

Outcome Completion Marking is done by course graders and is the final check point for if a learner has earned an outcome or not. The proposed location is to use the Outcome tab that is currently in the grades section of the course. This would also highjack the Outcomes link under Grade administration and Course administration.

= Logging =

We want to audit most actions in the Moodle log. These actions include:

* CRUD type actions.
* Any association editing. EG: associating an outcome with a course, content item, etc.
* Outcome marking.
* User attempts against outcomes.

= Role capabilities =

{| class="nicetable"
|-
! Description
! Capability
|-
| Map outcomes to content
| moodle/outcome:contentmap
|-
| Map outcome sets to courses
| moodle/outcome:coursemap
|-
| Administering outcomes (Editing, importing, etc)
| moodle/outcome:edit
|-
| Each report will have a capability to view them.
| report/REPORTNAME:view
|-
| Marking outcomes by graders
| moodle/grade:edit
|}

= Backup and restore =

The workflow for backup and restore may go like:

* Same site, then just map to existing outcomes via the identifier.
* Not same site, then the assumption is that the outcome set should be imported first, then we can map to existing outcomes via the identifier like in our same site scenario.

Outcomes would probably take advantage of the backup plugin API to backup outcome associations to modules, question, etc. So, for example:

* Outcomes would plug into activities in order to backup outcome associations and attempts.
* Outcomes would plug into questions in order to backup outcome associations and attempts.
* Grading forms would have to manually outcome associations and attempts. Will strive to make code re-usable.

= Migration of old outcomes =

Please first review [[Migration and Technical Issues]]. This still needs to be flushed out some more, but since the migration requires user interaction, we may need to create an admin tool to handle the migration. Until outcomes are migrated, it might make sense to continue to show the outcomes in the grade book, but remove the ability to do any editing of old outcomes.