Learning analytics indicators: Difference between revisions
- For site administrators
- Learning analytics models
- For teachers
- For developers
- For researchers
(new page created) |
No edit summary |
||
| (9 intermediate revisions by 2 users not shown) | |||
| Line 1: | Line 1: | ||
{{Learning analytics}} | |||
== Overview == | |||
Models can contain multiple indicators (predictors). Conceptually, indicators collect and aggregate values from a set of data and will combine with targets to make predictions using the selected machine learning algorithms. Indicators can be re-used between models, or can be created specifically for a new model. However, it is best to understand the target you are trying to predict before selecting indicators. [[File:indicators.png|thumb]] | |||
To be used in a model, indicators need to be centered and scaled. In Moodle learning analytics, indicators need to return values converted to a range between -1 and 1. Not all of this range needs to be used-- some indicators will make more sense if scaled from 0 to 1, with 0 indicating a missing value. [[File:centering_scaling_indicators.png|thumb]] | |||
How the data points are processed involves important assumptions that affect the indicators. In particular, indicators can be absolute, meaning that the value of the indicator stays the same no matter what other samples are in the context, or relative, meaning that the indicator compares the sample to others in the context. | |||
Currently, Moodle does not filter the available indicators for a model according to the analyser (controled by the [[Learning analytics targets | target]] of the model). However, not all indicators are designed to work with all targets. For example, an indicator about a user will not perform well in a model about course design. In the future, Moodle may offer more guidance about how to select indicators that match the target. | |||
== General indicators == | == General indicators == | ||
* Course accessed after end date | * '''Course accessed after end date''' -- This binary indicator is set to 1 if the user has accessed the course after the end date. | ||
* Course accessed before start date | * '''Course accessed before start date''' -- This binary indicator is set to 1 if the user has accessed the course before the start date. | ||
* Any write action | * '''Any write action''' -- This binary indicator is set to 1 if the user has completed a "write" action (saved content) anywhere on the site. | ||
* Any write action in the course | * '''Any write action in the course''' -- This binary indicator is set to 1 if the user has completed a "write" action (saved content) in the enroled course. | ||
* Read actions amount | * '''Read actions amount''' -- This indicator estimates the amount of content the user has accessed (read) on a scale of -1 to 1. | ||
* Activities due | * '''Activities due''' -- This binary indicator is set to 1 if the user has activities due and not yet submitted. | ||
* Completion tracking enabled | * '''Completion tracking enabled''' -- This binary indicator is set to 1 if the course has completion tracking enabled. | ||
* There are no students | * '''There are no students''' -- This binary indicator is set to 1 if there are no students enroled in the course. | ||
* There are no teachers | * '''There are no teachers''' -- This binary indicator is set to 1 if there are no teachers enroled in the course. | ||
* '''User profile is completed''' -- This binary indicator is set to 1 if the user has completed their profile. | |||
* '''User is tracking forums''' -- This binary indicator is set to 1 if the user has forum tracking turned on. | |||
== Community of Inquiry indicators == | |||
A set of indicators is provided that are based on the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement. These indicators are divided into two types: '''cognitive depth''' and '''social breadth'''. For each activity in Moodle core, one indicator of each type has been implemented. There are also two course-level "potential" indicators that estimate the potential cognitive depth and social breadth of the course as a whole. | |||
Third-party activity plugins require support by their maintainers to add indicators of any type. | |||
=== Potential indicator levels for selected activity modules === | |||
The potential for engagement via cognitive presence and social presence constitutes '''instructional design''', which is one key element of teaching presence. | |||
This diagram shows the potential cognitive depth and social breadth of all core activities and select non-core activities: | |||
[[File:coi_model_indicators.png|thumb]] | |||
By categorizing each activity by potential cognitive depth and social breadth, we can anticipate what level of engagement is supported (and possibly expected) of the learner, even without a history of many learners’ actions in that activity instance. Note that higher levels along each axis include all lower levels, i.e. an activity that involves a student and all peers (social breadth 3) automatically includes levels 1 (student only) and 2 (student +1). In many cases, the specific level can only be determined by analyzing the parameter settings for the activity. | |||
=== Cognitive depth === | === Cognitive depth === | ||
Cognitive depth is a measure of the construct "cognitive presence" within the Community of Inquiry theoretical framework. Cognitive presence is defined as “The extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained communication” ([http://www.sciencedirect.com/science/article/pii/S1096751600000166 Garrison, Anderson & Archer, 2000], p 89). Cognitive presence has usually been determined in research by manual content analysis. In | Cognitive depth is a measure of the construct "cognitive presence" within the Community of Inquiry theoretical framework. Cognitive presence is defined as “The extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained communication” ([http://www.sciencedirect.com/science/article/pii/S1096751600000166 Garrison, Anderson & Archer, 2000], p 89). Cognitive presence has usually been determined in research by manual content analysis. In these indicators, we define this construct based on the type of activity offered to the student, and the extent to which the student demonstrates cognitive engagement in that activity. The level of depth ranges from 0 to 5, where 0 indicates that the learner has not even viewed the activity. The levels of potential cognitive depth are: | ||
# The learner has viewed the activity details | # The learner has viewed the activity details | ||
# The learner has submitted content to the activity | # The learner has submitted content to the activity | ||
# The learner has viewed feedback from an instructor or peer for the activity | # The learner has viewed feedback from an instructor or peer for the activity (including automated feedback) | ||
# The learner has provided feedback to the instructor or a peer within the activity | # The learner has provided feedback to the instructor or a peer within the activity | ||
# The learner has revised and/or resubmitted content to the activity | # The learner has revised and/or resubmitted content to the activity | ||
This model begins by assigning a maximum potential value of cognitive depth to each activity module. For example, the Assignment module allows up to cognitive depth of | This model begins by assigning a maximum potential value of cognitive depth to each activity module. For example, the Assignment module allows ''up to'' cognitive depth of 5. | ||
Once the potential levels are assigned, each student enroled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 and the student has reached level 3, the student is participating at 100 percent of the possible level of cognitive depth. | Once the potential levels are assigned, each student enroled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 (viewing feedback) and the student has reached level 3, the student is participating at 100 percent of the possible level of cognitive depth and the value of the indicator is 1. If the student does not view the activity, they are at cognitive depth of 0, and the value of the indicator is -1. If the student has viewed the activity and submitted a response, but not viewed feedback, they are at cognitive depth level 2 (out of a potential depth in this activity of 3) so the indicator is calculated at .33. | ||
* Course potential cognitive depth | ==== Course level cognitive depth indicators ==== | ||
* | |||
* | * '''Course potential cognitive depth''' - calculated from all the activities in the course as a measure of potential engagement | ||
* | |||
* | ==== Course enrolment level cognitive depth indicators ==== | ||
* | The following cognitive depth indicators have 1 level and report 0% (not viewed), or 100% (viewed). | ||
* | * '''Book cognitive''' | ||
* | * '''Folder cognitive''' | ||
* | * '''IMS pkg cognitive''' | ||
* Glossary cognitive | * '''Label cognitive''' (considered viewed if that portion of the course is viewed) | ||
* | * '''Page cognitive''' | ||
* | * '''File cognitive''' | ||
* | * '''URL cognitive''' | ||
* LTI cognitive | |||
* | The following cognitive depth indicators have 2 levels and report 0% (not viewed), 50% (viewed) or 100% (interacted with at least one other participant). | ||
* '''Choice cognitive''' (if results are not viewable) | |||
* | * '''Database cognitive''' | ||
* | * '''Feedback cognitive''' | ||
* '''Glossary cognitive''' | |||
* | * '''Survey cognitive''' | ||
* | * '''Wiki cognitive''' | ||
* | |||
* Workshop cognitive | The following cognitive depth indicators have 3 levels and report 0% (not viewed), 33% (viewed), 67% (submitted) or 100% (submitted and viewed feedback). | ||
* '''Choice cognitive''' (if results are viewable) | |||
* '''LTI cognitive''' | |||
* '''SCORM cognitive''' | |||
The following cognitive depth indicators have 4 levels and report 0% (not viewed), 25% (viewed), 50% (submitted), 75% (submitted and viewed feedback) or 100% (provided feedback to instructor or peer). | |||
* '''Chat cognitive''' | |||
* '''Forum cognitive''' | |||
The following cognitive depth indicators have 4 levels and report 0% (not viewed), 20% (viewed), 40% (submitted), 60% (submitted and viewed feedback), 80% (provided feedback to instructor or peer), or 100% (revised and resubmitted). | |||
* '''Assignment cognitive''' | |||
* '''Lesson cognitive''' | |||
* '''Quiz cognitive''' | |||
* '''Workshop cognitive''' | |||
=== Social breadth === | === Social breadth === | ||
| Line 64: | Line 94: | ||
# The learner has not interacted with any other participant in this activity (e.g. they have read a page) | # The learner has not interacted with any other participant in this activity (e.g. they have read a page) | ||
# The learner has interacted with at least one other participant (e.g. they have submitted an assignment or attempted a self-grading quiz providing feedback) | # The learner has interacted with at least one other participant (e.g. they have submitted an assignment or attempted a self-grading quiz providing feedback) | ||
# The learner has interacted with multiple participants in this activity, e.g. posting to a discussion forum, wiki, database, etc. | # ''The learner has interacted with multiple participants in this activity, e.g. posting to a discussion forum, wiki, database, etc.*'' | ||
# The learner has interacted with participants in at least one "volley" of communications back and forth | # ''The learner has interacted with participants in at least one "volley" of communications back and forth*'' | ||
# The learner has interacted with people outside the class, e.g. in an authentic community of practice | # ''The learner has interacted with people outside the class, e.g. in an authentic community of practice*'' | ||
''* Not yet implemented'' | |||
This model begins by assigning a maximum potential value of social breadth to each activity module. For example, the Assignment module allows up to social breadth of 2. See below for more details of how these levels are assigned for core activity modules. | This model begins by assigning a maximum potential value of social breadth to each activity module. For example, the Assignment module allows up to social breadth of 2. See below for more details of how these levels are assigned for core activity modules. | ||
| Line 72: | Line 103: | ||
Once the potential levels are assigned, each student enroled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 and the student has reached level 3, the student is participating at 100 percent of the possible level of social breadth. | Once the potential levels are assigned, each student enroled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 and the student has reached level 3, the student is participating at 100 percent of the possible level of social breadth. | ||
* Course potential social breadth | '''Note''': at present, Moodle only provides indicators up to Social Breadth level 2. All "[[Resources|resources]]" have a maximum social breadth of 1. [[Activities|Activities]] vary in their supported social breadth. This feature is meant to be expanded in future versions of Moodle. | ||
==== Course level social breadth indicators ==== | |||
* '''Course potential social breadth''' - calculated from all the activities in the course | |||
* | |||
* Database social | ==== Enrolment level social breadth indicators==== | ||
* | The following social breadth indicators have only one levels and report either 0% (not viewed) or 100% (viewed). | ||
* | * '''Book social''' | ||
* '''Database social''' | |||
* IMS pkg social | * '''Folder social''' | ||
* Label social | * '''Glossary social''' | ||
* | * '''IMS pkg social''' | ||
* '''Label social''' | |||
* '''Page social''' | |||
* | * '''File social''' | ||
* '''Survey social''' | |||
* | * '''URL social''' | ||
* '''Wiki social''' | |||
* URL social | |||
* Wiki social | |||
The following social breadth indicators have two levels and report 0% (not viewed), 50% (viewed) or 100% (interacted with at least one other participant). | |||
* '''Assignment social''' | |||
* '''Chat social''' | |||
* '''Choice social''' | |||
* '''Feedback social''' | |||
* '''Forum social''' | |||
* '''Lesson social''' | |||
* '''LTI social''' | |||
* '''Quiz social''' | |||
* '''SCORM social''' | |||
* '''Workshop social''' | |||
[[es:Indicadores de analítica del aprendizaje]] | |||
[[de:Analytics-Indikatoren]] | |||
Latest revision as of 08:44, 25 July 2019
Overview
Models can contain multiple indicators (predictors). Conceptually, indicators collect and aggregate values from a set of data and will combine with targets to make predictions using the selected machine learning algorithms. Indicators can be re-used between models, or can be created specifically for a new model. However, it is best to understand the target you are trying to predict before selecting indicators.
To be used in a model, indicators need to be centered and scaled. In Moodle learning analytics, indicators need to return values converted to a range between -1 and 1. Not all of this range needs to be used-- some indicators will make more sense if scaled from 0 to 1, with 0 indicating a missing value.
How the data points are processed involves important assumptions that affect the indicators. In particular, indicators can be absolute, meaning that the value of the indicator stays the same no matter what other samples are in the context, or relative, meaning that the indicator compares the sample to others in the context.
Currently, Moodle does not filter the available indicators for a model according to the analyser (controled by the target of the model). However, not all indicators are designed to work with all targets. For example, an indicator about a user will not perform well in a model about course design. In the future, Moodle may offer more guidance about how to select indicators that match the target.
General indicators
- Course accessed after end date -- This binary indicator is set to 1 if the user has accessed the course after the end date.
- Course accessed before start date -- This binary indicator is set to 1 if the user has accessed the course before the start date.
- Any write action -- This binary indicator is set to 1 if the user has completed a "write" action (saved content) anywhere on the site.
- Any write action in the course -- This binary indicator is set to 1 if the user has completed a "write" action (saved content) in the enroled course.
- Read actions amount -- This indicator estimates the amount of content the user has accessed (read) on a scale of -1 to 1.
- Activities due -- This binary indicator is set to 1 if the user has activities due and not yet submitted.
- Completion tracking enabled -- This binary indicator is set to 1 if the course has completion tracking enabled.
- There are no students -- This binary indicator is set to 1 if there are no students enroled in the course.
- There are no teachers -- This binary indicator is set to 1 if there are no teachers enroled in the course.
- User profile is completed -- This binary indicator is set to 1 if the user has completed their profile.
- User is tracking forums -- This binary indicator is set to 1 if the user has forum tracking turned on.
Community of Inquiry indicators
A set of indicators is provided that are based on the Community of Inquiry model of student engagement. These indicators are divided into two types: cognitive depth and social breadth. For each activity in Moodle core, one indicator of each type has been implemented. There are also two course-level "potential" indicators that estimate the potential cognitive depth and social breadth of the course as a whole.
Third-party activity plugins require support by their maintainers to add indicators of any type.
Potential indicator levels for selected activity modules
The potential for engagement via cognitive presence and social presence constitutes instructional design, which is one key element of teaching presence. This diagram shows the potential cognitive depth and social breadth of all core activities and select non-core activities:
By categorizing each activity by potential cognitive depth and social breadth, we can anticipate what level of engagement is supported (and possibly expected) of the learner, even without a history of many learners’ actions in that activity instance. Note that higher levels along each axis include all lower levels, i.e. an activity that involves a student and all peers (social breadth 3) automatically includes levels 1 (student only) and 2 (student +1). In many cases, the specific level can only be determined by analyzing the parameter settings for the activity.
Cognitive depth
Cognitive depth is a measure of the construct "cognitive presence" within the Community of Inquiry theoretical framework. Cognitive presence is defined as “The extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained communication” (Garrison, Anderson & Archer, 2000, p 89). Cognitive presence has usually been determined in research by manual content analysis. In these indicators, we define this construct based on the type of activity offered to the student, and the extent to which the student demonstrates cognitive engagement in that activity. The level of depth ranges from 0 to 5, where 0 indicates that the learner has not even viewed the activity. The levels of potential cognitive depth are:
- The learner has viewed the activity details
- The learner has submitted content to the activity
- The learner has viewed feedback from an instructor or peer for the activity (including automated feedback)
- The learner has provided feedback to the instructor or a peer within the activity
- The learner has revised and/or resubmitted content to the activity
This model begins by assigning a maximum potential value of cognitive depth to each activity module. For example, the Assignment module allows up to cognitive depth of 5.
Once the potential levels are assigned, each student enroled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 (viewing feedback) and the student has reached level 3, the student is participating at 100 percent of the possible level of cognitive depth and the value of the indicator is 1. If the student does not view the activity, they are at cognitive depth of 0, and the value of the indicator is -1. If the student has viewed the activity and submitted a response, but not viewed feedback, they are at cognitive depth level 2 (out of a potential depth in this activity of 3) so the indicator is calculated at .33.
Course level cognitive depth indicators
- Course potential cognitive depth - calculated from all the activities in the course as a measure of potential engagement
Course enrolment level cognitive depth indicators
The following cognitive depth indicators have 1 level and report 0% (not viewed), or 100% (viewed).
- Book cognitive
- Folder cognitive
- IMS pkg cognitive
- Label cognitive (considered viewed if that portion of the course is viewed)
- Page cognitive
- File cognitive
- URL cognitive
The following cognitive depth indicators have 2 levels and report 0% (not viewed), 50% (viewed) or 100% (interacted with at least one other participant).
- Choice cognitive (if results are not viewable)
- Database cognitive
- Feedback cognitive
- Glossary cognitive
- Survey cognitive
- Wiki cognitive
The following cognitive depth indicators have 3 levels and report 0% (not viewed), 33% (viewed), 67% (submitted) or 100% (submitted and viewed feedback).
- Choice cognitive (if results are viewable)
- LTI cognitive
- SCORM cognitive
The following cognitive depth indicators have 4 levels and report 0% (not viewed), 25% (viewed), 50% (submitted), 75% (submitted and viewed feedback) or 100% (provided feedback to instructor or peer).
- Chat cognitive
- Forum cognitive
The following cognitive depth indicators have 4 levels and report 0% (not viewed), 20% (viewed), 40% (submitted), 60% (submitted and viewed feedback), 80% (provided feedback to instructor or peer), or 100% (revised and resubmitted).
- Assignment cognitive
- Lesson cognitive
- Quiz cognitive
- Workshop cognitive
Social breadth
Social breadth is a measure of the construct "social presence" within the Community of Inquiry theoretical framework. It is defined as “The ability of participants to identify with the group or course of study, communicate purposefully in a trusting environment, and develop personal and affective relationships progressively by way of projecting their individual personalities” (Garrison, 2009, p 352). In the past, social presence has usually been measured via post-course surveys and manual discourse analysis, though there have been increasing attempts to automate this process. This model implements social presence as "social breadth" by examining the breadth of opportunities the participant has to communicate with others. The level of breadth ranges from 0 to 5, where 0 indicates the learner has not interacted with anyone. The levels of potential social breadth are:
- The learner has not interacted with any other participant in this activity (e.g. they have read a page)
- The learner has interacted with at least one other participant (e.g. they have submitted an assignment or attempted a self-grading quiz providing feedback)
- The learner has interacted with multiple participants in this activity, e.g. posting to a discussion forum, wiki, database, etc.*
- The learner has interacted with participants in at least one "volley" of communications back and forth*
- The learner has interacted with people outside the class, e.g. in an authentic community of practice*
* Not yet implemented
This model begins by assigning a maximum potential value of social breadth to each activity module. For example, the Assignment module allows up to social breadth of 2. See below for more details of how these levels are assigned for core activity modules.
Once the potential levels are assigned, each student enroled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 and the student has reached level 3, the student is participating at 100 percent of the possible level of social breadth.
Note: at present, Moodle only provides indicators up to Social Breadth level 2. All "resources" have a maximum social breadth of 1. Activities vary in their supported social breadth. This feature is meant to be expanded in future versions of Moodle.
Course level social breadth indicators
- Course potential social breadth - calculated from all the activities in the course
Enrolment level social breadth indicators
The following social breadth indicators have only one levels and report either 0% (not viewed) or 100% (viewed).
- Book social
- Database social
- Folder social
- Glossary social
- IMS pkg social
- Label social
- Page social
- File social
- Survey social
- URL social
- Wiki social
The following social breadth indicators have two levels and report 0% (not viewed), 50% (viewed) or 100% (interacted with at least one other participant).
- Assignment social
- Chat social
- Choice social
- Feedback social
- Forum social
- Lesson social
- LTI social
- Quiz social
- SCORM social
- Workshop social
