Learning analytics indicators

Overview

Models can contain multiple indicators (predictors). Conceptually, indicators collect and aggregate values from a set of data and will combine with targets to make predictions using the selected machine learning algorithms. Indicators can be re-used between models, or can be created specifically for a new model. However, it is best to understand the target you are trying to predict before selecting indicators.

To be used in a model, indicators need to be centered and scaled. In Moodle learning analytics, indicators need to return values converted to a range between -1 and 1. Not all of this range needs to be used-- some indicators will make more sense if scaled from 0 to 1, with 0 indicating a missing value.

How the data points are processed involves important assumptions that affect the indicators. In particular, indicators can be absolute, meaning that the value of the indicator stays the same no matter what other samples are in the context, or relative, meaning that the indicator compares the sample to others in the context.

Currently, Moodle does not filter the available indicators for a model according to the analyser (controlled by the target of the model). However, not all indicators are designed to work with all targets. For example, an indicator about a user will not perform well in a model about course design. In the future, Moodle may offer more guidance about how to select indicators that match the target.

General indicators

Course accessed after end date -- This binary indicator is set to 1 if the user has accessed the course after the end date.
Course accessed before start date -- This binary indicator is set to 1 if the user has accessed the course before the start date.
Any write action -- This binary indicator is set to 1 if the user has completed a "write" action (saved content) anywhere on the site.
Any write action in the course -- This binary indicator is set to 1 if the user has completed a "write" action (saved content) in the enrolled course.
Read actions amount -- This indicator estimates the amount of content the user has accessed (read) on a scale of -1 to 1.
Activities due -- This binary indicator is set to 1 if the user has activities due and not yet submitted.
Completion tracking enabled -- This binary indicator is set to 1 if the course has completion tracking enabled.
There are no students -- This binary indicator is set to 1 if there are no students enroled in the course.
There are no teachers -- This binary indicator is set to 1 if there are no teachers enroled in the course.
User profile is completed -- This indicator returns a value ranging between -1 and 1 depending on how much the user has completed the user's profile.
User is tracking forums -- This binary indicator is set to 1 if the user has forum tracking turned on.

Community of Inquiry indicators

A set of indicators is provided that are based on the Community of Inquiry model of student engagement. These indicators are divided into two types: cognitive depth and social breadth. For each activity in Moodle core, one indicator of each type has been implemented. There are also two course-level "potential" indicators that estimate the potential cognitive depth and social breadth of the course as a whole.

Third-party activity plugins require support by their maintainers to add indicators of any type.

Potential indicator levels for selected activity modules

The potential for engagement via cognitive presence and social presence constitutes instructional design, which is one key element of teaching presence. This diagram shows the potential cognitive depth and social breadth of all core activities and select non-core activities:

By categorizing each activity by potential cognitive depth and social breadth, we can anticipate what level of engagement is supported (and possibly expected) of the learner, even without a history of many learners’ actions in that activity instance. Note that higher levels along each axis include all lower levels, i.e. an activity that involves a student and all peers (social breadth 3) automatically includes levels 1 (student only) and 2 (student +1). In many cases, the specific level can only be determined by analyzing the parameter settings for the activity.

Cognitive depth

Cognitive depth is a measure of the construct "cognitive presence" within the Community of Inquiry theoretical framework. Cognitive presence is defined as “The extent to which the participants in any particular configuration of a community of inquiry are able to construct meaning through sustained communication” (Garrison, Anderson & Archer, 2000, p 89). Cognitive presence has usually been determined in research by manual content analysis. In these indicators, we define this construct based on the type of activity offered to the student, and the extent to which the student demonstrates cognitive engagement in that activity. The level of depth ranges from 0 to 5, where 0 indicates that the learner has not even viewed the activity. The levels of potential cognitive depth are:

The learner has viewed the activity details
The learner has submitted content to the activity
The learner has viewed feedback from an instructor or peer for the activity (including automated feedback)
The learner has provided feedback to the instructor or a peer within the activity
The learner has revised and/or resubmitted content to the activity

This model begins by assigning a maximum potential value of cognitive depth to each activity module. For example, the Assignment module allows up to cognitive depth of 5.

Once the potential levels are assigned, each student enrolled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 (viewing feedback) and the student has reached level 3, the student is participating at 100 percent of the possible level of cognitive depth and the value of the indicator is 1. If the student does not view the activity, they are at cognitive depth of 0, and the value of the indicator is -1. If the student has viewed the activity and submitted a response, but not viewed feedback, they are at cognitive depth level 2 (out of a potential depth in this activity of 3) so the indicator is calculated at .33.

Course level cognitive depth indicators

Course potential cognitive depth - calculated from all the activities in the course as a measure of potential engagement

Course enrolment level cognitive depth indicators

The following cognitive depth indicators have 1 level and report 0% (not viewed), or 100% (viewed).

Book cognitive
Folder cognitive
IMS pkg cognitive
Label cognitive (considered viewed if that portion of the course is viewed)
Page cognitive
File cognitive
URL cognitive

The following cognitive depth indicators have 2 levels and report 0% (not viewed), 50% (viewed) or 100% (interacted with at least one other participant).

Choice cognitive (if results are not viewable)
Database cognitive
Feedback cognitive
Glossary cognitive
Survey cognitive
Wiki cognitive

The following cognitive depth indicators have 3 levels and report 0% (not viewed), 33% (viewed), 67% (submitted) or 100% (submitted and viewed feedback).

Choice cognitive (if results are viewable)
LTI cognitive
SCORM cognitive

The following cognitive depth indicators have 4 levels and report 0% (not viewed), 25% (viewed), 50% (submitted), 75% (submitted and viewed feedback) or 100% (provided feedback to instructor or peer).

Chat cognitive
Forum cognitive

The following cognitive depth indicators have 4 levels and report 0% (not viewed), 20% (viewed), 40% (submitted), 60% (submitted and viewed feedback), 80% (provided feedback to instructor or peer), or 100% (revised and resubmitted).

Assignment cognitive
Lesson cognitive
Quiz cognitive
Workshop cognitive

Social breadth

Social breadth is a measure of the construct "social presence" within the Community of Inquiry theoretical framework. It is defined as “The ability of participants to identify with the group or course of study, communicate purposefully in a trusting environment, and develop personal and affective relationships progressively by way of projecting their individual personalities” (Garrison, 2009, p 352). In the past, social presence has usually been measured via post-course surveys and manual discourse analysis, though there have been increasing attempts to automate this process. This model implements social presence as "social breadth" by examining the breadth of opportunities the participant has to communicate with others. The level of breadth ranges from 0 to 5, where 0 indicates the learner has not interacted with anyone. The levels of potential social breadth are:

The learner has not interacted with any other participant in this activity (e.g. they have read a page)
The learner has interacted with at least one other participant (e.g. they have submitted an assignment or attempted a self-grading quiz providing feedback)
The learner has interacted with multiple participants in this activity, e.g. posting to a discussion forum, wiki, database, etc.*
The learner has interacted with participants in at least one "volley" of communications back and forth*
The learner has interacted with people outside the class, e.g. in an authentic community of practice*

* Not yet implemented

This model begins by assigning a maximum potential value of social breadth to each activity module. For example, the Assignment module allows up to social breadth of 2. See below for more details of how these levels are assigned for core activity modules.

Once the potential levels are assigned, each student enrolled in a course is evaluated based on the proportion of the potential depth reached. For example, if an activity only supports up to level 3 and the student has reached level 3, the student is participating at 100 percent of the possible level of social breadth.

Note: at present, Moodle only provides indicators up to Social Breadth level 2. All "resources" have a maximum social breadth of 1. Activities vary in their supported social breadth. This feature is meant to be expanded in future versions of Moodle.

Course level social breadth indicators

Course potential social breadth - calculated from all the activities in the course

Enrolment level social breadth indicators

The following social breadth indicators have only one levels and report either 0% (not viewed) or 100% (viewed).

Book social
Database social
Folder social
Glossary social
IMS pkg social
Label social
Page social
File social
Survey social
URL social
Wiki social

The following social breadth indicators have two levels and report 0% (not viewed), 50% (viewed) or 100% (interacted with at least one other participant).

Assignment social
Chat social
Choice social
Feedback social
Forum social
Lesson social
LTI social
Quiz social
SCORM social
Workshop social

Documentation