Difference between revisions of "Analytics"

Jump to: navigation, search
(Students at risk of dropping out: Added link to CoI Wikipedia entry)
(moved a statement from "analytics vs reporting" to "Features")
 
(50 intermediate revisions by 8 users not shown)
Line 1: Line 1:
{{Tracking progress}}{{New features}}
+
{{Learning analytics}}
 +
==Overview==
  
==Overview==
+
{{MediaPlayer | url = https://youtu.be/UHwfG6q9UsA | desc = 3.7 Analytics overview}}
Beginning in version 3.4, Moodle core now implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.
+
 
 +
Beginning in version 3.4, Moodle core implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.
 +
 
 +
=== What are learning analytics? ===
 +
Learning analytics are software algorithms that are used to predict or detect unknown aspects of the learning process, based on historical data and current behavior. There are four main categories of learning analytics:
 +
* descriptive (what happened?)
 +
* predictive (what will happen next?)
 +
* diagnostic (why did it happen?)
 +
* prescriptive (do this to improve)
 +
 
 +
Most commercial solutions are descriptive only. Those that are predictive or proactive make certain assumptions about learning that don’t apply to everyone.
  
In Moodle 3.4, this system ships with two built-in models:
+
=== Analytics vs. reporting ===
 +
Moodle provides a variety of built-in reports based on log data, but they are primarily descriptive in nature -- they tell participants what happened, but not why, and they don’t predict outcomes or advise participants how to improve outcomes. Log entries, while very detailed, are not in themselves descriptive of the learning process.  They tell us “who,” “what,” and “when,” but not “why” or “how well.” Much more context is needed around each micro-action to develop a pattern of engagement.
  
* Students at risk of dropping out
+
Many third-party plugins also exist for Moodle that provide descriptive analytics. There are also integrations with third-party off-site reporting solutions. Again, these primarily provide descriptive analytics that rely on human judgment to interpret reports and generate predictions and prescriptions.
* No teaching activity
 
  
The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the [[dev:Analytics API| Analytics API]] developer documentation.
+
Often in the past, learning analytics systems have attempted to analyze past activities to predict future activities in real time. With Moodle Learning Analytics, we are more ambitious. We believe a full learning analytics solution will help us not only predict events, but change them to be more positive.
  
 
===Features===
 
===Features===
* Two built-in prediction models: "Students at risk of dropping out of courses" and "No Teaching Activity".
+
* Two types of models supported:
* A set of student engagement indicators based on the Community of Inquiry.
+
** Machine-learning based models, including predictive models
 +
** "Static" models to detect situations of concern using simple rules
 +
* Three built-in models: "[[Students at risk of dropping out]]", "Upcoming activities due" and "No Teaching".{{New features}}
 +
* A set of student engagement [[Learning_analytics_indicators|indicators]] based on the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry].
 
* Built-in tools to evaluate models against your site's data
 
* Built-in tools to evaluate models against your site's data
* Proactive notifications for instructors using Events
+
* Proactive notifications using [[Events_list_report|Events]]
* Instructors can easily send messages to students identified by the model, or jump to the Outline report for that student for more detail about student activity
+
* A list of suggested Actions is provided with the Insight notifications for each model. For example, in the [[Students at risk of dropping out]] model, instructors can easily send messages to students identified by the model, or jump to the [[Activity_report|Activity report]] for that student for more detail about student activity within the course
 
* An [[dev:Analytics API|  API]] to build indicators and prediction models for third-party Moodle plugins
 
* An [[dev:Analytics API|  API]] to build indicators and prediction models for third-party Moodle plugins
 
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends
 
* Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends
 
+
* The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the [[dev:Analytics API| Analytics API]] developer documentation.
'''Note: PHP 7.x is required.'''
 
  
 
===Limitations===
 
===Limitations===
This release of Moodle Learning Analytics has the following limitations:
 
  
* Models included in this release must be "trained" on a site with previous completed courses, ideally using the Moodle course completion feature. The current models cannot make predictions on a site until this is done.
+
* Machine learning models such as [[Students at risk of dropping out]] must be trained on a site with data . These models cannot make predictions on a site until this is done.
* The prediction model included with this version requires that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle.
+
* Models must be designed and selected to match the educational priorities of the institution.
* Models and predictions are only visible to teachers and administrators at present.
 
 
 
We are continuing to enhance Moodle Learning Analytics, and expanded capabilities will be released going forward. To help contribute to our progress, please join the conversation at the [https://moodle.org/project_inspire Moodle Learning Analytics Community]. In particular, we still need data sets from a wide variety of Moodle-using institutions in order to be able to ship a working prediction model that does not depend on local site data before it can be used.
 
  
 
== Settings ==
 
== Settings ==
  
You can access ''Analytics settings'' from ''Site administration > Analytics > Analytics settings''.
+
The Moodle learning analytics system requires some initial configuration before use. See [[Analytics settings]] for more detail.
  
=== Predictions processor ===
+
== Using analytics ==
 +
The Moodle Learning Analytics API is an open system that can become the basis for a very wide variety of models. Models can contain indicators (a.k.a. predictors), targets (the outcome we are trying to predict), insights (the predictions themselves), notifications (messages sent as a result of insights), and actions (offered to recipients of messages, which can become indicators in turn).
  
[[Image:analytics01_predictions_processor34.png|frame|center|Predictions processor selection]]
+
Most learning analytics models are not enabled by default. Enabling models for use should be done after considering the institutional goals the models are meant to support. See [[Using analytics]] for more information.
  
Prediction processors are the machine learning backends that process the datasets generated from the calculated indicators and targets and return predictions. Moodle core includes 2 prediction processors:
+
== Managing models ==
 +
Once models have been enabled and trained, insights will be generated. Models should also be monitored for performance and accuracy. See [[Managing analytics]] for more information.
  
* The PHP processor is the default. There are no other system requirements to use this processor.
+
==Capabilities==
* The Python one is more powerful and it generates [https://www.tensorflow.org/get_started/summaries_and_tensorboard graphs that explain the model performance]. It requires setting up extra tools: Python itself (https://wiki.python.org/moin/BeginnersGuide/Download) and the moodlemlbackend python package.
 
  
    pip install moodlemlbackend
+
There are two analytics capabilities:
  
=== Time splitting methods ===
+
* [[Capabilities/moodle/analytics:managemodels|Manage models]] - allowed for the default role of manager only
 +
* [[Capabilities/moodle/analytics:listinsights|List insights]] - allowed for the default roles of manager, teacher and non-editing teacher
  
Time splitting methods allow insights generated from one course to be used on another course, even if the two courses are not exactly the same length.
 
  
[[Image:06_timesplitting.png|frame|center|Time splitting methods]]
+
==Frequently Asked Questions==
 +
[[Moodle Learning Analytics FAQ]]
  
Each time splitting method divides the course duration into segments. At the end of each defined segment, the predictions engine will run and generate insights. It is recommended that you only enable the time splitting methods you are interested in using; the evaluation process will iterate through all enabled time-spitting methods, so the more time-splitting methods enabled, the slower the evaluation process will be.
+
[[Category:Analytics]]
 
 
===Models output directory===
 
 
 
[[Image:03_models_output_directory.png|frame|center|Models output directory]]
 
 
 
This setting allows you to define a directory where machine learning backends data is stored. Be sure this directory exists and is writable by the web server.
 
 
 
== Model management ==
 
 
 
Moodle can support multiple prediction models at once, even within the same course. This can be used for A/B testing to compare the performance and accuracy of multiple models.
 
 
 
Moodle core ships with two prediction models ''Students at risk of dropping out'' and ''No teaching.'' Additional prediction models can be created by using [[dev:Analytics API| Analytics API]]. Each model is based on the prediction of a single, specific "target," or outcome (whether desirable or undesirable), based on a number of selected indicators.
 
 
 
You can manage your system models from ''Site Administration > Analytics > Analytics models''.
 
 
 
[[File:prediction-models-list.jpeg]]
 
 
 
These are some of the actions you can perform on a model:
 
 
 
* '''Get predictions:''' Train machine learning algorithms with the new data available on the system and get predictions for ongoing courses. ''Predictions are not limited to ongoing courses, this depends on the model.''
 
* '''Evaluate:''' (disabled by default) Evaluate the prediction model by getting all the training data available on the site, calculating all the indicators and the target and passing the resulting dataset to machine learning backends, they will split the dataset into training data and testing data and calculate its accuracy. Note that the evaluation process use all information available on the site, even if it is very old, the accuracy returned by the evaluation process will generally be lower than the real model accuracy as indicators are more reliably calculated straight after training data is available because the site state changes along time. The metric used as accuracy is the ''Matthew’s correlation coefficient'' (good metric for binary classifications)
 
* '''Log:''' View previous evaluations log, including the model accuracy as well as other technical information generated by the machine learning backends like ROC curves, learning curves graphs, the tensorboard log dir or the model's Matthew’s correlation coefficient. The information available will depend on the machine learning backend in use.
 
* '''Edit:''' You can edit the models by modifying the list of indicators or the time-splitting method. All previous predictions will be deleted when a model is modified. Models based on assumptions (static models) can not be edited.
 
* '''Enable / Disable:''' The scheduled task that trains machine learning algorithms with the new data available on the system and gets predictions for ongoing courses skips disabled models. Previous predictions generated by disabled models are not available until the model is enabled again.
 
* '''Export''': Export your site training data to share it with your partner institutions.
 
* '''Invalid site elements''': Reports on what elements in your site can not be analysed by this model
 
* '''Clear predictions''': Clears all the model predictions and training data
 
 
 
[[File:model-evaluation.jpeg]]
 
 
 
You can force the model evaluation process to run from the command line:
 
 
 
$ admin/tool/analytics/cli/evaluate_model.php
 
 
 
 
 
=== Core models ===
 
 
 
==== Students at risk of dropping out ====
 
 
 
This model predicts students who are at risk of non-completion (dropping out) of a Moodle course, based on low student engagement. In this model, the definition of "dropping out" is "no student activity in the last week of the course." The prediction model uses the [https://en.wikipedia.org/wiki/Community_of_inquiry Community of Inquiry] model of student engagement, consisting of three parts:
 
  
* Cognitive presence
+
[[es:Analítica]]
* Social presence
+
[[de:Analytics]]
* Teacher presence
+
[[fr:Analyses de données]]
 
 
This prediction model is able to analyze and draw conclusions from a wide variety of courses, and apply those conclusions to make predictions about new courses. The model is not limited to making predictions about student success only in exact duplicates of courses offered in the past. However, there are some limitations:
 
 
 
# This prediction model assumes that courses have fixed start and end dates, and is not designed to be used with rolling enrollment courses. Models that support a wider range of course types will be included in future versions of Moodle. Because of this model design assumption, it is very important to properly set course start and end dates for each course to use this model. If both past courses and ongoing courses start and end dates are not properly set predictions cannot be accurate. Because the course end date field was only introduced in Moodle 3.2 and some courses may not have set a course start date in the past, we include a command line interface script:
 
 
 
$ admin/tool/analytics/cli/guess_course_start_and_end.php
 
 
 
This script attempts to estimate past course start and end dates by looking at the student enrolments and students' activity logs. After running this script, please check that the estimated start and end dates script results are reasonably correct.
 
 
 
# This model requires a certain amount of in-Moodle data with which to make predictions. At the present time, only core Moodle activities are included in the indicator set (see below). Courses which do not include several core Moodle activities per “time slice” will have poor predictive support in this model. This prediction model will be most effective with fully online or “hybrid” or “blended” courses with substantial online components.
 
 
 
==== No teaching ====
 
 
 
This model's insights will inform site managers of which courses with an upcoming start date will not have teaching activity. This is a simple model and it does not use machine learning backend to return predictions. It bases the predictions on assumptions, e.g. there is no teaching if there are no students.
 
 
 
== Predictions and Insights ==
 
 
 
Models will start generating predictions at different point in time, depending on the site prediction models and the site courses start and end dates.
 
 
 
Each model defines which predictions will generate insights and which predictions will be ignored. This is an example of ''Students at risk of dropping out'' prediction model; if a student is predicted as not at risk no insight is generated as what is interesting is to know which students are at risk of dropping out of courses, not which students are not at risk.
 
 
 
[[File:prediction-model-insights.jpeg]]
 
 
 
[[Category:Analytics]]
 

Latest revision as of 17:26, 16 July 2019

Overview

3.7 Analytics overview

Beginning in version 3.4, Moodle core implements open source, transparent next-generation learning analytics using machine learning backends that go beyond simple descriptive analytics to provide predictions of learner success, and ultimately diagnosis and prescriptions (advisements) to learners and teachers.

What are learning analytics?

Learning analytics are software algorithms that are used to predict or detect unknown aspects of the learning process, based on historical data and current behavior. There are four main categories of learning analytics:

  • descriptive (what happened?)
  • predictive (what will happen next?)
  • diagnostic (why did it happen?)
  • prescriptive (do this to improve)

Most commercial solutions are descriptive only. Those that are predictive or proactive make certain assumptions about learning that don’t apply to everyone.

Analytics vs. reporting

Moodle provides a variety of built-in reports based on log data, but they are primarily descriptive in nature -- they tell participants what happened, but not why, and they don’t predict outcomes or advise participants how to improve outcomes. Log entries, while very detailed, are not in themselves descriptive of the learning process. They tell us “who,” “what,” and “when,” but not “why” or “how well.” Much more context is needed around each micro-action to develop a pattern of engagement.

Many third-party plugins also exist for Moodle that provide descriptive analytics. There are also integrations with third-party off-site reporting solutions. Again, these primarily provide descriptive analytics that rely on human judgment to interpret reports and generate predictions and prescriptions.

Often in the past, learning analytics systems have attempted to analyze past activities to predict future activities in real time. With Moodle Learning Analytics, we are more ambitious. We believe a full learning analytics solution will help us not only predict events, but change them to be more positive.

Features

  • Two types of models supported:
    • Machine-learning based models, including predictive models
    • "Static" models to detect situations of concern using simple rules
  • Three built-in models: "Students at risk of dropping out", "Upcoming activities due" and "No Teaching".New feature
    in Moodle 3.7!
  • A set of student engagement indicators based on the Community of Inquiry.
  • Built-in tools to evaluate models against your site's data
  • Proactive notifications using Events
  • A list of suggested Actions is provided with the Insight notifications for each model. For example, in the Students at risk of dropping out model, instructors can easily send messages to students identified by the model, or jump to the Activity report for that student for more detail about student activity within the course
  • An API to build indicators and prediction models for third-party Moodle plugins
  • Machine learning backend plugin type - supports PHP and Python, and can be extended to implement other ML backends
  • The system can be easily extended with new custom models, based on reusable targets, indicators, and other components. For more information, see the Analytics API developer documentation.

Limitations

  • Machine learning models such as Students at risk of dropping out must be trained on a site with data . These models cannot make predictions on a site until this is done.
  • Models must be designed and selected to match the educational priorities of the institution.

Settings

The Moodle learning analytics system requires some initial configuration before use. See Analytics settings for more detail.

Using analytics

The Moodle Learning Analytics API is an open system that can become the basis for a very wide variety of models. Models can contain indicators (a.k.a. predictors), targets (the outcome we are trying to predict), insights (the predictions themselves), notifications (messages sent as a result of insights), and actions (offered to recipients of messages, which can become indicators in turn).

Most learning analytics models are not enabled by default. Enabling models for use should be done after considering the institutional goals the models are meant to support. See Using analytics for more information.

Managing models

Once models have been enabled and trained, insights will be generated. Models should also be monitored for performance and accuracy. See Managing analytics for more information.

Capabilities

There are two analytics capabilities:

  • Manage models - allowed for the default role of manager only
  • List insights - allowed for the default roles of manager, teacher and non-editing teacher


Frequently Asked Questions

Moodle Learning Analytics FAQ