Learning Analytics Specification
| Learning Analytics | |
|---|---|
| Project state | Planning |
| Tracker issue | XXX |
| Discussion | XXX |
| Assignee | XXX |
Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.
This is a specification for a Learning Analytics system including an API for collection of learning analytics data and a number of interfaces for presenting this information.
The specification started at the Learning Analytics working group during MoodleMoot US 2015 (course, notes).
What are Learning Analytics?
Learning Analytics are any piece of information that can help an LMS user improve learning outcomes. Users include students, teachers, administrators and decision-makers.
There are a number of reports, blocks and other plugins for Moodle that provide analytics.
| Plugin | Plugin type | Standard/Third-party | Useful for | Reported usage* |
|---|---|---|---|---|
| Logs | Report | Standard | Teachers, Admins, Decision-makers | 71.4% |
| Activity | Report | Standard | Teachers | 69.1% |
| Activity completion | Report | Standard | Teachers | 66.3% |
| Live logs | Report | Standard | Teachers, Admins | 55.2% |
| (Quiz) Statistics | Report | Standard | Teachers | 53.0% |
| (Course) Participation | Report | Standard | Teachers | 49.9% |
| Course overview | Report | Standard | Admins, Decision-makers | 45.0% |
| Course completion status | Block | Standard | Students | 41.4% |
| Progress Bar | Block | Third-party | Students, Teachers | 32.0% |
| Events list | Report | Standard | Teachers, Admins | 28.6% |
| Activity results block | Block | Standard | Students | 26.1% |
| Configurable Reports | Block | Third-party | Teachers, Admins, Decision-makers | 22.7% |
| Ad-hoc database queries | Report | Third-party | Teachers, Admins, Decision-makers | N/A |
| Engagement Analytics | Block, Report, Activity module | Third-party | Teachers | N/A |
| Course Dedication | Block | Third-party | Students, Teachers | N/A |
| Graph Stats | Block | Third-party | Teachers, Admins | N/A |
* Reported usage is drawn from the Plugins Usage Survey from 2015.
Why is a Learning Analytics system needed?
Although there are already a number of sources of learning analytics, there is no API or consistent source of learning analytics that combines data from multiple sources and makes it accessible to by plugins throughout Moodle.
Learning analytics are needed to support teachers, the staff that support teachers and for students themselves. The primary need for learning analytics has been identified as promoting student retention by identifying students at risk of failure, based on learning analytics.
A learning analytics system should be:
- predictive (combining data to show current status and enable prediction of outcomes)
- proactive (messaging users when actions are needed)
- verifiable (able to compare predictions to actual outcomes to measure accuracy)
A learning analytics system will be able to make use of pre-fetched data that can be combined in varying ways, on the fly.
Predictions will use models (weights applied to sources) based on institution/organisation calibration, able to be varied on the fly.
Learning Analytics data gathering API
XXX DESCRIPTION OF API XXX
Sources of Learning Analytics
Each of the designated Learning Analytics sources should be collected so they can be included/excluded
If calculation of values is required, this should be done regularly as a background task so calculated values are available for combination as needed.
- Course access/views (useful early in course) (on the fly DB access)
- Activity/resource views, reads (event observer, using R in CRUD)
- Activity resource submissions, postings, etc. (event observer, using C,U in CRUD)
- Gradebook current grades and course grade (on the fly DB access?)
- Completion status, if used (on the fly DB access)
- Assessment Feedback views (quiz, assignment, workshop, others?) (events observer, specific events)
- Interaction between students (Social Network Analysis) (forum posts vs views, diversity of interaction vs number of) (pre-calculated from DB)
Users will be able to weight the value of each source and combine them through interfaces on the fly.
Data structure
Status code (e.g., “At risk”, “Normal”, “Ahead”...) calculated for a user with a role based on multiple...
- Metric
- Name
- Event type (e.g., submission)
- Event filter (all activities, all activities of a type, specific activity)
- Transformation/Scale --> [0,1] divide by #, c.f., norm of class, % of possible max grade, etc.
OR
Criterion --> operator, value (e.g., expected by date), date range - Revision number
- Timestamp
...and having a...
- Validity calculation - vs Final grade? Course completion?
- Calculated later
- Would want to be able to run all possible metrics and find what would have been the best predictor for next time the course is taught
Interfaces
XXX
List of students & status (teachers, advisors, financial aid… )
- track notifications sent
- Graphic, coloured icons
- Relative performance
Detail view per student
- Graphic, coloured icons
- Relative performance
- Would students see this? Maybe not by default? Perhaps with a capability.
Configuration Interface (narrowest audience)
- metrics, including notifications sent
- criteria, when, any activity, any quiz, normative vs absolute
- weight
- name
- validity
- Would want to have global metrics (set by admins) and course metrics (additional metrics created by teachers)
Push notifications (widest audience)
- messaging
- based on trigger or manual
- audience templates
- target of message could be student, teacher, mentor or other user
(A useful interface mockup tool is Balsamiq)
How it works
XXX
Benefits
- XXX
Usage scenarios
- XXX
Future work
XXX
Other potential extensions to the system
- Reporting of resource/activity effectiveness in contributing to learning outcomes
- Potential for students to opt in/out of data collection, analysis, reporting or exports. This may be necessary in some places. It might be possible to have student logging be anonymous or exported anonymously.
- Ability to export data as a static dump or as a stream of analytics information, possibly conforming to the IMS Caliper standard, for extra-system analysis.
- Third-party plugins that use the Learning Analytics API
Other potential sources of Learning Analytics data
There are other possible sources of Learning Analytics. As well as those suggested above, here are some additional potential sources.
- performance in other courses enrolled (now & previously)
- student ratings on forums
- self-evaluation
- history of activity within the current course
- history of course activity within a faculty/category
- mood/sense-making
- activity in relation to deadlines
- time spent online based on activity captured through JavaScript
- demographic/profile data
- year/progress
- past grades, prior credit, GPA
- completion of prerequisites vs recommended prerequisites
- personal background
- native language, exceptionalities / accommodations
- cross-instance performance across multiple Moodle instances
Other potential interfaces
- An institutional summary view showing an overview of student retention in courses
- Activity restrictions based on risk status based on learning analytics
