Learning Analytics Specification
| Learning Analytics | |
|---|---|
| Project state | Planning |
| Tracker issue | XXX |
| Discussion | XXX |
| Assignee | XXX |
Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.
This is a specification for a Learning Analytics system including an API for collection of learning analytics data and a number of interfaces for presenting this information.
The specification started at the Learning Analytics working group during MoodleMoot US 2015 (course, notes).
What are Learning Analytics?
Learning Analytics are any piece of information that can help an LMS user improve learning outcomes. Users include students, teachers, administrators and decision-makers.
There are a number of reports, blocks and other plugins for Moodle that provide analytics. A list of Learning Analytics plugins is listed in the user docs.
Why is a Learning Analytics system needed?
Although there are already a number of sources of learning analytics, there is no API or consistent source of learning analytics that combines data from multiple sources and makes it accessible to plugins throughout Moodle.
Learning analytics are needed to inform teachers about students, to inform the staff that support teachers and to inform students themselves. The primary need for learning analytics has been identified as promoting student retention by identifying student's status (at risk, normal, ahead), based on their activity.
A learning analytics system should be:
- predictive (combining data to show current status and enable prediction of future status)
- proactive (messaging users when action is needed to change status)
- verifiable (able to compare predictions to actual outcomes to measure accuracy)
A learning analytics system will be able to make use of pre-fetched data that can be combined in varying ways, on the fly.
Predictions will use models (weights applied to source data) based on institution/organisation calibration and able to be adjusted on the fly.
Learning Analytics data gathering API
XXX DESCRIPTION OF API XXX
Sources of Learning Analytics
Each of the designated Learning Analytics sources should be collected so they can be included/excluded in combined calculations on the fly. Teachers and other users should be able to manipulate weights to explore the effect this has on student models and predictions.
If pre-calculation of values is required, this should be done regularly as a background task so values are available for combination as needed.
The following values were identified as important learning analytics that need to be made available.
- Course access/views (useful early in course) (on the fly DB access)
- Activity/resource views, reads (event observer, using R in CRUD)
- Activity resource submissions, postings, etc. (event observer, using C,U in CRUD)
- Gradebook current grades and course grade (on the fly DB access?)
- Completion status, if used (on the fly DB access)
- Assessment Feedback views (quiz, assignment, workshop, others?) (events observer, specific events)
- Interaction between students (Social Network Analysis) (forum posts vs views, diversity of interaction vs number of) (pre-calculated from DB)
A learning analytics source API is a pluggable system so that additional sources can be added. each source would need to store learning analytics when required or on a regular basis (depending on what needs to be gathered/calculated) so that it is ready to be combined in status calculations based on a particular configuration (student model).
Sources of analytics produce a floating point value between zero and one. This value limitation is important so that learning analytics can have relative meaning. The way of calculating this value will differ for each source because they are drawing on different information. Such information may be calculated based on:
- a proportion of a possible maximum grade/value,
- a normalised relative score for a user within a context,
- events taking place in relation to expected dates (eg., submitting before a deadline) or
- a more complex algorithmic combination of information.
LA sources themselves may have some configuration so they can be tweaked under different circumstances (such as education sector or class scales). Such configuration would probably be limited to the site level.
Sources should continue to work when there is an absence of relevant information on which to base a calculation. When this is the case, zero values should be returned.
Some sources would relate to multiple activities. For example, viewing may related to different activities/resources within a course. ??? IS IT BEST TO COMBINE THESE OR SHOULD THEY BE SEPARATELY STORED ANALYTICS ??? BIG DECISION: GRANULARITY VS SIMPLICITY ??? IF SEPARATED, EACH LA SOURCE WOULD NEED ITS OWN CONFIGURATION ???
Data structures
To calculate status of individual students, learning analytics values are combined as a weighted sum, according to a configuration, and stored as status values. These status values can be used by numerous interfaces via requests to the Learning Analytics API.
Learning Analytics
- User ID
- Context ID
- Source
- Timestamp
- Value {0..1}
Learning Analytics are gathered and/or pre-calculated pieces of information from LA sources. They relate to an individual user's (student's) success as reported by the source. Each calculated learning analytics value is stored so they can be accessed later. Usually only the latest values will be used.
Configuration
- Context ID
- Source
- Weight
- Revision
The configuration defines how learning analytics should be combined within a particular context (course). Each source is given a weight in the configuration and the weighted sum is calculated to form the student status.
Configurations can be validated when compared to final outcomes, such as course completion or course grade.
Configurations can be saved as revisions so that a series of status values can be recalculated later and so that the validity of different configurations can be compared.
Student Status
- User ID
- Context ID
- Status value {0..1}
- Status code {'At risk','Normal','Ahead'}
- Timestamp
Student status values are a constantly available for each user (student) in each relevant context (their enrolled courses). This information is available to various interfaces throughout Moodle.
Status values are the weighted sum of learning analytics values, calculated based on the configuration within the context (course). Status values relate to status codes, which denote whether the user (student) is at risk, making normal progress or is ahead within the context (course).
Student status values would need to be recalculated when there is an event within the context that would effect one of the learning analytics relating to the student; such recalculations would prompt each relevant learning analytics value to be fetched (which may require data gathering) and a new combination of these values based on the configuration. Only the latest status value needs to be kept. If old status values need to be recalculated later, the calculations can be redone based on configuration history and the record of past learning analytics values.
Interfaces
XXX
Configuration Interface (narrowest audience)
- metrics, including notifications sent
- criteria, when, any activity, any quiz, normative vs absolute
- weight
- name
- validity
- Would want to have global metrics (set by admins) and course metrics (additional metrics created by teachers)
List of students & status (teachers, advisors, financial aid… )
- track notifications sent
- Graphic, coloured icons
- Relative performance
Detail view per student
- Graphic, coloured icons
- Relative performance
- Would students see this? Maybe not by default? Perhaps with a capability.
Push notifications (widest audience)
- messaging
- based on trigger or manual
- audience templates
- target of message could be student, teacher, mentor or other user
(A useful interface mockup tool is Balsamiq)
How it works
XXX
Learning analytics source value calculations
Status caclulation
Thresholds for status codes
Benefits
- XXX
Usage scenarios
- XXX
Future work
XXX
Other potential extensions to the system
- Reporting of resource/activity effectiveness in contributing to learning outcomes
- Potential for students to opt in/out of data collection, analysis, reporting or exports. This may be necessary in some places. It might be possible to have student logging be anonymous or exported anonymously.
- Ability to export data as a static dump or as a stream of analytics information, possibly conforming to the IMS Caliper standard, for extra-system analysis.
- Third-party plugins that use the Learning Analytics API
Other potential sources of Learning Analytics data
There are other possible sources of Learning Analytics. As well as those suggested above, here are some additional potential sources.
- performance in other courses enrolled (now & previously)
- student ratings on forums
- self-evaluation
- history of activity within the current course
- history of course activity within a faculty/category
- mood/sense-making
- activity in relation to deadlines
- time spent online based on activity captured through JavaScript
- demographic/profile data
- year/progress
- past grades, prior credit, GPA
- completion of prerequisites vs recommended prerequisites
- personal background
- native language, exceptionalities / accommodations
- cross-instance performance across multiple Moodle instances
Other potential interfaces
- An institutional summary view showing an overview of student retention in courses
- Activity restrictions based on risk status based on learning analytics
- A block showing a student their current status and linking to their details, showing teachers a list of students (with at risk status?)
- A profile page element showing the student's status
