Learning Analytics Enriched Rubric
The Learning Analytics Enriched Rubric (LA e-Rubric) is an advanced grading method used for criteria-based assessment. As a rubric, it consists of a set of criteria. For each criterion, several descriptive levels are provided. A numerical grade is assigned to each of these levels.
An enriched rubric contains some criteria and related grading levels that are associated to data from the analysis of learners’ interaction and learning behavior in a Moodle course, such as number of post messages, times of accessing learning material, assignments grades and so on.
Using learning analytics from log data that concern collaborative interactions, past grading performance and inquiries of course resources, the LA e-Rubric can automatically calculate the score of the various levels per criterion. The total rubric score is calculated as a sum of the scores per each criterion.
You may click this link to view the Introduction to Learning Analytics Enriched Rubric video.
Creating a new Learning Analytics Enriched Rubric
Selecting a Learning Analytics Enriched Rubric
There are two ways a user can choose a LA e-Rubric as an advanced grading method.
- Make the selection during the creation of an assignment, in the Grade section of the creation form.
- Click Advanced grading in the settings block of the assignment and then make the selection from the Change active grading method to select form field.
Editing a Learning Analytics Enriched Rubric
In the Advanced grading page of the assignment, the user can
- Define a new grading form from scratch or,
- Create a new grading form from a template or,
- Edit a current form definition
Either way, the grading form editor page appears where the LA e-Rubric can be created or edited.
In that form, the user provides a name for the LA e-Rubric, an optional description, adds or edits the criteria and chooses the options meeting his requirements.
Then the LA e-Rubric can be saved as a draft (for further editing), or saved and made ready for use.
Adding or editing criteria in a Learning Analytics Enriched Rubric
In order to add or edit a criterion, the user can:
- Add or edit the criterion description.
- Add or edit the level description and points values.
- Add or edit the enrichment criterion type (collaboration-grade-study).
- Add or edit the enrichment collaboration type (simple occurrences, file submissions, forum replies, people interacted), in case collaboration is chosen as the criterion type.
- Add or delete the corresponding course modules according to criterion type, from which data mining is conducted.
- Add or edit the operator used for enrichment calculations between the enrichment benchmark found and level enrichment check values (equal-more than). This defines if discrete or continuous range values are checked for comparison operations.
- Add or edit the checking scope of calculations according to one student or all (student-students).
- Add or edit the level enrichment check values needed for setting the check points in comparison operations.
Before adding or editing the above form fields the user should consider the following:
- A criterion type must be selected first in order for all other enrichment fields to be edited.
- All enrichment criteria and level values must be edited in case of enrichment.
- To keep a rubric criterion simple, leave enrichment fields blank. Criteria enrichment is not mandatory!
- The criterion type defines the kind of course modules that will be included in the enrichment.
- In case of collaboration check, the collaboration type field is available and mandatory.
- The collaboration type defines what kind of checking will be made from the course modules.
- Collaboration type simple occurrences checks simple add post and talk logs in the selected course modules.
- Collaboration type file submissions checks file uploads ONLY for forum course modules.
- Collaboration type forum replies checks user(s) replies to posts ONLY for forum course modules.
- Collaboration type people interacted checks the number of classmates a student has interacted in the selected course modules.
- One or more course modules of the selected criterion type and of the particular course should be added.
- The criterion operator is used for calculating logical conditions associated with the levels enrichment values.
- Related to All(students) or One(student) defines whether the calculations will be according to the student evaluated, thus absolute values will be processed or according to all other students, thus percentages will be processed.
- If the relation of the criterion is according to percentage, the arithmetic mean of all other students from all selected course modules will be considered as the students benchmark.
- Level enrichment values should be ascending (or descending) according to level ordering, otherwise logical errors may occur during evaluation.
Checking options of a Learning Analytics Enriched Rubric
The following options can be checked while editing a LA e-Rubric:
Rubric options
- Sort order for levels
Sort level viewing according to grade points ascending or descending.
Important: ordering of levels is taken into account in enrichment in order to pick the appropriate level according to enrichment check points. For example if level grade values are 0 – 10 – 20 – 30, enrichment check points should be ascending accordingly, for instance 5 – 6 – 7 – 8. Using this example, if the enrichment operator is more than (>=), the enrichment benchmark is calculated to 9 and the enrichment check points are 5 – 6 – 8 – 7, then 7 will be picked as opposed to 8! - Allow users to preview rubric used in the module (otherwise rubric will only become visible after grading)
Checking this option, provides the student the ability to preview the LA e-Rubric before the student submits his assignment or been graded. - Display rubric description during evaluation
- Display rubric description to those being graded
- Display points for each level to those being graded
- Display points for each level during evaluation
- Allow grader to add text remarks for each criteria
- Show remarks to those being graded
Enriched criteria options
- Display enrichment check points for each level to those being graded
- Display enrichment check points for each level during evaluation
- Display enrichment of criteria to those being graded
Un-check this option to hide enrichment of rubric criteria. - Display enrichment of criteria during evaluation
Un-check this option to hide enrichment of rubric criteria. - Override automatic criterion evaluation in case of enrichment logical error (If enrichment logical error exists, evaluation is not possible without overriding it!)
Check this option to enable the evaluator to pick a level according to his own judgment in case an enrichment benchmark is not found or there is a logical error in the enrichment criteria and a level can't be automatically picked. - Enrichment calculations are conducted from assignment available date (if enabled)
If an availability date is defined for the assignment, check this option to time stamp enrichment calculations on data mining. - Enrichment calculations are conducted until submission due date (if enabled)
If an due date is defined for the assignment, check this option to time stamp enrichment calculations on data mining. - Display calculated enrichment benchmark to those being graded
- Display calculated enrichment benchmark during evaluation
Saving and Previewing a Learning Analytics Enriched Rubric
The user can save this form as a draft for further checking or save and make it ready to be used immediately. Either way, afterwards the user can preview the LA e-Rubric form as is was created or edited.
Video tutorial
You may click this link to view the tutorial on how to Create Criteria in Learning Analytics Enriched Rubric video.
Using a Learning Analytics Enriched Rubric to evaluate students
The grading process is where the Learning Analytics Enriched Rubric performs its magic. Analysis of data from log files is performed in order that all enriched criteria can automatically be evaluated and the corresponding criterion level gets a value. The evaluator can provide optional remarks, and just click ‘save’ or ‘save and grade next’, in order to grade a student.
First, the user clicks on View/grade all submissions in the assignment view page, or in the assignment's settings box. In the grading page of the assignment the user clicks on the grade icon, or chooses Grade from the editing icon in the edit column on the left.
Inside the evaluation form, the user sees all enriched criteria with the enrichment benchmark displayed and the appropriate level chosen for each one. If the enrichment evaluation procedure succeeded, in each criterion the user can see the checking icon of the enriched level whose value corresponds to the benchmark according to enrichment.
Handling enrichment evaluation failure
If the enrichment evaluation failed for an enriched criterion, the evaluator can pick a level according to his own judgment ONLY IF Override automatic criterion evaluation is enabled from the LA e-Rubric options. If there is a failure on enrichment evaluation and the evaluator can't pick a level himself, student evaluation won't be possible because all criteria will not have a level checked. In such cases it is strongly recommended to check the enrichment criteria again to avoid these errors, rather than override the enrichment evaluation procedure.
Evaluation according to student
If criterion enrichment evaluation is conducted according to student values, the student's benchmark appears upon succession in order for the evaluator to get the exact view of student performance.
Evaluation according to global scope
If the enrichment evaluation is conducted according to all students participating, then upon successful findings, the evaluator views two benchmarks. One that represents all of the participating students average score, and another for the score of the student currently evaluated. Again this is done in order for the evaluator to gain a better scope of student performance in reference to all participating students (including him).
Something very important about global scope evaluation is that only students actively participating in the selected course modules of the enrichment are accounted for, which means that they may be, less than all students enrolled in a course. This is done for 2 reasons:
- Because the LA e-Rubric performs qualitative evaluation to students according to those participating, not to all. We want to measure true participation and collaboration results that concern active students only.
- Another equally important reason could be explained with an example: Let's say that we have 20 students attending a course and we want to evaluate them according to how much they collaborated the past week. Let's also say that 5 of them were sick the past week, or could not attend. It wouldn't be right for 5 students missing to bring down the hole week's average.
The above estimates are effective only for checking collaboration! For checking grades and studying, all enrolled students are accounted in the process.
Video tutorial
You may click this link to view the tutorial on how to perform Student Evaluation in Learning Analytics Enriched Rubric video.
How students view the Learning Analytics Enriched Rubric
Preview of a Learning Analytics Enriched Rubric
If the corresponding option is engaged, students can preview the LA e-Rubric before they are graded. This is an excellent method to let students know how they are evaluated and get a better view of their evaluation criteria.
In order for the students to preview the LA e-Rubric they just click Submissions grading in the submenu of their assignment on the left.
View grading results produced by a Learning Analytics Enriched Rubric
After graded, students can view how their evaluation occurred and they can also view their own benchmarks according to the LA e-Rubric criteria that affected their evaluation outcome.
The LA e-Rubric elements displayed to students, are defined in the LA e-Rubric options.
Students view their completed LA e-Rubric when the visit their corresponding assignment page.
Video tutorial
You may click this link to view the tutorial on how students View Evaluation Results in Learning Analytics Enriched Rubric video.
Backup & restore, template sharing and importing a Learning Analytics Enriched Rubric
Procedures concerning backup, restore, import or template sharing are carried out according to all advanced grading methods of Moodle.
However, regarding the LA e-Rubric there are some restrictions.
The LA e-Rubric uses specific and resident course modules belonging to the moodle course in which the assignment is created. Thus when a LA e-Rubric is restored or imported or shared in another course, the particular course modules won't exist. The structure of the entire LA e-Rubric stays intact, but the user has to replace the missing course modules with similar ones obtained by the new course.
When an entire course is restored, the expected scenario is that most course modules have been given a new id, thus this restriction may still be in effect. Again, the structure of the entire LA e-Rubric stays intact, but the user has to replace the missing course modules with the same ones obtained by the restored course, in order to update the course modules ids.
During the sharing procedure of a LA e-Rubric, user gets an information message concerning this restriction.
If the user imports or restores or uses a LA e-Rubric from another course, another message appears informing the user about the course modules missing from the enriched criteria and advises him to make the appropriate changes so that the LA e-Rubric may be operational.
Images below display all these messages.
Grade calculation and Data mining for enrichment in a Learning Analytics Enriched Rubric
Grade calculation
Grade calculation is done the same way as in simple rubrics. For more information check Grade calculation.
Learning Analytics for enriching the grading method
The data acquired during the log file analysis are distinguished according to analysis indicators as presented in these cases below.
- For simple occurrences in collaboration, moodle log data concern forum add posts and chat talks.
- For file submissions in collaboration, the number of attachments attached to forum post messages.
- For forum replies in collaboration, forum reply post messages are counted (not including the replies one has made to himself).
- For people interacted, forum post and chat messages data are measured.
- For checking study behavior, the number of students' views upon selected course recourses are taken into account.
- For checking grades, moodle grading scores on selected assignments are processed.
General advices – instructions
- First create all course resources and activities and then generate a LA e-Rubric.
- Create enrichment criteria carefully and thoroughly to avoid logical errors.
- Don’t delete course resources or activities used in a LA e-Rubric.
- Log data are needed for evaluation so don't purge or empty moodle data logs.
Future improvements
Future improvements may be done in order to:
- Visualize Learning Analytics during evaluation for each criterion.
- Export evaluation outcomes to various formats.
- Import LA e-Rubrics from known rubric creation tools.
See also
- [Rubric description[ at Wikipedia
- [Learning Analytics] at Wikipedia
- Advanced grading methods page for general concepts of advanced grading in Moodle
- [Enriched assessment rubrics: a new medium for enabling teachers to easily assess student's performance when participating in complex interactive learning scenarios] written by O. Petropoulou, M. Vassilikopoulou and Symeon Retalis published in Operational Research journal
- [Evaluation in e-Learning] written by Y. Psaromiligkos, A. Spyridakos and S. Retalis
- [Student evaluation in collaborative educational environments through enriched rubrics] Greek article written by O. Petropoulou, M. Vassilikopoulou and Symeon Retalis, published in Themes in Science and Technology Education journal
- [CosyLab channel on Youtube]
