Workshop 2.0 specification

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Moodle 2.0

This page tracks and summarizes the progress of my attempt to try and rewrite the Workshop module for Moodle 2.0 (yes, yet another attempt).

PROJECT STATE: Planning

Introduction

Workshop module is an advanced Moodle activity designed for peer-assessments within a structured review/feedback/grading framework. It is generally agreed the Workshop module has a huge pedagogical potential and there is quite a big demand of the community (including some Moodle Partners) to have such module. It was originally written by Ray Kingdon, and was the very first third-party module written for Moodle. For a long time it's been largely un-maintained except for emergency fixes by various developers to keep it operational. During last years, several attempts were done by various volunteers to rewrite the module or to replace it with an alternative. At the moment it is removed from the HEAD branch and lies in the CONTRIB area. This document specifies the proposed way of revitalizing the Workshop module so it becomes the standard core module again in Moodle 2.0.

Key concepts

There are two key features making the Workshop module unique among other Moodle activies

• Advanced grading methods - contrary to the standard Assignment module, Workshop supports more structured forms of assessment/evaluation/grading. Teacher defines several aspects of the work, they are assessed separately and then aggregated. Examples include multi-criterion evaluation and rubrics.
• Peer-assessment - according to some psychology/education theories (for example the Bloom's taxonomy), the evaluation is one of the highest level of cognitive operations. It implicitly requires paying attention, understanding the concepts, critical thinking, knowledge applying and analysing of a subject being evaluated. Peer-assessment fits very well into the social constructionist model of learning. In Workshop, students not only create (which is the other non-trivial cognitive operation) and submit their own work. They also participate on the assessment of others' submissions, give them feedback and suggest a grade for them. The Workshop module provides some (semi)automatic ways how to measure the quality of the peer-assessment and calculates a "grading grade", ie. the grade for how well a student assessed her peers.

Basic scenario of the module usage

UML: Basic teacher's and student's use cases

The following scenario includes all supported workshop features and phases. Some of them are optional, eg. teacher may decide to disable the peer-assessment completely so the Workshop behaves similarly to the Assignment module with the benefit of multi-criteria evaluation.

1. Workshop setup - teacher prepares workshop assignment (eg. an essay, research paper etc.) and sets up various aspects of the activity instance. Especially, the grading strategy (see below) has to be chosen and the assessment elements (criteria) defined.
2. Examples from teacher (optional)
   1. teacher publishes example submissions - eg. examples of a good work and a poor work
   2. teacher publishes example assessments of the example submissions. Again, there may (and should, in fact) be an example of very well done evaluation/assessment and an example of a poor one.
   3. students can try and train the process of assessment of example submissions (this part to be discussed: how would the module compute the grading grade?)
3. Students work on their submissions - the typical result is a file (several files?) that can be submitted into the workshop (together with a comment maybe?). Students may upload draft version of their work before they decide it is a final submission (or there is a deadline for submissions). Other types of work are possible using the new Repository API - eg. students can prepare a view in Mahara, GoogleDoc, publish video at YouTube etc.
4. Self-assessment (optional) - after the work is submitted, the student is asked to assess her own work using the selected grading strategy evaluation form
5. Peer-assessment (optional) - the module randomly selects a given number of submissions to be reviewed/commented/evaluated/assessed by student.
6. Assessment by teacher (optional)
   1. teacher manually evaluates submissions using the selected grading strategy evaluation form
   2. teacher manually evaluates the quality of peer-assessments
7. Final grade aggregation - TODO: more research of the current code and behaviour is still needed in this area
   1. generally, the final grade consists of a grade for submission and grade for assessments
   2. every grading strategy defines how the final grade is computed

Current implementation problems

• Very old and unmaintained code - the vast majority of the code is the same as it was in Moodle 1.1. mforms are not used (accessibility). Missing modularity. No unit testing.
• Does not talk with the Gradebook - there are some patches around in forums fixing partial problems.
• User interface and usability
• Grades calculation is kind of dark magic - see this discussion. It is not easy to understand the way how Workshop calculates the grading grade even if you read the code that actually does the calculation. More than that, the whole algorithm depends on a set of strange constants without any explanation or reasoning of the value. The grade calculation must be clear and easy to understand to both teacher and student. Teachers must be always able to explain why students got their final grades. Therefore the plan is to get rid of the fog above the grades calculation, even if it breaks backward compatibility.
• Lack of custom scales support - see this discussion
• Phases overlapping, balancing, overall allocation issues - see this discussion

Project analysis

See the project analysis mindmap

Project schedule

Milestone 1

• Date: 15/05/2009
• Goal: The functional specification is in docs wiki and is reviewed and agreed by the community and Moodle HQ. The implementation plan is transferred into sub-tasks in the tracker.

Milestone 2

• Date: 15/07/2009
• Goal: All features implemented. The community is asked for the testing.

Milestone 3

• Date: 15/08/2009
• Goal: Major bugs fixed, upgrading from pre-2.0 versions works. The community is asked for the QA testing.

Milestone 4

• Date: 15/09/2009
• Goal: The module is moved from contrib back to the core.

User interaface mockups

• MDL-18688 mod_form.php
• MDL-18912 grading forms

Implementation plan

The structure of the code

Files, libraries, interfaces, classes, unit tests

File API integration

How Workshop will use the new File API.

Repository/portfolio API integration

If and how will be implemented.

Grading strategies

We are currently working on this part of specification, see - the Google doc

Pluggable submodules in mod/workshop/grading/xxxx/. Implement grading_strategy interface. Basically provide grading form and way of grade calculation. During the current phase, four strategies will be implemented (the names are to discuss)

See MDL-18912 for UI mockups of grading forms.

No grading

Just comment(s)

Number of errors

Currently called Error banded

Criterial rubrics

Merges the current Rubrics and Criterion strategy into a single one (basically, current Criterion is an one-dimensional Rubrics)

Out of scope

What are not goals of this project (may be discussed later for 2.1 etc.)

• No new/other grading strategies are to implement in the current phase
• No new Workshop features will be implemented (if possible)
• No group grading support

Other links and resources

• MDL-17827 Workshop upgrade/conversion from 1.9 to 2.0 (META)
• Yet another attempt to rewrite Workshop for 2.0 forum thread
• New Workshop Module forum thread
• the new workshop module forum thread
• a lot of interesting feature requests

Diagram used

• File:workshop20 uml usecases.dia

Credits

Many thanks to Stephan Rinke for his valuable comments and ideas.