Development:Scheduled Tasks Proposal: Difference between revisions
Penny Leach (talk | contribs) No edit summary |
Penny Leach (talk | contribs) |
||
Line 24: | Line 24: | ||
== Approach == | == Approach == | ||
=== Plugin cron registration === | |||
Each plugin will be able to provide a db/tasks.php (alongside access.php and events.php etc) that lists all the cronjobs that it wants to have run. This will look something like the following: | Each plugin will be able to provide a db/tasks.php (alongside access.php and events.php etc) that lists all the cronjobs that it wants to have run. This will look something like the following: | ||
Line 55: | Line 57: | ||
A field may be an asterisk (*), which always stands for ``first-last''. | A field may be an asterisk (*), which always stands for ``first-last''. | ||
Ranges of numbers are allowed. Ranges are two numbers separated with a hyphen. The specified range is inclusive. For example, 8-11 for an ``hours'' entry specifies | Ranges of numbers are allowed. Ranges are two numbers separated with a hyphen. The specified range is inclusive. For example, 8-11 for an ``hours'' entry specifies | ||
execution at hours 8, 9, 10 and 11. | execution at hours 8, 9, 10 and 11. | ||
Lists are allowed. A list is a set of numbers (or ranges) separated by commas. Examples: ``1,2,5,9'', ``0-4,8-12''. | Lists are allowed. A list is a set of numbers (or ranges) separated by commas. Examples: ``1,2,5,9'', ``0-4,8-12''. | ||
Step values can be used in conjunction with ranges. Following a range with ``/<number>'' specifies skips of the number's value through the range. For example, | Step values can be used in conjunction with ranges. Following a range with ``/<number>'' specifies skips of the number's value through the range. For example, | ||
``0-23/2'' can be used in the hours field to specify command execution every other hour (the alternative in the V7 standard is ``0,2,4,6,8,10,12,14,16,18,20,22''). Steps | ``0-23/2'' can be used in the hours field to specify command execution every other hour (the alternative in the V7 standard is ``0,2,4,6,8,10,12,14,16,18,20,22''). Steps |
Revision as of 04:04, 5 January 2010
Template:Moodle 2.0== Introduction ==
This proposal is meant both to provide a replacement for the moodle cron job, and provide a means to schedule once off tasks to be run outside of the user's request lifecycle.
Terminology
- *Subtask* an individual piece of cron processing that should be run (equivalent to forum_cron now, or maybe even smaller)
- *Moodle cron instance* a cron.php process
Rationale
The moodle cronjob currently delegates all scheduling to each subtask that is run - for example, the forum cron is responsible for checking when it last run, and making decisions about whether or not it should be run again. This sort of decision process should be centralised, and individual cron subtasks should be called by the central controller.
Additionally, there is not any central locking of subtasks. At the moment, some subtasks that expect that they might take a long time to run implement their own locking (for example statistics), but it's not centralised. Each moodle cron instance runs to completion, no matter how long it takes, and it processes tasks in the order that they're programmed, regardless of if there are any other moodle cron instances running, that might be processing sub tasks in parallel
Finally, we need to be able to run non-related tasks in parallel so that the entire moodle queue isn't held up by single long running jobs.
Goals
- Centralised locking for *all* tasks
- A way consistent for *all plugin types* to register with Moodle (at installation/upgrade) when they want their jobs run
- More sophisticated scheduling rather than just intervals in seconds (eg every sunday at 11pm or similar) based on unix cron
- An administration screen in Moodle to allow site administrators to adjust the scheduling of individual tasks
Approach
Plugin cron registration
Each plugin will be able to provide a db/tasks.php (alongside access.php and events.php etc) that lists all the cronjobs that it wants to have run. This will look something like the following:
<?php
$tasks = array(
array(
'function' => 'yourmodule_cron_somedescription',
'minute' => '*',
'hour' => '*',
'day' => '*',
'month' => '*',
'dayofweek' => '*',
'description' => 'langstringkey', // this must correspond to get_string('langstringkey', 'yourmodule');
),
);
The fields are the same as normal unix cron, with the exception that you cannot use 3 letter words for the month and day of week fields like you can for unix cron. The following is straight from the unix manpage about cron:
field allowed values ----- -------------- minute 0-59 hour 0-23 day of month 1-31 month 1-12 (or names, see below) day of week 0-7 (0 or 7 is Sun, or use names)
A field may be an asterisk (*), which always stands for ``first-last. Ranges of numbers are allowed. Ranges are two numbers separated with a hyphen. The specified range is inclusive. For example, 8-11 for an ``hours entry specifies execution at hours 8, 9, 10 and 11. Lists are allowed. A list is a set of numbers (or ranges) separated by commas. Examples: ``1,2,5,9, ``0-4,8-12. Step values can be used in conjunction with ranges. Following a range with ``/<number> specifies skips of the number's value through the range. For example, ``0-23/2 can be used in the hours field to specify command execution every other hour (the alternative in the V7 standard is ``0,2,4,6,8,10,12,14,16,18,20,22). Steps are also permitted after an asterisk, so if you want to say ``every two hours, just use ``*/2.
Database
Module cron registration
Locking
Unresolved issues/ideas
- Do we need to allow for scheduling different subtasks on different servers?
- We need to find a way to separate subtasks by some logic so that subtasks that write to the same areas of the database never run at the same time. We could do this by getting each cron job to say what areas of Moodle they write to, but this is problematic.
- We also have to deal with the order of some subtasks - we could maybe do this by introducing dependencies
- When the first cron in a long time is running, we should lock the entire cron and let it run to completeness, because the order is really important then.
Psuedo code proposal
Moved to the talk page
Audit of current cron
Main section | Subtask | Frequency | Notes |
---|---|---|---|
session_gc | every run | ||
mod/assignment | plugins (none) | every minute | |
mod/assignment | message submissions | every minute | checks last run time |
mod/chat | update chat times | every five minutes | |
mod/chat | update_events | every five minutes | |
mod/chat | delete old chat_users and add quits | every five minutes | |
mod/chat | delete old messages | every five minutes | |
mod/data | every minute | no _cron function (includes file unnecessarily) | |
mod/forum | mail posts | every minute | checks last run time |
mod/forum | digest processing | every minute | |
mod/forum | delete old read tracking | every minute | |
mod/scorm | reparse all scorms | every five minutes | does hourly checking |
mod/wiki | delete expired locks | every hour | |
blocks/rss_client | update feeds | every five minutes | |
quiz/report/statistics | delete old statistics | every run | |
admin/reports | none | every run | |
language_cache | every run | ||
remove expired enrolments | every run | ||
main gradebook | lock pending grades (*2) | every run | |
main gradebook | clean old grade history | every run | has a TODO to not process as often |
event queue | every run | potentially large | |
portfolio cron | clean expired exports | every run | potentially large |
longtimenosee | 20% | ||
deleteunconfirmedusers | 20% | ||
deleteincompleteusers | 20% | ||
deleteoldlogs | 20% | ||
deletefiltercache | 20% | ||
notifyloginfailures | 20% | ||
metacourse syncing | 20% | ||
createpasswordemails | 20% | ||
tag cron | 20% | ||
clean contexts | 20% | ||
gc_cache_flags | 20% | ||
build_context_path | 20% | ||
scheduled backups | daily (admin defined) | ||
make rss feeds | every run | ||
auth/mnet | keepalives | every run | |
auth/mnet | delete old sessions | every run | |
auth/ldap | sync users | custom | not scheduled (external cronjob) |
auth/cas | sync users | custom | not scheduled (external cronjob) |
auth/db | sync users | custom | not scheduled (external cronjob) |
enrol/authorize | clears old data | daily (admin defined) | |
enrol/authorize | notifies administrators of old data | daily (admin defined) | |
enrol/authorize | process orders & email teachers | every run | |
enrol/flatfile | read file and sync users | every run | !?! |
enrol/imsenterprise | read file and sync users | every run | !?!?! |
enrol/manual | notify people of pending unenrolments | daily | |
statistics | daily (admin defined) | huge | |
grade/import | none | every run | |
grade/export | none | every run | |
grade/reports | none | every run | |
fetch blog entries | every run | ||
file gc | (optional) daily, else every run? | ||
local cron |