Perth Hackfest October 2012/Logging spec

People present: Shane Elliott, Justin Filip, Loic Jeannin, Rex Lorenzo, Dan Marsden, Matt Sharpe

Introduction/Problems

Performance concern/bottleneck
- writes are slow when table is huge
- mdl_log uses 90% of the db storage on big installations
We aren’t doing enough of it
- cannot do analytics
Some Moodle components depend on mdl_log table, when they shouldn’t be
Data consumed live is treated the same as data needed by reports and later analysis

Different levels of logging:
- Critical, warning, notice, developer.
- exceptions should be handled on separate path to avoid deadlocking/looping, but logged either way.
- For default moodle logging, should default to almost current behavior
Different kind of log data
- User action (CRUD), system action, moodle exceptions
- Usages of log data:
  - Data analytics
  - Debugging
  - Forensics

Potential long-term requirement: Ability to generate automated test from log data? =)

Pluggable backends for storing log data
- Default plugin would be logging into mdl_log
  - Must not be worse performance than existing add_to_log() call
- Multiple backends at the same time
- If the plugin needs to send data back to Moodle, it is up to the plugin to implement that
- Each plugin can configure its own level of information to capture
- External logging solution could store log data in another system and then ship data back into mdl_log. Could save on the number of consecutive DB writes.
Dependency problems:
- live logs - might need to be “realtime”
- recent activity - (probably) shouldn’t depend on mdl_log table. Should tie into events api and store its own data.
- conditional access - uses mdl_log to check access, should use something else ( event api when resource/activity is accessed?)

auth/mnet/auth.php - mnet is going away anyways -- could also be easily re-written

Exceptions:
- Not logged by default, but an option. Exceptions can impact user behaviour so for forensics or research it must be possible to have them logged.

Deprecating add_to_log
- places in core code will be changed to events
- have add_to_log still exist, but trigger event instead (deprecatedlib.php)
Backwards compatibility
- Default settings would keep current behavior in terms of frequency
- May need to add more columns to mdl_log

Data can be shipped to another server: As a administrator I want to offload those data from the live system So that moodle runs faster (Flames !)

/dev/null logging backend: As a developer I want to turn off logging So that when I do development or run unit tests I don’t have lag or add load on the event api/logging server

Notes: should only be configurable in config.php