Perth Hackfest October 2012/Logging spec

From MoodleDocs
Jump to: navigation, search

People present: Andrew Davis, Shane Elliott, Justin Filip, Loic Jeannin, Rex Lorenzo, Dan Marsden, Matt Sharpe, Kris Stokking

Introduction/Problems

  • Performance concern/bottleneck
    • writes are slow when table is huge
    • mdl_log uses 90% of the db storage on big installations
  • We aren’t doing enough of it
    • cannot do analytics
  • Some Moodle components depend on mdl_log table, when they shouldn’t be
  • Data consumed live is treated the same as data needed by reports and later analysis

Requirements

  • Different levels of logging:
    • Critical, warning, notice, developer.
    • exceptions should be handled on separate path to avoid deadlocking/looping, but logged either way.
    • For default moodle logging, should default to almost current behavior
  • Different kind of log data
    • User action (CRUD), system action, moodle exceptions
    • Usages of log data:
      • Data analytics
      • Debugging
      • Forensics
  • Potential long-term requirement: Ability to generate automated test from log data? =)
  • Pluggable backends for storing log data
    • Default plugin would be logging into mdl_log
      • Must not be worse performance than existing add_to_log() call
    • Multiple backends at the same time
    • If the plugin needs to send data back to Moodle, it is up to the plugin to implement that
    • Each plugin can configure its own level of information to capture
    • External logging solution could store log data in another system and then ship data back into mdl_log. Could save on the number of consecutive DB writes.
  • Dependency problems:
    • live logs - might need to be “realtime”
    • recent activity - (probably) shouldn’t depend on mdl_log table. Should tie into events api and store its own data.
    • conditional access - uses mdl_log to check access, should use something else ( event api when resource/activity is accessed?)
  • auth/mnet/auth.php - mnet is going away anyways -- could also be easily re-written
  • Exceptions:
    • Not logged by default, but an option. Exceptions can impact user behaviour so for forensics or research it must be possible to have them logged.
  • Deprecating add_to_log
    • places in core code will be changed to events
    • have add_to_log still exist, but trigger event instead (deprecatedlib.php)
  • Backwards compatibility
    • Default settings would keep current behavior in terms of frequency
    • May need to add more columns to mdl_log

User stories/Use cases

Data can be shipped to another server
As a administrator I want to offload those data from the live system So that moodle runs faster (Flames !)
/dev/null logging backend
As a developer I want to turn off logging So that when I do development or run unit tests I don’t have lag or add load on the event api/logging server
Notes: should only be configurable in config.php
Using the event api to log data
Personal tools
User docs (English)