Perth Hackfest October 2012/Logging User activity / Reporting / Analytics

Why is Logging an important topic?

• Not much of new Moodle development is based on measured data. We need to
  • Prove there is a problem
  • Prove that a given change is an improvement.
• Data comes from logging
• Better data would allow researchers to study what happens in online teaching.
• Better data would give admins more feedback on how to run their site (both technically and process)
• Better data would give teachers better feedback to improve their teaching process.
• Better data would give students better feedback to improve the learning process.
• Note we can't predict all the analysis

Problems

• Logging is added adhoc by developers, has spotty coverage. No admin logging!
• Log tables are joined in many popular queries and are slow.
• No archiving, so have to delete old logs
• Deletion is wholesale, no control.
• No synchronization with server logs to help debugging performance issues
• No storage of traces when there are problems, so we can debug Moodle issues

Parts of Moodle that aren’t being logged, that should have logs

• Errors and warnings
  • People report errors, but cannot duplicate it.
  • Logging history with stack trace would be very useful
  • Also dump session variables and all other data that can be useful for debugging
• Admin activities
• Micro activities in a page (more than one per load)
• AJAX calls (eg on course page)
• Should be careful about logging confidential information (e.g. do not log any POST data)
• Shouldn’t delete logs when course is deleted
• All events should all be logged
  • Add more events (they are cheap and very useful)
  • Need more standardisation of event data for all triggered events
  • Add a catch-all handler that pushes events to the logging system
• --> Action logging improvements META - MDL-28443

Users of log table at the moment

A lot of the time-critical uses of the log table can be moved to new tables specifically for this purpose. It will speed things up a lot.

• recent activity (block, myMoodle) EVENTS -> NEW TABLE
• conditional access EVENTS -> NEW TABLE
• failed logins EVENTS -> NEW TABLE
• files via clamav EVENTS -> NEW TABLE

Other things can can continue to query the main logs:

• course activity report, participation report, user outline report
• stats
• cron
• automated backup locking/heartbeats (probably)
• auth/mnet/auth.php
• backup/restore (target logs and own logging)
• grade history (currently separate log)

And some future things that will need the full log:

• Engagement reporting

Basic approach

We can't actually predict all the use cases so we must think generically and plan for worst cases.

• Make logging calls as cheap as possible
• Use MUC-like plug-ins to determine where to put logs permanently (NoSQL, SAS, file...). Default will be to the current log table.
• Log everything we possibly can think of.
• Define log level settings (on each logging call) so different sites can choose what they want logged and so control size, speed etc
• add_to_log retained for backward compatibility, make it a wrapper for new logger function.
• provide hooks to expose logs everywhere. For example, each page should have a link that shows logs and stats for that page.

Moodle -> Logs

• MUC-like plugins to interface between our logging calls and log backends
  • File
  • logsstash
  • fluentd
  • memcached?
  • http://logging.apache.org/log4php/ ?

Logs -> Moodle

• Each plugin implements some methods to query and extract logs back to Moodle
• Do we auto-sync anything back to Moodle tables for convenience?

See also

Examples of data-based learning technology development:

• http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8535.2008.00928.x/abstract
• http://www.sciencedirect.com/science/article/pii/S0360131510000461 -> qtype_pmatch
• http://www.tandfonline.com/doi/abs/10.1080/02680513.2011.567754
• http://oro.open.ac.uk/24619/