Logging 2

Note: This page is a work-in-progress. Feedback and suggested improvements are welcome. Please join the discussion on moodle.org or use the page comments.

Logging stage 2
Project state	In very early specification
Tracker issue	MDL-37658
Discussion
Assignee	moodle.com DEV team

Mandate

To rewrite the logging API in Moodle to:

• Capture richer information from plugins/core about actions.
• Cover “all” of Moodle.
• Provide control over how much is logged.
• Improve efficiency by separating writing and reading so it can scale.
• Allow simpler control over how much logging history is kept.
• Support research, reporting and analytics.
• Support personal timelines.

Why is Logging an important topic?

Logging is fundamental to research, reporting and analytics.

• Not much of new Moodle development is based on measured data. We need to
  • Prove there is a problem
  • Prove that a given change is an improvement.
• Data comes from logging
• Better data would allow researchers to study what happens in online teaching.
• Better data would give admins more feedback on how to run their site (both technically and process)
• Better data would give teachers better feedback to improve their teaching process.
• Better data would give students better feedback to improve the learning process.
• We can't predict all the analysis that might happen in future.

Examples of possible developments that would require better logs

Here are some random ideas, please add more!

• Timely notifications of critical events (defined by users), eg more than five people posted in the same forum within an hour.
• "Red circle" notification badge counter icons everywhere in Moodle highlighting things that need attention for each user.
• Colour heatmaps overlaid on course page showing recent usage patterns on a course
• Visualisation of live activity across a whole site (for admins)
• Live engagement analytics that take every little action into account
• Identification of students at risk based on complete history

Existing Problems

• Logging is added adhoc by developers, has spotty coverage. No admin logging!
• Some logged actions do not contain necessary information.
• Performance and scalability problem.
  • Log tables are joined in many popular queries and are slow.
  • All logs go to database and it's not scalable.
  • It is not possible to configure level of logging
  • Large number of database writes even when carrying out ‘read-only’ requests.
• No archiving, so have to delete old logs.
• Deletion is wholesale, no control.
• No synchronization with server logs to help debugging performance issues
• No storage of traces when there are problems, so we can debug Moodle issues

Existing log usage analysis

See Logging_usage

Places where we need more logs

• Errors and warnings
  • People report errors, but cannot duplicate it.
  • Logging history with stack trace would be very useful
  • Also dump session variables and all other data that can be useful for debugging
• Admin activities
• Micro activities in a page (more than one per load)
• AJAX calls (eg on course page)
• Should be careful about logging confidential information (e.g. do not log any POST data)
• Shouldn’t delete logs when course is deleted
• All events should all be logged
  • Add more events (they are cheap and very useful)
  • Need more standardisation of event data for all triggered events
  • Add a catch-all handler that pushes events to the logging system
• --> Action logging improvements META - MDL-28443
• Logging of long events (such as ldap synchronization) should have
  • one unique identifier per event (like for example postfix mail server) to help admin monitor the cron processes
  • at least 2 log entries (begining and end) for each event, possibly more

Logging Proposal

This section is deliberately not called "Logging API" because Moodle does not provide any logging API. All plugins that perform logging and analyse logs to display reports can do it as they wish.

• In places where we previously added information to 'the' log we must generate the new event.
• The standard Moodle distribution will include simple DB logging plugin storing the data in new format. There will also be an section (optional) legacy logging plugin that will store data in the old format in {log} table. All standard reports will be upgraded to use the new logging system that will use a logging retrieval plugin to access log information. Custom report plugins will be able to query the {log} table, but should ideally shift to the new logging system over time, so that organisations are not forced to continue double logging.

Example scenario

1. Student submits an assignment.
2. Event is triggered by assignment module.
3. The logging handler (plugin) observes the event and determines if it is log-worthy.
4. The logging handler decides which plugin(s) will store the log.
5. Logging storage plugin(s) stores the information about the submission (on disc, in DB or wherever).
6. Teacher requests report of student submissions.
7. Report requests user submission counts from the logging access API (plugin).
8. The logging API decides which logging retrieval plugin can supply the submission counts and requests it.
9. The chosen logging retrieval plugin responds to the logging API with submission count.
10. The logging API returns the counts.
11. Report receives counts and displays count.

From event to report

There are four steps of how event is converted into report:

1. Handling of events and filtering what needs to be logged.
2. Storing the events data in the log storage (DB, filesystem, etc.).
3. Retrieving the data from log storage - each plugin implements some methods to query and extract logs back to Moodle.
4. Displaying the data in the report.

One plugin can cover one or several steps. It is also possible to create a report that has built-in logging and listens to events (all 4 steps). Also external logging systems do not need to care about steps 3 and 4 at all.

Moodle standard distribution provides a suggestion on logging-report chain that can be followed by 3rd party plugins and may be not.

Logging plugins relation

Logging plugins sequence diagram

Standard logging plugins

Standard Moodle 2.6 distribution will include 3 plugins:

Event logging handling plugin (tool_eventobserver)

Responsible for step 1 from above. Has functions pluginname_register_log_instance() and pluginname_unregister_log_instance() that allow to add/remove log storage instance and also allows admin to configure what kind of events to store for each log instance.

Event logging DB storage plugin (tool_logdbstorage)

Responsible for step 2 from above. Has function pluginname_get_log_instances(). Allows admin to create storage instances and assign event handling plugin to fill each of them

Event logging driver plugin (tool_logdbdriver)

Responsible for step 3 from above. This plugin can only work with tool_logdbstorage. It provides functions to access data in log.

Mockups

Log storage settings page

Log storage settings page

Log storage Instance configuration page

Log storage instance configuration page

File log instance configuration page

File Log instance configuration page

Event observer initial configuration page

Event observer initial configuration page

Event observer individual logging device event management page

Event observer individual logging device event management page

Summary

We can't actually predict all the use cases so we must think generically and plan for worst cases.

• Things that don't need to use logs should not use logs. (Recent activity, etc)
• Make logging calls as cheap as possible
• Use Events API for the originating calls to create lots of hooks for other things
• Use MUC-like plug-ins to determine where to put logs permanently (NoSQL, SAS, file...). Default will be to the current log table.
• Log everything we possibly can think of.
• Define log level settings (on each logging call) so different sites can choose what they want logged and so control size, speed etc
• add_to_log retained for backward compatibility, make it a wrapper for new logger function.
• provide hooks to expose logs everywhere. For example, each page should have a link that shows logs and stats for that page.

See also

• Event 2

Example logging systems:

• http://logstash.net/
• http://fluentd.org/
• http://logging.apache.org/log4php/

Examples of data-based learning technology development:

• http://onlinelibrary.wiley.com/doi/10.1111/j.1467-8535.2008.00928.x/abstract
• http://www.sciencedirect.com/science/article/pii/S0360131510000461 -> qtype_pmatch
• http://www.tandfonline.com/doi/abs/10.1080/02680513.2011.567754
• http://oro.open.ac.uk/24619/