Student projects/Email interface

Programmer: Peter Boswood

(This is very much still a work in progress, data here may be inaccurate or outdated!)

Summary

Moodle uses email in order to communicate with end users in a variety of manners - for authentication, subscriptions in the Forum module, etc. In some instances, rather than simply ignoring replies from users to these emails, it may be possible to "interpret" them intelligently, carrying out specific actions depending on the Module that sent this email.

Previous Work

Moodle currently has functions to allow for dealing with handling email bounces - see Email processing for more details. This framework allows for the module of a returned email to be uniquely identified, as well as arguments to be passed as part of the email that will be supplied to the module's handling code.

This framework currently uses base64 encoding in order to encode data used in the return address of the email - this requires case sensitivity, which most MTA's do not support by default. This will need to be changed to use base32 instead, which significantly decreases the amount of space available in the argument section. This can be mitigated using a "email session cookie table", as suggested here.

Database structures

A database table will be necessary in order to keep track of the relevant data attached to each email "session". The following design is proposed:

mdl_email_sessions:

• id : char(26), key field - session id of the email, storing the actual base32 encoded session number which matches the "argument" field of the email's return address.
• userid : bigint(10) - user id which this email was sent to. This is used only in bounce processing.
• data : text - field to store data of any length mapped to this email session. This will probably be either a plain ascii parameter or a serialized object.
• timestamp : int(10) - timestamp of the email sent which registered this session. This is to allow "old" sessions to be removed after a defined time interval when they are no longer valid during cron.

An entry is inserted into the session table if either bounce handling or the email interface is enabled. We must ensure our data field is large enough to handle decently sized serialized objects. The TEXT type allows for 65535 characters in mysql, we can use medium- or large-text to increase this if neccesary.

By uniquely identifying each email by a session ID, we can solve the problem mentioned of handling repeats by removing the appropriate row from the table after the handler has run. Replies without an appropriate row could simply be discarded.

Core functions

• Framework - non-module code to support framework of the email interface.
  • Functions handling email sending - encoding of the reply-to address header.
    • Currently implemented as generate_email_process_address in lib/moodlelib.php.
    • Could be changed to something more specific - e.g. generate_email_session?
  • Functions handling email receiving - decode of the destination (reply-to) header.
    • Currently implemented as the command line mail parsing script admin/process_email.php.
    • This currently handles bounces as well as decoding and calling appropriate modules, and may not need to be changed much.
• Module code - changes to specific modules to actually use the email interface.
  • Forum module
    • Subscribers to forums receiving "single messages" can have their emailled replies directly added as a reply to that specific post.
    • Subscribers to forums receiving "digest messages" may need some further interface to specify which post they are replying to, e.g. "3>>Message".
  • (Investigate other modules)

Major issues

Handling bounce messages (Delivery Status Notifications)

It is neccesary for the framework to be able to distinguish between a message sent by a user in response to an email, or an automated DSN / "bounce" message sent by a MTA. Once a message has been identified as such, it can simply be discarded but must not remove the appropriate session entry, as it is possible for these messages to be generated if an email is only temporarily unreachable.

Pruning the session table

Each session entry must have an associated timestamp to identify when it was created - this can be used to prune the table of all entries before a certain period. This can be done with a cron function, and suitable defaults must be determined for how long a email session can be "kept".

Handling HTML

Some email clients send HTML encodings of the actual email message - should we parse these or simply provided functions to allow module code to parse them? Should we also handle MIME attachments like this?

Backwards compatibility

Existing code already is in place that implements a email interface (in base64) as well as a email bounce handling facility - hopefully this can be retained and new code made avaliable so that existing modules depending on these do not need to be modified.

Digest messages for the Forum module

Users can choose for messages to be queued up in a single "digest" that is sent to them regularly. A method needs to be chosen to determine which message a reply to this digest is intended for.

Major tasks

• Change configuration files (config-dist.php) to include constants for activation of the entire interface - same section as the handling of bounce messages. This should include documentation for them.
• Addition of new database table into moodle - how is this handled for non-module code?
• Creation / Modification of functions to handle encoding data - including insertion into the session table.
• Creation / Modification of functions to handle decoding data - from a session table, checking the appropriate hash. This will require us to change admin/process_email.php yet still retain the ability to handle bounces.
• Creation of a forum_process_email function within the Forum module to allow for action to be taken upon receipt of a message (reply) from a forum subscriber.
• Modification of the Forum module to pass data about the specific post(s) being sent, possibly by the session table.
• Look into how to better handle "digest"-style messages?
• Create appropriate settings for moodle admin settingpage under server->email (admin/settings/server.php).

User interface ideas

• In "digest" style emails, we need to be able to identify which discussion the reply is intended for.
  • HTML formatted emails can have mailto: links which specify a pre-existing body which contains data on which discussion this is intended for.
  • Plain text emails could be handled by requiring the user to insert some seperator (">>1" to refer to message one) before inserting his post.
  • Plain text emails can have some sort of seperators between posts, and the reply must be inserted after the correct seperator, e.g:

(Original email)
!Message 1 Seperator!
Text of message one goes here.
!Message 2 Seperator!
Text of message two goes here.

(Replied email)
> !Message 1 Seperator!
> Text of message one goes here.
User inserts message one reply here if he wants to reply to this specific message.
> !Message 2 Seperator!
> Text of message two goes here.

• We need to be able to seperate the "reply" of the email from the quote and any prefix added by the mail client.
  • HTML could contain specific elements (hr's) with id tags identifying the current discussion that the user would post between.
  • Plain text email could also emulate the above with a rather more crude "insert reply below this line" approach.

Timeline

July 9th should be a milestone (mid-term evaluations) for the completion of the framework and most (if not all) changes to the Forum module. The period after that (till August 20th) can be used for further changes and refinements to the framework and other modules. Hopefully this can be updated as further details become apparent.

Back to: Student projects