MoodleDocs - User contributions [en]

Email processing

2007-08-21T22:20:21Z

Pfb: Experimental email processing

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Student projects/Email interface

2007-07-22T06:03:22Z

Pfb: Updated to latest changes

'''Programmer:''' Peter Boswood

''(This is very much still a work in progress, data here may be inaccurate or outdated!)''

==Summary==

Moodle uses email in order to communicate with end users in a variety of manners - for authentication, subscriptions in the Forum module, etc. In some instances, rather than simply ignoring replies from users to these emails, it may be possible to "interpret" them intelligently, carrying out specific actions depending on the Module that sent this email.

==Previous Work==

Moodle currently has functions to allow for dealing with handling email bounces - see [[Email processing]] for more details. This framework allows for the module of a returned email to be uniquely identified, as well as arguments to be passed as part of the email that will be supplied to the module's handling code.

This framework currently uses base64 encoding in order to encode data used in the return address of the email - this requires case sensitivity, which most MTA's do not support by default. This will need to be changed to use base32 instead, which significantly decreases the amount of space available in the argument section. This can be mitigated using a "email session cookie table", as suggested [http://moodle.org/mod/forum/discuss.php?d=24077#114289 here].

==Database structures==

A database table will be necessary in order to keep track of the relevant data attached to each email "session". The following design is proposed:

mdl_email_sessions:
* id : int(10), key field, auto-inc - required by DML
* b32key : int(10), unsigned - session number of the email, encoded in a base32 part of the VERP field. An index has to be created on this to allow for fast lookup when an email is received.
* data : text(medium) - field to store data of any length mapped to this email session. This will probably be either a plain ascii parameter or a serialized object.
* timestamp : int(10), unsigned - timestamp of the email sent which registered this session. This is to allow "old" sessions to be removed after a defined time interval when they are no longer valid during cron.

An entry is inserted into the session table if either bounce handling or the email interface is enabled. We must ensure our data field is large enough to handle decently sized serialized objects - a medium text field in mysql stores 16 million characters, which is more than enough.

By uniquely identifying each email by a session ID, we can solve the problem mentioned of [[Email processing#Security_issues|handling repeats]] by removing the appropriate row from the table after the handler has run. Replies without an appropriate row could simply be discarded.

==VERP structure==

The structure of the VERP field has been changed several times since coding started - here is the latest version:
* 4 characters: prefix for MTA ("mdl+", etc.)
* 2 characters: encoded (base32 + pack('C')) module id to determine which processing function to call - module id 0 stands for Moodle 'core'.
* 16 characters: encoded (base32 + pack('C')) user id - this is only used for bounce processing, and is not passed upwards to handling functions. Any user identification is expected to be stored within the session data field.
* 16 characters: encoded (base32 + pack('C')) session id - this is used to identify the session row in mdl_email_sessions, and matches the b32key.
* 26 characters: hash validating and securing the VERP data. HMAC-MD5-26 is used for this.

==Security concerns==

Rather than using the auto-incremental id field as the session key, which would reduce the amount of data we need to store per email in the session table, a randomly generated number is used instead. Because identity should be stored within the session data, it would be extremely unlikely that even if the site identifier were known and a collision computed that a person would know another session key that should be using. The identity of the user id field in the VERP address is used only for bounce detection purposes, but this requires a session key to have been generated beforehand (and MD5 to match) to prevent denial of service attacks to force a users email address to go over their bounce-quota.

==Core functions==

* Framework - non-module code to support framework of the email interface.
** Functions handling email sending - encoding of the reply-to address header.
*** Currently implemented as generate_email_process_address in lib/moodlelib.php.
*** Could be changed to something more specific - e.g. generate_email_session?
** Functions handling email receiving - decode of the destination (reply-to) header.
*** Currently implemented as the command line mail parsing script admin/process_email.php.
*** This currently handles bounces as well as decoding and calling appropriate modules, and may not need to be changed much.
* Module code - changes to specific modules to actually use the email interface.
** Forum module
*** Subscribers to forums receiving "single messages" can have their emailed replies directly added as a reply to that specific post.
*** Subscribers to forums receiving "digest messages" may need some further interface to specify which post they are replying to, e.g. "3>>Message".
** ''(Investigate other modules)''

==Major issues==
===Handling bounce messages (Delivery Status Notifications)===
The framework is able to distinguish between messages sent by a user in response to an email, or automated DSN / "bounce" messages sent by a MTA. This is done via checking for a null return path, or existence of specific content-types which are identified as DSN specific by RFC 1894. Once a message has been identified as such, it is simply discarded - the session entry is not removed, as it is possible for these messages to be generated if an email is only temporarily unreachable.
===Pruning the session table===
Each session entry has an associated timestamp to identify when it was created - this can be used to prune the table of all entries before a certain period. A config variable - $CFG->mailsessiontime - stores the number of hours email sessions persist (defaulting to 1 month). A cron function (cleanup function) written in admin/cron.php is used to prune the session table.
===Handling HTML===
Some email clients send HTML encodings of the actual email message - should we parse these or simply provided functions to allow module code to parse them? Should we also handle MIME attachments like this?
===Backwards compatibility===
Existing code already is in place that implements a email interface (in base64) as well as a email bounce handling facility - this has been retained and new code made available so that existing modules depending on these do not need to be modified.
===Digest messages for the Forum module===
Users can choose for messages to be queued up in a single "digest" that is sent to them regularly. A method needs to be chosen to determine which message a reply to this digest is intended for.

==User interface ideas==

* In "digest" style emails, we need to be able to identify which discussion the reply is intended for.
** HTML formatted emails can have mailto: links which specify a pre-existing body which contains data on which discussion this is intended for.
** Plain text emails could be handled by requiring the user to insert some separator (">>1" to refer to message one) before inserting his post.
** Plain text emails can have some sort of separators between posts, and the reply must be inserted after the correct separator, e.g:
(Original email)
!Message 1 Separator!
Text of message one goes here.
!Message 2 Separator!
Text of message two goes here.

(Replied email)
> !Message 1 Separator!
> Text of message one goes here.
User inserts message one reply here if he wants to reply to this specific message.
> !Message 2 Separator!
> Text of message two goes here.
* We need to be able to separate the "reply" of the email from the quote and any prefix added by the mail client.
** HTML could contain specific elements (hr's) with id tags identifying the current discussion that the user would post between.
** Plain text email could also emulate the above with a rather more crude "insert reply below this line" approach.

==Timeline and current progress==

July 9th was used as a milestone for completion of the framework and some changes to the Forum module - at this stage, there was a working prototype of the framework and appropriate Forum module code, but many further refinements to the framework could still be made, specifically parsing and utility functions that all modules could use to help separate data in returned email bodies from redundant lines.

Another milestone should be reached by the end of July - a working prototype of the framework and the Forum module processing functions that:
* Correctly sends emails with appropriate bodies instructing users how to use the interface
* Recognises responses from single-post mode mails, strips them of erroneous content, and replies to that specific post (including preserving the title of the post if necessary).
* Recognises responses from digest mode mails including which post the reply is intended from, strips them of erroneous content and replies to that specific post.
* Some sort of response email for replies to the email interface that encounter errors (such as the email session having expired).
''(This will not necessarily be bug-free!)''

The time until August 20th can be used to test the code, fix/debug it, and hopefully write a few more handlers for other modules - once the framework utility functions are in place, this should be fairly easy!

Back to: [[Student projects]]

Student projects/Email interface

2007-07-07T22:19:44Z

Pfb: Added UI ideas.

'''Programmer:''' Peter Boswood

''(This is very much still a work in progress, data here may be inaccurate or outdated!)''

==Summary==

Moodle uses email in order to communicate with end users in a variety of manners - for authentication, subscriptions in the Forum module, etc. In some instances, rather than simply ignoring replies from users to these emails, it may be possible to "interpret" them intelligently, carrying out specific actions depending on the Module that sent this email.

==Previous Work==

Moodle currently has functions to allow for dealing with handling email bounces - see [[Email processing]] for more details. This framework allows for the module of a returned email to be uniquely identified, as well as arguments to be passed as part of the email that will be supplied to the module's handling code.

This framework currently uses base64 encoding in order to encode data used in the return address of the email - this requires case sensitivity, which most MTA's do not support by default. This will need to be changed to use base32 instead, which significantly decreases the amount of space available in the argument section. This can be mitigated using a "email session cookie table", as suggested [http://moodle.org/mod/forum/discuss.php?d=24077#114289 here].

==Database structures==

A database table will be necessary in order to keep track of the relevant data attached to each email "session". The following design is proposed:

mdl_email_sessions:
* id : char(26), key field - session id of the email, storing the actual base32 encoded session number which matches the "argument" field of the email's return address.
* userid : bigint(10) - user id which this email was sent to. This is used only in bounce processing.
* data : text - field to store data of any length mapped to this email session. This will probably be either a plain ascii parameter or a serialized object.
* timestamp : int(10) - timestamp of the email sent which registered this session. This is to allow "old" sessions to be removed after a defined time interval when they are no longer valid during cron.

An entry is inserted into the session table if either bounce handling or the email interface is enabled. We must ensure our data field is large enough to handle decently sized serialized objects. The TEXT type allows for 65535 characters in mysql, we can use medium- or large-text to increase this if neccesary.

By uniquely identifying each email by a session ID, we can solve the problem mentioned of [[Email processing#Security_issues|handling repeats]] by removing the appropriate row from the table after the handler has run. Replies without an appropriate row could simply be discarded.

==Core functions==

* Framework - non-module code to support framework of the email interface.
** Functions handling email sending - encoding of the reply-to address header.
*** Currently implemented as generate_email_process_address in lib/moodlelib.php.
*** Could be changed to something more specific - e.g. generate_email_session?
** Functions handling email receiving - decode of the destination (reply-to) header.
*** Currently implemented as the command line mail parsing script admin/process_email.php.
*** This currently handles bounces as well as decoding and calling appropriate modules, and may not need to be changed much.
* Module code - changes to specific modules to actually use the email interface.
** Forum module
*** Subscribers to forums receiving "single messages" can have their emailled replies directly added as a reply to that specific post.
*** Subscribers to forums receiving "digest messages" may need some further interface to specify which post they are replying to, e.g. "3>>Message".
** ''(Investigate other modules)''

==Major issues==
===Handling bounce messages (Delivery Status Notifications)===
It is neccesary for the framework to be able to distinguish between a message sent by a user in response to an email, or an automated DSN / "bounce" message sent by a MTA. Once a message has been identified as such, it can simply be discarded but must not remove the appropriate session entry, as it is possible for these messages to be generated if an email is only temporarily unreachable.
===Pruning the session table===
Each session entry must have an associated timestamp to identify when it was created - this can be used to prune the table of all entries before a certain period. This can be done with a cron function, and suitable defaults must be determined for how long a email session can be "kept".
===Handling HTML===
Some email clients send HTML encodings of the actual email message - should we parse these or simply provided functions to allow module code to parse them? Should we also handle MIME attachments like this?
===Backwards compatibility===
Existing code already is in place that implements a email interface (in base64) as well as a email bounce handling facility - hopefully this can be retained and new code made avaliable so that existing modules depending on these do not need to be modified.
===Digest messages for the Forum module===
Users can choose for messages to be queued up in a single "digest" that is sent to them regularly. A method needs to be chosen to determine which message a reply to this digest is intended for.

==Major tasks==

* Change configuration files (config-dist.php) to include constants for activation of the entire interface - same section as the handling of bounce messages. This should include documentation for them.
* Addition of new database table into moodle - how is this handled for non-module code?
* Creation / Modification of functions to handle encoding data - including insertion into the session table.
* Creation / Modification of functions to handle decoding data - from a session table, checking the appropriate hash. This will require us to change admin/process_email.php yet still retain the ability to handle bounces.
* Creation of a forum_process_email function within the Forum module to allow for action to be taken upon receipt of a message (reply) from a forum subscriber.
* Modification of the Forum module to pass data about the specific post(s) being sent, possibly by the session table.
* Look into how to better handle "digest"-style messages?
* Create appropriate settings for moodle admin settingpage under server->email (admin/settings/server.php).

==User interface ideas==

* In "digest" style emails, we need to be able to identify which discussion the reply is intended for.
** HTML formatted emails can have mailto: links which specify a pre-existing body which contains data on which discussion this is intended for.
** Plain text emails could be handled by requiring the user to insert some seperator (">>1" to refer to message one) before inserting his post.
** Plain text emails can have some sort of seperators between posts, and the reply must be inserted after the correct seperator, e.g:
(Original email)
!Message 1 Seperator!
Text of message one goes here.
!Message 2 Seperator!
Text of message two goes here.

(Replied email)
> !Message 1 Seperator!
> Text of message one goes here.
User inserts message one reply here if he wants to reply to this specific message.
> !Message 2 Seperator!
> Text of message two goes here.
* We need to be able to seperate the "reply" of the email from the quote and any prefix added by the mail client.
** HTML could contain specific elements (hr's) with id tags identifying the current discussion that the user would post between.
** Plain text email could also emulate the above with a rather more crude "insert reply below this line" approach.

==Timeline==

July 9th should be a milestone (mid-term evaluations) for the completion of the framework and most (if not all) changes to the Forum module. The period after that (till August 20th) can be used for further changes and refinements to the framework and other modules. Hopefully this can be updated as further details become apparent.

Back to: [[Student projects]]

Student projects/Email interface

2007-06-28T21:32:00Z

Pfb: Updated DB table structure

'''Programmer:''' Peter Boswood

''(This is very much still a work in progress, data here may be inaccurate or outdated!)''

==Summary==

Moodle uses email in order to communicate with end users in a variety of manners - for authentication, subscriptions in the Forum module, etc. In some instances, rather than simply ignoring replies from users to these emails, it may be possible to "interpret" them intelligently, carrying out specific actions depending on the Module that sent this email.

==Previous Work==

Moodle currently has functions to allow for dealing with handling email bounces - see [[Email processing]] for more details. This framework allows for the module of a returned email to be uniquely identified, as well as arguments to be passed as part of the email that will be supplied to the module's handling code.

This framework currently uses base64 encoding in order to encode data used in the return address of the email - this requires case sensitivity, which most MTA's do not support by default. This will need to be changed to use base32 instead, which significantly decreases the amount of space available in the argument section. This can be mitigated using a "email session cookie table", as suggested [http://moodle.org/mod/forum/discuss.php?d=24077#114289 here].

==Database structures==

A database table will be necessary in order to keep track of the relevant data attached to each email "session". The following design is proposed:

mdl_email_sessions:
* id : char(26), key field - session id of the email, storing the actual base32 encoded session number which matches the "argument" field of the email's return address.
* userid : bigint(10) - user id which this email was sent to. This is used only in bounce processing.
* data : text - field to store data of any length mapped to this email session. This will probably be either a plain ascii parameter or a serialized object.
* timestamp : int(10) - timestamp of the email sent which registered this session. This is to allow "old" sessions to be removed after a defined time interval when they are no longer valid during cron.

An entry is inserted into the session table if either bounce handling or the email interface is enabled. We must ensure our data field is large enough to handle decently sized serialized objects. The TEXT type allows for 65535 characters in mysql, we can use medium- or large-text to increase this if neccesary.

By uniquely identifying each email by a session ID, we can solve the problem mentioned of [[Email processing#Security_issues|handling repeats]] by removing the appropriate row from the table after the handler has run. Replies without an appropriate row could simply be discarded.

==Core functions==

* Framework - non-module code to support framework of the email interface.
** Functions handling email sending - encoding of the reply-to address header.
*** Currently implemented as generate_email_process_address in lib/moodlelib.php.
*** Could be changed to something more specific - e.g. generate_email_session?
** Functions handling email receiving - decode of the destination (reply-to) header.
*** Currently implemented as the command line mail parsing script admin/process_email.php.
*** This currently handles bounces as well as decoding and calling appropriate modules, and may not need to be changed much.
* Module code - changes to specific modules to actually use the email interface.
** Forum module
*** Subscribers to forums receiving "single messages" can have their emailled replies directly added as a reply to that specific post.
*** Subscribers to forums receiving "digest messages" may need some further interface to specify which post they are replying to, e.g. "3>>Message".
** ''(Investigate other modules)''

==Major issues==
===Handling bounce messages (Delivery Status Notifications)===
It is neccesary for the framework to be able to distinguish between a message sent by a user in response to an email, or an automated DSN / "bounce" message sent by a MTA. Once a message has been identified as such, it can simply be discarded but must not remove the appropriate session entry, as it is possible for these messages to be generated if an email is only temporarily unreachable.
===Pruning the session table===
Each session entry must have an associated timestamp to identify when it was created - this can be used to prune the table of all entries before a certain period. This can be done with a cron function, and suitable defaults must be determined for how long a email session can be "kept".
===Handling HTML===
Some email clients send HTML encodings of the actual email message - should we parse these or simply provided functions to allow module code to parse them? Should we also handle MIME attachments like this?
===Backwards compatibility===
Existing code already is in place that implements a email interface (in base64) as well as a email bounce handling facility - hopefully this can be retained and new code made avaliable so that existing modules depending on these do not need to be modified.
===Digest messages for the Forum module===
Users can choose for messages to be queued up in a single "digest" that is sent to them regularly. A method needs to be chosen to determine which message a reply to this digest is intended for.

==Major tasks==

* Change configuration files (config-dist.php) to include constants for activation of the entire interface - same section as the handling of bounce messages. This should include documentation for them.
* Addition of new database table into moodle - how is this handled for non-module code?
* Creation / Modification of functions to handle encoding data - including insertion into the session table.
* Creation / Modification of functions to handle decoding data - from a session table, checking the appropriate hash. This will require us to change admin/process_email.php yet still retain the ability to handle bounces.
* Creation of a forum_process_email function within the Forum module to allow for action to be taken upon receipt of a message (reply) from a forum subscriber.
* Modification of the Forum module to pass data about the specific post(s) being sent, possibly by the session table.
* Look into how to better handle "digest"-style messages?
* Create appropriate settings for moodle admin settingpage under server->email (admin/settings/server.php).

==Timeline==

July 9th should be a milestone (mid-term evaluations) for the completion of the framework and most (if not all) changes to the Forum module. The period after that (till August 20th) can be used for further changes and refinements to the framework and other modules. Hopefully this can be updated as further details become apparent.

Back to: [[Student projects]]

Student projects/Email interface

2007-06-16T01:14:01Z

Pfb: Added issues section.

Student projects/Email interface

2007-06-05T23:55:59Z

Pfb: Updated database structures

Student projects/Email interface

2007-05-30T23:56:16Z

Pfb: Added another task

Student projects/Email interface

2007-05-28T03:08:35Z

Pfb:

Student projects/Email interface

2007-05-28T03:08:08Z

Pfb:

Student projects/Email interface

2007-05-28T03:04:12Z

Pfb: Initial update