Note: You are currently viewing documentation for Moodle 1.9. Up-to-date documentation for the latest stable version is available here: Email processing.

Email processing

From MoodleDocs
Revision as of 21:28, 1 February 2007 by Daniel Pouliot (talk | contribs)

Features

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

  • Works on most modern MTAs (at least on Unix systems).
  • All processing of bounces and replies is secured using HMAC-MD5-8.
  • Bounces are handled correctly and increase a "bad email" score for the user.
  • noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
  • noreply@host has an autorresponder
  • Makes it easy for modules to send emails with a signed VERP reply-to.
  • Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

Moodle configuration

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

 // once handlebounces is true, we will be using VERP for the return address of every sent email
 $CFG->handlebounces = true;
 // minimum bounces allowed per user
 $CFG->minbounces = 10;
 // ratio of bad emails to sent emails
 // if we get more than 20% bounces 
 // for a given user, his/her email is marked bad
 $CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be /usr/bin/php.

Make sure process_email.php is executable by running "chmod ugo+rx process_email.php".

Setup under Postfix

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

   mdl:     |/var/www/moodle/admin/process_email.php
   noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

Setup under Qmail

Depending on your setup, your aliases will be controlled by one or more of

  • /etc/aliases
  • /var/qmail/alias/.qmail-PREFIX

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

   mdl:     |/var/www/moodle/admin/process_email.php
   noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

 echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
 echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

Setup under Exim

Open /etc/exim/exim.conf and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

   mdl:     |/var/www/moodle/admin/process_email.php
   noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found here. You may have to tell Exim not to lowercase the local-part.

Developer info

Changed functions:

  • email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
  • email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
  • associated string changes/additions

New functions:

  • generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"
  • moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

admin/process_email.php

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

Module Authors

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

  • encode and unencode the arguments your module needs to do the processing
  • how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array. Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

Security issues

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.