Inbound message API

Inbound Message API
Project state	Complete
Tracker issue	MDL-46282
Discussion	https://moodle.org/mod/forum/discuss.php?d=269424
Assignee	Andrew Nicols

Moodle 2.8

Summary

By encoding pieces of data within an e-mail address, it is possible to route a message within Moodle and to handle it accordingly. With the encoding of appropriate data within the e-mail address, it is also possible to perform some level of sender verification, and to store additional data to enable further routing.

To ensure that the solution remains practical, a single incoming e-mail account is used, making use of the e-mail Subaddress extension defined in RFC 5233 (Sieve Email Filtering: Subaddress Extension). This RFC specifies that a the '+' symbol can be used to separate the mailbox from any additional subaddress data. For example, in the e-mail address `bob+moodle@example.com, the e-mail would be delivered to `bob@example.com` but the recipient would be able to filter mail based on the +moodle extension. This extension is referred to as the subaddress.

The maximum length of the localpart of an e-mail address (including subadress) as defined in RFC 5232, is 64 characters. The current maximum length of the subaddress component is 48 characters, with a further character used for the subaddress separator. This leaves space for 15 characters in the address component.

Benefits

The main benefits of this kind of handling are the addition of new ways of interacting with Moodle for certain content:

• the ability for users to reply to content received from Moodle; and
• the ability to e-mail new content in to Moodle.

This has other potential knock-on benefits for users who cannot use the Moodle mobile application, or to unusual scenarios where Internet access may be unusually limited. It also allows users to respond in a natural, and familiar environment which mirrors that of other systems.

Components

The Inbound Message Handling system is broken down into several main components:

1. Address Management
2. Message Handling
3. Handler Management
4. Message retrieval system

In order to preserve user-verification, each address used in the process must be unique to that user. Additionally, the message must identify how the message should be handled within Moodle.

A handler is only passed a message following successful verification of that message and may decide how it will handle the message from that point on.

Handlers may define certain characteristics of the message handling system including:

1. the extent of the verification which takes place; and
2. the default validity period.

Address Manager

An address and routing manager exists to:

1. generate unique addresses;
2. process received messages, and extract the data stored within them;
3. validate addresses; and
4. verify sender authenticity.

Both an encode, and a decode stage exist.

Note: It is up to the developer to use the generated address in the correct fashion. Typically this will be as a 'reply-to' address when sending new messages.

Creating an e-mail address (Encode stage)

When creating a new e-mail address, the following information is required:

1. the handler - the system which will handle any e-mail received to the generated address;
2. the data value - a single integer data which the received message relates to; and
3. the user id - the id of the user that a message relates to.

A data key is generated for the supplied data value.

In the following example, an e-mail address is generated when sending a new forum post notification to a user. This address is used in the reply-to field when sending the message, thereby allowing a user to reply to a post by e-mail.

A new handler has been written for the forum \mod\forum\message\inbound\reply_handler which is aware of how it should handle any received information. The forum post notification will be sent to the user described by $recipient with a unique user id of $recipient->id. The forum post notification describes a new post described by $post and with a unique post id of $post->id.

// Create a new instance of the address manager. $generator = new \core\message\inbound\address_manager();

// Specify the handler which will process any incoming messages for the generated address. $generator->set_handler('\mod\forum\message\inbound\reply_hander');

// Specify the post that is being replied to. $generator->set_data($post->id);

// Generate an e-mail address to use for the reply-to, unique to this recipient. $replyaddress = $generator->generate($recipient->id);

$eventdata = new stdClass(); $eventdata->component = 'mod_forum'; $eventdata->name = 'posts'; $eventdata->userfrom = $userfrom; $eventdata->userto = $userto; $eventdata->subject = $postsubject; $eventdata->fullmessage = $posttext; $eventdata->fullmessageformat = FORMAT_PLAIN; $eventdata->fullmessagehtml = $posthtml; $eventdata->notification = 1; $eventdata->replyto = $replyaddress;

The e-mail address generated will be of the nature:

incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com

Parsing an e-mail address (Decode and validation stage)

When an e-mail is received, the 'To' address field is parsed to determine the way in which the e-mail should be processed.

The following example address can be parsed to determine the sender, the handler, and the item of data to which the message refers. Note that this has been implemented as core functionality and developers are not expected to need to implement this section themselves.

incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com

// The message was sent by bob to the incoming mailbox:
$recipient = 'incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com';
$sender = 'bob@example.com';

// Create a new instance of the address manager.
$parser = new \core\message\inbound\address_manager();

// First parse the envelope data.
$result = $parser->process_envelope($recipient, $sender);

// We can check that the data validation passed.
if ($result !== \core\message\inbound\address::VALIDATION_SUCCESS) {
  echo "Some part of the data could not be verified\n";
} else {
  // We can retrieve the data stored in the message.
  $data = $result->get_data();

  // Get the instance of the handler - this will be the \mod_forum\message\inbound\reply_hander used to define the message originally.
  $data->get_handler();

  // The user key is the user record of the user who the original message was sent to.
  $data->user;

  // The datavalue is the integer value that was specified when the message was originally sent.
  $data->datavalue;
}

Handlers

Once a message has been processed by the address manager, it is passed to the handler specified in the address data.

Each Handler belongs to a specific component, and comprises of a class extending \core\message\inbound\handler, which specifies a function to handle any received e-mail messages.

Creating a new Handler

To create a new handler, you must:

1. create a class extending the handler class; and
2. specify it's default values in that component's db/messageinbound_handlers.php file.

Each new handler must have a unique class name within Moodle and must be loadable using the Moodle class autoloader.

As an example, to create a forum post reply handler called reply_handler:

Component Name	mod_forum
Handler Name	reply_handler
Namespace	mod_forum\message\inbound
Class Name	mod_forum\message\inbound\reply_handler

In mod/forum/db/messageinbound_handlers.php, define the available handlers.

<?php

defined('MOODLE_INTERNAL') || die();

$handlers = array(

   array(
     'classname' => '\mod_forum\message\inbound\reply_handler',
     'enabled' => false,
     'validateaddress' => true,
   ),

); And define the handler itself in /mod/forum/classes/message/inbound/reply_handler.php: <?php

namespace mod_forum\message\inbound;

defined('MOODLE_INTERNAL') || die();

class reply_handler extends \core\message\inbound\handler {

 // Place the content of the handler here.

}

A handler class must define three abstract functions:

1. get_name() - Typically a string returned by get_string() to display as a name in the admin interface;
2. get_description() - Typically a string returned by get_string() to display in the admin interface; and
3. process_message() - The function called after successful validation of a message, which takes information including the recipient, sender, headers, body, and a list of attachments.

It may optionally define some additional functions:

1. allow_validateaddress_change() - To prevent an Moodle administrator from changing whether E-mail address validation is required. Typically this is used to force sender validation.
2. allow_enabled_change() - To prevent a Moodle administrator from enabling, or disabling the handler

A new handler must also be defined within the component in lib/db/messageinbound_handlers.php. This array of handlers includes the default settings for the handler used at creation time.

Removing quoted text from email messages

If you want the quoted text to be removed from email messages you can call the method remove_quoted_text from inside your handler as below:-

list ($message, $format) = self::remove_quoted_text($messagedata);

The method will make an educated guess and remove quoted text from the message and return the format and message contents. This method doesn't work perfectly at the moment and MDL-50058 is supposed to improve this.

Validation and Verification

The Inbound Message system performs a step of both validation, and verification.

Note: E-mail is never a secure method of communication, and it is easy to `spoof` the `sender` field when sending an e-mail very easily. As a result, administrators should be fully aware of the consequences when enabling any Inbound Message handler.

When an address is generated, it will be composed of a base64-encoded binary set containing: [handlerid][userid][datavalue][verifiationstring]

1. the handlerid is the database ID of the handler used to process the message;
2. the userid is the database ID of the user associated with the message;
3. the datavalue is an integer value associated with the message; and
4. the verifiationstring is part of an md5 hash of a user key, and data key.

A new key will be generated for each individual user. This is unique to the user. A new key will be generated for each piece of data for a handler. This is unique to the handler and datavalue combination.

The keys generated for both the user, and for the handler-data pair are never exposed in the string and are always verified. The userid specified in the address must match the key stored for that user in Moodle; and similarly the datavalue specified in the address must match the key stored for the handler-pair in moodle.

This combination of three pieces of plain-text information, with two pieces of hidden additional information should make it harder for any potential attacks.

Additionally, an administrator can choose whether to compare the e-mail address of the sender against the e-mail address held against a user record. This is enabled by default, but can be disabled by an administrator.

As a further protection, it is possible to specify an expiry timestamp for each handler-datavalue keypair.

Potential attack vectors

Several potential attack vectors exist, and handler authors should determine the optimum solution required. At the very least, it is advisable to recommend sender address comparison, but this is not foolproof on it's own.

As an additional prevention method, the handler could define a further e-mail based verification check whereby on reception of an inboune e-mail, the handler system would:

1. store the retrieved message (park it);
2. send a new e-mail to the user described in $user containing a link or a further reply message; and
3. await for approval from the e-mail.

Spoofed e-mail address

As mentioned above, it is incredibly easy to spoof the Sender field in an e-mail. If the full address is captured by an attacker, then it would be possible to send e-mail as that user and the Inbound Message system would be unable to distinguish the message from a valid address.

Mitigations =

Prevention

1. When creating keys, specify an appropriate expiry time for that key pair
2. When sending e-mail within Moodle, only use the Reply-To field as this is not typically exposed when a message is forwarded to a third-party
3. Provide information to users advising against forwarding of e-mail

Damage Control

1. Expire any affected keys

Brute force attack of e-mail accounts

As with any system of this nature (including passwords), it is possible to apply a brute force attack. It would be possible to reverse engineer the e-mail address for a known user, and then recompile it with a range of combinations.

With the proposed length of the verification has (24 characters), this provides for 79,228,162,514,264,337,593,543,950,336 combinations of hash.

Prevention

1. When creating data keys, specify an appropriate expiry time for that key pair

Damage Control

1. Expire any affected keys

Other notes

It is difficult to increase the length of the verification hash. At present the hash is 24 characters which makes the maximum length of the subaddress 48 characters, leaving space for 15 characters in the mailbox part of the address. Although it would be possible to increase the length of an address, doing so would invalidate all previously used addresses.