Note:

If you want to create a new page for developers, you should create it on the Moodle Developer Resource site.

Inbound message API: Difference between revisions

From MoodleDocs
(Add notes on creating a handler.)
(Updated key length and corrected some mistakes.)
Line 17: Line 17:
To ensure that the solution remains practical, a single incoming e-mail account is used, making use fo the e-mail
To ensure that the solution remains practical, a single incoming e-mail account is used, making use fo the e-mail
Subaddress extension defined in RFC 5233 (Sieve Email Filtering: Subaddress Extension). This RFC specifies that a '+'
Subaddress extension defined in RFC 5233 (Sieve Email Filtering: Subaddress Extension). This RFC specifies that a '+'
can be used to separate the mailbox from any additional subaddress data.
can be used to separate the mailbox from any additional subaddress data. For example, in the e-mail address
 
`bob+moodle@example.com, the e-mail would be delivered to `bob@example.com` but the recipient would be able to filter
For example, in the e-mail address `bob+moodle@example.com, the e-mail would be delivered to `bob@example.com` but the
mail based on the +moodle extension.
recipient would be able to filter mail based on the +moodle extension.


Note: The maximum length of the localpart of an e-mail address (including subadress) as defined in RFC 5232, is 64
Note: The maximum length of the localpart of an e-mail address (including subadress) as defined in RFC 5232, is 64
characters. The current maximum length of the subaddress component is 45 characters, with a further character used for
characters. The current maximum length of the subaddress component is 48 characters, with a further character used for
the subaddress separator. This leaves space for 18 characters in the address component.
the subaddress separator. This leaves space for 15 characters in the address component.


== Benefits ==
== Benefits ==
Line 93: Line 92:
The e-mail address generated will be of the nature:
The e-mail address generated will be of the nature:
<pre>
<pre>
incoming+AAAAAAAAAAYAAAAAAAAAAgAAAAAAAAAA6jphkaQdmIM=@example.com
incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com
</pre>
</pre>


Line 104: Line 103:
message refers.
message refers.
<pre>
<pre>
incoming+AAAAAAAAAAYAAAAAAAAAAgAAAAAAAAAA6jphkaQdmIM=@example.com
incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com
</pre>
</pre>


<pre>
<pre>
// The message was sent by bob to the incoming mailbox:
// The message was sent by bob to the incoming mailbox:
$recipient = 'incoming+AAAAAAAAAAYAAAAAAAAAAgAAAAAAAAAA6jphkaQdmIM=@example.com';
$recipient = 'incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com';
$sender = 'bob@example.com';
$sender = 'bob@example.com';


Line 147: Line 146:
To create a new handler, you must:
To create a new handler, you must:
# create a handle class; and
# create a handle class; and
# specify it's default values in that component's lib/verp_classes.php file.
# specify it's default values in that component's db/verp_handlers.php file.


Each new handler must have a unique class name within Moodle and must be loadable using the Moodle class autoloader.
Each new handler must have a unique class name within Moodle and must be loadable using the Moodle class autoloader.
Line 266: Line 265:
possible to reverse engineer the e-mail address for a known user, and then recompile it with a range of combinations.
possible to reverse engineer the e-mail address for a known user, and then recompile it with a range of combinations.


With the proposed length of the verification has (16 characters), this provides for 18,446,744,073,709,551,616
With the proposed length of the verification has (24 characters), this provides for
combinations of hash.
79,228,162,514,264,337,593,543,950,336 combinations of hash.


===== Prevention =====
===== Prevention =====
Line 279: Line 278:
===== Other notes =====
===== Other notes =====


It would be possible to increase the length of the verification key from 16 characters to something greater, but
It is difficult to increase the length of the verification hash. At present the hash is 24 characters which makes the
care would need to be given to ensure that there was enough space left for the address component given the 64 character
maximum length of the subaddress 48 characters, leaving space for 15 characters in the mailbox part of the address.
restriction defined in RFC 5232. A 24 character hash would give 79,228,162,514,264,337,593,543,950,336 potential
Although it would be possible to increase the length of an address, doing so would invalidate all previously used
combinations - 4,294,967,296 times more combinations.
addresses.

Revision as of 03:08, 17 July 2014

VERP
Project state Development in Progress
Tracker issue MDL-3714
Discussion {{{discussion}}}
Assignee Andrew Nicols

Moodle 2.8

VERP

Variable Envelope Return Path (VERP) is a method typically used to manage bouncing e-mail addresses on mail systems, but more accurately describes a way of routing e-mail messages within an application.

By encoding pieces of data within an e-mail address, it is possible to route that message within the application and handle it accordingly. With the encoding of appropriate data within that e-mail address, it is also possible to perform some level of sender verification, and to store additional data to enable further routing.

To ensure that the solution remains practical, a single incoming e-mail account is used, making use fo the e-mail Subaddress extension defined in RFC 5233 (Sieve Email Filtering: Subaddress Extension). This RFC specifies that a '+' can be used to separate the mailbox from any additional subaddress data. For example, in the e-mail address `bob+moodle@example.com, the e-mail would be delivered to `bob@example.com` but the recipient would be able to filter mail based on the +moodle extension.

Note: The maximum length of the localpart of an e-mail address (including subadress) as defined in RFC 5232, is 64 characters. The current maximum length of the subaddress component is 48 characters, with a further character used for the subaddress separator. This leaves space for 15 characters in the address component.

Benefits

The main benefits of this kind of handling are the addition of new ways of interacting with Moodle for certain content:

  • The ability for users to reply to content received from Moodle
  • The ability to e-mail new content in to Moodle

This has other potential knock-on benefits for users who cannot use the Moodle mobile application, or to unusual scenarios where Internet access may be unusually limited.

Components

The VERP Handling system is broken down into several main components:

  1. Address Management
  2. Message Handling
  3. Handler Management
  4. Message retrieval system

In order to preserve user-verification, each message used in the VERP process must be unique to that user. Additionally, the message must identify how the message should be handled within Moodle.

A handler is only passed a message following successful verification of that message and may decide how it will handle the message from that point on.

Address Manager

An address and routing manager exists to:

  1. Generate unique address;
  2. Process received messages, and extract the data stored within them;
  3. Validate addresses; and
  4. Verify sender authenticity.

Both an encode, and a decode stage exist.

Creating an e-mail address (Encode stage)

When creating a new e-mail address, the following information is required:

  1. the system which will handle any e-mail received to the generated address;
  2. a single numeric piece of data which the received message relates to; and
  3. the id of the user that a message relates to.

In the following example, an e-mail address is generated when sending a new forum post notification to a user. This address is used in the reply-to field when sending the message, thereby allowing a user to reply to a post by e-mail.

A new handler has been written for the forum '\mod\forum\verp\handler' which is aware of how it should handle any received information. The forum post notification will be sent to the user described by $recipient with a unique user id of $recipient->id. The forum post notification describes a new post described by $post and with a unique post id of $post->id.


// Create a new instance of the address manager.
$generator = new \core\verp\address();

// Specify the handler which will process any incoming messages for the generated address.
$generator->set_handler('\mod\forum\verp\reply_hander');

// Specify the post that is being replied to.
$generator->set_data($post->id);

// Generate an e-mail address to use for the reply-to, unique to this recipient.
$generator->generate($recipient->id);

The e-mail address generated will be of the nature:

incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com

Parsing an e-mail address (Decode and validation stage)

When an e-mail is received, the 'To' address field is parsed to determine the way in which the e-mail should be processed.

The following example address can be parsed to determine the sender, the handler, and the item of data to which the message refers.

incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com
// The message was sent by bob to the incoming mailbox:
$recipient = 'incoming+AAAAFXrQKf8AA3QP92///wABUl2Ukn//uCbIXwoNLnXy94kV@example.com';
$sender = 'bob@example.com';

// Create a new instance of the address manager.
$parser = new \core\verp\address();

// First parse the envelope data.
$result = $parser->process_envelope($recipient, $sender);

// We can check that the data validation passed.
if ($result !== \core\verp\address::VALIDATION_SUCCES) {
  echo "Some part of the data could not be verified\n";
} else {
  // We can retrieve the data stored in the message.
  $data = $result->get_data();

  // Get the instance of the handler - this will be the \mod_forum\verp\reply_hander used to define the message originally.
  $data->get_handler();

  // The user key is the user record of the user who the original message was sent to.
  $data->user;

  // The datavalue is the integer value that was specified when the message was originally sent.
  $data->datavalue;
}

Handlers

Once a message has been processed by the address manager, it is passed to the handler specified in the address data.

Each Handler belongs to a specific component, and comprises of a class extending \core\verp\handler, which specifies a function to handle any received e-mail messages.

Creating a new Handler

To create a new handler, you must:

  1. create a handle class; and
  2. specify it's default values in that component's db/verp_handlers.php file.

Each new handler must have a unique class name within Moodle and must be loadable using the Moodle class autoloader.

As an example, to create a forum post reply handler called reply_handler:

Component Name mod_forum
Handler Name reply_handler
Namespace mod_forum\verp
Class Name mod_forum\verp\reply_handler

In mod/forum/db/verp_handlers.php, define the available handlers.

<?php

defined('MOODLE_INTERNAL') || die();

$handlers = array(
    array(
      'classname' => '\mod_forum\verp\reply_handler',
    ),
);

And define the handler itself in /mod/forum/classes/verp/reply_handler.php:

<?php

namespace mod_forum\verp;

defined('MOODLE_INTERNAL') || die();

class reply_handler extends \core\verp\handler {
  // Place the content of the handler here.
}

A handler class must define two abstract functions:

  1. get_description() - Typically a string returned by get_string() to display in the admin interface; and
  2. process_message() - The function called after successful validation of a message, which takes information including the recipient, sender, headers, body, and a list of attachments.

A new handler must also be defined within the component in lib/db/verp_handlers.php. This array of VERP handlers includes the default settings for the handler used at creation time.

Validation and Verification

The VERP system performs a step of both validation, and verification.

Note: E-mail is never a secure method of communication, and it is easy to `spoof` the `sender` field when sending an e-mail very easily. As a result, administrators should be fully aware of the consequences when enabling any VERP handler.

When an address is generated, it will be composed of a base64-encoded binary set containing: [handlerid][userid][datavalue][verifiationstring]

  1. the handlerid is the database ID of the handler used to process the message;
  2. the userid is the database ID of the user associated with the message;
  3. the datavalue is an integer value associated with the message; and
  4. the verifiationstring is part of an md5 hash of a user key, and data key.

A new key will be generated for each individual user. This is unique to the user. A new key will be generated for each piece of data for a handler. This is unique to the handler and datavalue combination.

The keys generated for both the user, and for the handler-data pair are never exposed in the string and are always verified. The userid specified in the address must match the key stored for that user in Moodle; and similarly the datavalue specified in the address must match the key stored for the handler-pair in moodle.

This combination of three pieces of plain-text information, with two pieces of hidden additional information should make it harder for any potential attacks.

Additionally, an administrator can choose whether to compare the e-mail address of the sender against the e-mail address held against a user record. This is enabled by default, but can be disabled by an administrator.

As a further protection, it is possible to specify an expiry timestamp for each handler-datavalue keypair.

Potential attack vectors

Several potential attack vectors exist, and handler authors should determine the optimum solution required. At the very least, it is advisable to recommend sender address comparison, but this is not foolproof on it's own.

As an additional prevention method, the handler could define a further e-mail based verification check whereby on reception of a VERP e-mail, the handler system would:

  1. store the retrieved message (park it);
  2. send a new e-mail to the user described in $user containing a link or a further reply message; and
  3. await for approval from the e-mail.

Spoofed e-mail address

As mentioned above, it is incredibly easy to spoof the Sender field in an e-mail. If the full VERP address is captured by an attacker, then it would be possible to send e-mail as that user and the VERP system would be unable to distinguish the message from a valid address.

Mitigations =

Prevention
  1. When creating keys, specify an appropriate expiry time for that key pair
  2. When sending e-mail within Moodle, only use the Reply-To field as this is not typically exposed when a message is forwarded to a third-party
  3. Provide information to users advising against forwarding of e-mail
Damage Control
  1. Expire any affected keys

Brute force attack of e-mail accounts

As with any system of this nature (including passwords), it is possible to apply a brute force attack. It would be possible to reverse engineer the e-mail address for a known user, and then recompile it with a range of combinations.

With the proposed length of the verification has (24 characters), this provides for 79,228,162,514,264,337,593,543,950,336 combinations of hash.

Prevention
  1. When creating data keys, specify an appropriate expiry time for that key pair
Damage Control
  1. Expire any affected keys
Other notes

It is difficult to increase the length of the verification hash. At present the hash is 24 characters which makes the maximum length of the subaddress 48 characters, leaving space for 15 characters in the mailbox part of the address. Although it would be possible to increase the length of an address, doing so would invalidate all previously used addresses.