MoodleDocs - User contributions [en]

Email processing

2008-03-17T09:02:56Z

Matm: /* Email confirmation or registration words, and how to edit them. */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words, and how to edit them.==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, the normal way to markup a hyperlink is:-
**<nowiki><a href="http://www.blah_blah">click here</a></nowiki> If you do this in the moodle.php file, your edited text won't appear.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank\" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.
[[Image:Example.jpg]]

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T19:42:19Z

Matm: /* Email confirmation or registration words, and how to edit them. */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words, and how to edit them.==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, the normal way to markup a hyperlink is:-
**<nowiki><a href="http://www.blah_blah">click here</a></nowiki> If you do this in the moodle.php file, your edited text won't appear.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.
[[Image:Example.jpg]]

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

File:Example.jpg

2007-09-26T19:32:13Z

Matm: How to edit text strings using Site Administration.

How to edit text strings using Site Administration.

Email processing

2007-09-26T19:31:11Z

Matm: /* Email confirmation or registration words, and how to edit them. */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words, and how to edit them.==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, the normal way to markup a hyperlink is:-
**<nowiki><a href="http://www.blah_blah">click here</a></nowiki> If you do this in the moodle.php file, your edited text won't appear.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.
[[Image:Example.jpg]]

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Sandbox

2007-09-26T19:29:53Z

Matm: /* The arrangement */

$$ sqrt( x + y ) $$

'''
== Professional Development =='''

{{Category:Capabilities|Question}}

Very nice :-) 
Moodle is Great! 

'''== אחת שתיים שלו ארבע
==== Headline text ==== Headline text =='''Bold text''''''

== The arrangement ==

#We want you to read the material [text, comic or video] in a lesson and do the questions at the end.
#We want you to read the background material that is supplied.
#We want you to log onto the forum and post answers to the questions posed

There are no marks for this course - your supervisor will only know that you have successfully completed each lesson but will be able to see your postings to the forum.
[[Image:Example.jpg]]

== Headline text == (This is a level-2 Headline).

== Headline text ==
[http://www.yahoo.com YaHoooo!]

== Aesthetic entymology to follow! ==
<hr size=5>
<hr size=10>
<hr size=20>
... well, that's too bad: <tt> I was expecting to be able to make larger headers! [pout!]</tt>

:Interesting. Don't know much about HTML but
#remember this is a Mediawiki.
#When I want to try something off the beaten track in MoodleDocs, I [http://meta.wikimedia.org/wiki/Help:Editing#Templates start here with a search on editing tips].
#Since I am a clutz, I created a web page using a word processor. Then opened my browser and revealed the page code. Then placed it below. So you can make larger headers several ways. chris collman
::see examples below

==This is 26.opt test==

Notice it still is a header just not the default font

'''Let's''' have a go of this
<math>x^2 + \sin x = \infty </math>

== moodle ==
business and see what the lark is all about!

--[[User:Jonathan Foley|Jonathan Foley]] 08:42, 11 June 2007 (CDT)

== Headline text ==

Martha is a radical

=Is a 1st level header - the larger header topic=

just to see what would happen

=This is a 400.opt font test

Hmm...

'''Bold text''' test A

Just testing it out!

Seems OK!

----

Here's some text.
:Here's some indented text.
::And a bit more indented.
::::::::::Now you're just being silly.
::::Indeed!

----

[[fr:Bac à sable]]
[[ja:サンドボックス]]

User:Matt Molloy

2007-09-26T19:12:12Z

Matm:

Working in apprenticeship training and part-time university lecturer.

Email processing

2007-09-26T19:06:11Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words, and how to edit them.==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, the normal way to markup a hyperlink is:-
**<nowiki><a href="http://www.blah_blah">click here</a></nowiki> If you do this in the moodle.php file, your edited text won't appear.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T19:05:09Z

Matm: /* Email confirmation or registration words, and how to edit them. */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, the normal way to markup a hyperlink is:-
**<nowiki><a href="http://www.blah_blah">click here</a></nowiki> If you do this in the moodle.php file, your edited text won't appear.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:52:53Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, the normal way to markup a hyperlink is:-
**<nowiki><a href="http://www.blah_blah">click here</a></nowiki> If you do this in the moodle.php file, your edited text won't appear.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:48:53Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, to markup a hyperlink:-
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:47:33Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, to markup a hyperlink:-
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
*Use the \ before each " when marking up things like target=\"_blank" and title=\"Blah blah\" and other similar codes, anywhere you must use " in the markup.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:45:27Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank. For example, to markup a hyperlink:-
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
**Use the \ before target= and title= and other similar codes, anywhere you must use " in the markup.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:44:07Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
**Use the \ before target= and title= and other similar codes, anywhere you must use " in the markup.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:43:20Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
**Use the \ before target= and title= and other similar codes, anywhere you must use " in the markup.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:42:06Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*If you feel the need to edit this text, the best way to do this is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
**Use the \ before target= and title= and other similar codes, anywhere you must use " in the markup.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

Email processing

2007-09-26T18:41:18Z

Matm: /* Email confirmation or registration words */

== Features ==

Moodle now makes better use of the SMTP protocol and email in general. Most of the benefits result from using a technique known as Variable Envelope Return Path (VERP).

* Works on most modern MTAs (at least on Unix systems).
* All processing of bounces and replies is secured using HMAC-MD5-8.
* Bounces are handled correctly and increase a "bad email" score for the user.
* noreply@host address is now on Reply-to field, avoiding accidental pollution of users address books.
* noreply@host has an autorresponder
* Makes it easy for modules to send emails with a signed VERP reply-to.
* Handles receving of VERP replies: Validates the HMAC-MD5-8 signature and Dispatches the encoded request data to the relevant module

== Moodle configuration ==

Edit config.php to enable bounce handling, and setup Moodle to match your MTA configuration. Here's how. Uncomment these lines in config.php (if you cannot find them, copy them from config-dist.php):

// once handlebounces is true, we will be using VERP for the return address of every sent email
$CFG->handlebounces = true;
// minimum bounces allowed per user
$CFG->minbounces = 10;
// ratio of bad emails to sent emails
// if we get more than 20% bounces
// for a given user, his/her email is marked bad
$CFG->bounceratio = .20;

Edit the $CFG->maildomain line and one of the $CFG->mailprefix lines (the one that matches your MTA).

Make sure your server has a command-line PHP interpreter, and that it is able to connect to mysql (or postgres if relevant). If you are able to run cron.php from the commandline or from crontab, this means PHP is ok.

Edit the process_email.php script to point to the location of your php binary. It will usually be ''/usr/bin/php''.

==Email confirmation or registration words==
* Students receive a confirmation email when they create a user account. This text can be found in the lang/?/moodle.php as the emailconfirmation "variable".
* Students receive a welcome email when they enroll in a course. This text can be found in the lang/?/moodle.php file as the welcometocoursetext "variable".
*The best way to edit these strings is to go to Site Administration -> Language editing, and then clicking on the string that you wish to edit. Once you have done this, hit the "Switch lang directory" button, and do your editing.
*If you decide to get your hands dirty and work by editing the moodle.php directly, be careful to open the file only in a simple text editor like notepad or wordpad.
*Tip: Be careful with markup - as you edit, you must use " for things like putting in hyperlinks, titles and targets. You MUST 'escape' them by putting a \ before each ". If you don't the resulting page will appear blank.
**Use <nowiki><a href=\"http://www.blah_blah\">click here</a></nowiki>
*** Do not use <nowiki><a href="http://www.blah_blah">click here</a></nowiki>
**Use the \ before target= and title= and other similar codes, anywhere you must use " in the markup.
*If you feel you must really go the route of editing the moodle.php file directly, save a copy of the file in its own folder (copy and paste the file in the same folder). That way, if it all goes wrong for you, all you have to do is delete the botched file and rename the 'copy of moodle.php' back to 'moodle.php' and you are back to square one, no harm done!
*Having said all that, it is best to do all your editing in the web interface by going to Site Administration -> Language editing, and then clicking on the string that you wish to edit.

Make sure ''process_email.php'' is executable by running "chmod ugo+rx process_email.php".

== Setup under Postfix ==

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '|', and the path of the script. For example for a prefix of 'mdl' and moodle installed under /var/www/moodle we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

== Setup under Qmail ==

Depending on your setup, your aliases will be controlled by one or more of

* ''/etc/aliases''
* ''/var/qmail/alias/.qmail-PREFIX''

If you edit /etc/aliases add a line like this (for a prefix of 'mdl'):

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you create /var/qmail/alias/.qmail-PREFIX, just do

echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-mdl
echo "|/var/www/moodle/admin/process_email.php" > /var/qmail/alias/.qmail-noreply

To this three letter prefix, we will add a '-' when sending and receiving messages. For more info, check out the manpage for dot-qmail.

== Setup under Exim ==

Open ''/etc/exim/exim.conf'' and add to trusted_users the user Apache and cron.php run as (usually "www-data" or "nobody").

Add a line to your aliases file. The line should list a 3-letter prefix, to which we'll add a '+', and the path of the script. For example for a prefix of 'mdl' we have in aliases:

mdl: |/var/www/moodle/admin/process_email.php
noreply: |/var/www/moodle/admin/process_email.php

If you are using virtualdomains, consult your server administrator for the correct configuration. It probably involves editing transports and mapping your address to a "pipe" transport.

More documentation about Exim can be found [http://www.exim.org/docs.html here]. You may have to tell Exim not to lowercase the local-part.

== Developer info ==

Changed functions:

* email_to_user() will set the envelope sender to a special bounce processing address (based on $CFG settings)
* email_to_user() will accept (and set) a reply-to header, to be generated by the module calling the function.
* associated string changes/additions

New functions:

* generate_email_processing_address() - ALWAYS use this to generate the reply-to header. reply-to header will look like this: (LIMIT: 64 chars total) prefix - EXACTLY four chars encodeded, packed, moduleid (0 for core) (2 chars) up to 42 chars for the modules to put anything they want it (can contain userid (or, eg for forum, postids to reply to), or anything really. 42 chars is ABSOLUTE LIMIT) 16 char hash (half an md5) of the first part of the address, together with a site "secret"

* moodle_process_email() - any non-module email processing goes here (currently used for processing bounces)

New files:

''admin/process_email.php''

This script needs to be called from your MTA for anything starting with the 3 char prefix described above (and optionally, the noreply address).

How does it work? It will break down and unencode the email address into moduleid and validate the half md5 hash, and call $modname_process_email (if it exists). Arguments to these functions are: $modargs (any part of the email address that isn't the prefix, modid or the hash) and the contents of the email (read from STDIN).

It doubles up as the noreplyaddress autorresponder if you configure it with that address as well. Replying with a friendly "this is not a real email address" message.

== Module Authors ==

Take a look at new functions moodle_process_email() and generate_email_processing_address() in moodlelib.php for ideas about how to

* encode and unencode the arguments your module needs to do the processing
* how to deal with multiple "actions" for any given module.

In a nutshell, users can send emails to Moodle using special dynamic addresses. These emails can trigger a call to a module function in the form of modulename_process_email($str, $bodyofemail). The $str part will be up to 42 characters of data generated by Moodle (presumably by your own module), and the $bodyofemailpart is the contents of the email (got by reading STDIN - usually generated by the users MUA).

The 42 characters come from the "local part" of an email address (the part before the @ sign) which can have up to 64 chars. Out of those 64 chars, Moodle uses 22 characters, leaving you with 42 characters to encode data.

What do we do with the 22 chars? Four go to the prefix, which we need to let the MTA know to pass the message to our script. Two go to identify the module ID so we know which module has generated the message (and so we dispatch the request to that module). The remaining 16 are a signature (HMAC-MD5-8) we use to authenticate the message.

Forty-two characters isn't a lot (although it could be the answer to life, the universe, and everything!) so make sure you use those characters wisely.

The most efficient way to encode database IDs in their full range (so that they can be placed in an email address) we have found is base64_encode(pack('V',2147483647)), which returns "/ / / / f w = =". The two trailing "= =" are redundant and you can remove them (you'll need to reappend them when retrieving your data). Join your parameters as encoded IDs in positional slots for efficiency.

To retrieve your data, use substr() to separate your parameters, and then unpack('V',base64_decode($str)). Note that it'll return a one-element array.
Note:

Using 'V' reaches 2147483647, half the range of mySQL's INT. Additionally, 'V' behaves like a signed value, rather than an unsigned, so I suspect there's a bug in PHP's documentation of pack().

With each ID taking 6-chars (8 chars if we find a way to use the full range of 'V'), you have a limited number of parameters. If you need to encode more information, store it in the DB and send emails that point to your stored data. Remember to cleanup this temporary data after a safe period of time.

Note:

Do not try to use variable-width encoding to put IDs, as it'll work in small installations and break in larger ones.

== Security issues ==

Any code in modulename_process_email() _must_ assume it will see repeat replies and handle them gracefully. The definition of 'gracefully' depends on what the code does.

Email servers (MTAs) will sometimes re-transmit a message if they are unsure that the receiving MTA got it -- and sysadmins may sometimes replay a whole email queue if something's gone wrong. In that case, they email body will be identical, the headers slightly different.

In a different case, the user may 'reply' to the message twice. Perhaps in error, perhaps purposefully. What to do depends on the specific scenario.

We /could/ support better protection at the framework level, by keeping track of every reply-to address we send out. We decided against that because (a) the performance impact will be important (b) we want the 1st cut to be lightweight and simple to change in case we need to.

With this the initial implementation, modules should expose functions that handle these "replay" cases correctly. If later we want to expose additional functions, we can add such tracking as an optional thing. It'd be awful to have it across all emails sent from Moodle.

== Experimental email processing, pending merge into 1.9/2.0 ==
===Module Authors===
The email interface allows you to let users respond to emails sent via your module and affect change in them. The basic idea is that each email is uniquely identified with an email session that carries with it a simple payload. This payload is stored via database, and cannot be changed by the user. Replies are handled by module code in your lib.php library - modulename_process_email($payload, $body) - which is passed the session payload and the body (with full headers) of the email received, to be parsed as you like. Some utility functions are available as well to help with this.

Here is an outline of the intended use:
* Before sending an email to a user via the email_to_user() function, the module should check to see if the config variable $CFG->emailinterface is enabled. (This is off by default)
* Assuming the email interface is enabled, the function generate_email_verp_address($moduleid, $payload, $userid) should be called to create an "email session". The payload should include any data you wish to use in your handling routine - such as the id of the user, or what part of the interface he is interacting with. The function will return a string containing the email session id.
* Call the email_to_user() function as normal, appending the email session id as the 11th parameter. Keep in mind that you will probably also need to change the text or html of the email to point out that the email interface is active.
* A function contained in the lib.php file of your module, named [modulename]_process_email($payload, $body) should be written. This is the handler function, which is called when an email reply is received. This function can inspect the contents of the payload received, then parse the email body for data. The $body parameter returns the full email, including the headers.

Note that each email can only be replied to once - the email session is destroyed once an email has been received. If you wish to have multiple replies from a user, you will have to create a new session (and new email) every time. The email session table is also regularly pruned, the default settings allow sessions to persist for one month.

The following utility functions are also defined in admin/process_email.php, which can help:
* get_email_subject($fullemail) - return the subject of an email
* strip_email_headers($fullemail) - strip all headers off an email
* email_is_multipart($fullemail) - determine if an email is a multipart message
* seperate_multipart($multipartmsg) - seperate a multipart email message into an array of parts, each of which has headers and a body.
* get_multipart_content_type($part) - determine the content-type of a multipart, generally text/plain or text/html.
* multipart_is_quoted_printable($part) - determine if a email part has been encoded with the quoted-printable encoding - some email clients tend to do this to html.
* decode_quoted_printable($partdata) - decode a email that has been encoded with quoted-printable encoding - this needs to be run on the data section of a part (strip_email_headers can be used to seperate the headers of a part).
* strip_email_reply($body, $ishtml, $identifier) - strip all irrelevant lines of an email message based on a unique identifier embedded within the original message. This is used to determine what is user input, and what is irrelevant data the email client has added.
* strip_email_reply_multi($body, $ishtml, $identifier) - as above, but allowing for multiple sections of possible user input in a single email message.

Examples of use of these functions can be seen in the handler for the forum module. The idea behind the strip reply functions is that the original email will contain a unique identifier, such as a random 10 character string. This is embedded in a line with instructions for the user to write below and above it, e.g:
\/ Please put your response here [ABCDEFGHIJ] \/
<User Data>
/\ Please put your response here [ABCDEFGHIJ] /\
This is necessary because of the wide differences between webmail and email clients in handling quotes and prefixing dates/times to replied emails. Keep in mind that text emails have an 80-character line width restrictions, which must also include the quote added by the email client, so any line of a text-email with this identifier should be kept to at least 77 characters to avoid wrapping by an email client. This unique identifier can be easily stored in the payload. Multiple input sections are handled similarly - a number is appended to the end of the identifier, e.g. [ABCDEFGHIJ1] to signfiy the first section of input. The strip_email_reply_multi returns an array containing a map from each numbered section it finds to a string containing all data in that section.

===Developer Info===
The reply-to header is now encoded in base32 instead of base64 to allow for case-insensitive MTA's. Generate_email_processing_address was left as a backwards-compatibility wrapper, modules should use generate_email_verp_address now instead.

Email sessions have been created to deal with the lack of space avaliable in the header under base32, payloads are stored in a new table mdl_email_sessions, and are identified with a unique base32 key which is embedded in the reply-to header. Each row has a timestamp, and the table is pruned during cleanup cron in admin/cron.php.

Bounce detection is now linked in with the email interface - bounce detection is included in admin/process_email.php. Some new configuration variables have also been created to help configure these changes - $CFG->emailinterface and $CFG->emailsessiontime - and both have been added to the server settings admin page.

[[Category:Administrator]]
[[Category:Developer]]

admin/environment/php extension/iconv

2007-08-12T13:05:41Z

Matm:

To install/enable iconv:

#Open the ''php.ini'' file found in the ''moodle/apache/bin'' folder
#Find the line: <code>;extension=php_iconv.dll</code>
#Remove the <code>;</code> at the beginning of the line
#Restart apache for the change to take effect.
{{stub}}

[[Category:Environment|PHP]]

Installation FAQ

2007-08-03T18:48:12Z

Matm: /* Fatal error allowed memory size exhausted. How do I increase my php memory limit? */

{{FAQ}}

==PHP - is it installed and what version do I have?==

Make a new file on your web site called ''info.php'', containing the following text, and call it from your browser:

<?PHP phpinfo() ?>

If nothing happens then you don't have PHP installed or your webserver is not configured to handle .php files properly. See the installation docs for some information about where to download it for your computer. See the [[phpinfo]] page for details about the content of this page.

== System information needed for Installation Forum ==
When posting questions to the installation forum, try to provide as much background information as possible about your moodle system. Use this template to copy and paste into your post:
* Server OS name (version also if possible):
* Browser name (version also if possible):
* Moodle version:
* Moodle config.php attached?(Y/N):
* Phpinfo attached? (Y/N):

For the last two items, try to include the following in your post as an attachment:
* A copy of your phpinfo output as shown in your browser (see the instructions above for an explanation of how to obtain this).
* A copy of the Moodle configuration file. This is located in the directory moodle and is named config.php

Copy and paste both of these into a single text file (using vi, Notepad, etc) and attach this to your post.

If you cannot provide your phpinfo, try to copy & paste and complete these in your post:
* Apache version:
* MySQL version:
* PHP version:

For installation on web hosting accounts: contact your support desk who should be able to tell you this information.

: '''Security Warning''': Make sure you edit any files and delete any passwords before posting onto the forum.

==What & where are Moodle's configuration settings stored?==
Configuration settings are stored in the config.php file stored in your moodle folder. This file is created during the installation process. If there is a problem and the installation cannot create the file, you can try creating it manually from the [[Configuration file]] docs. Please remember that manually editing the file is not recommended and may lead to blank pages, especially if there are additional spaces and/or lines after the final php closing tag "?>".

==Running a health check==
Moodle contains a script that will help identify common php and webserver configuration problems as well as configuration problems. It is a good idea to run this script to check if you are having post-installation problems. Use your browser to run this file:

http://www.mymoodle.com/moodle/admin/health.php

Change the above line if you have installed moodle in the webroot instead of a folder inside the webroot.

==Downloading previous releases of Moodle==
* '''Generic Packages''': If your server does not meet the [[Installing_Moodle#Requirements | requirements]] for the current version of Moodle, you can download previous releases by using this URL:
http://download.moodle.org/stable[version_number]
:For example: to download Moodle version 1.5, use http://download.moodle.org/stable15. You'll see a directory tree with the files displayed. Click on the one you want and download as normal - if you require the latest update of the version, scroll to the end of the list and download the "moodle-latest" file. Changes made in the version in the last month are listed in the "CHANGES" file. The files contain Moodle code and are not the Windows or Mac packages - so you need to have a webserver, a database server and PHP already installed. The earliest version available is Moodle 1.3.
* '''Windows Packages''': To download previous releases of the Moodle packages for Windows, use this URL:
http://download.moodle.org/windows/MoodleWindowsInstaller-latest-[version_number].zip
* '''Mac Packages''': To download previous releases of the Mac pacakges, use either of these URLs (depending on whether you need the Intel or PPC package):
http://download.moodle.org/macosx/Moodle4Mac-Intel-[version_number].dmg
http://download.moodle.org/macosx/Moodle4Mac-PPC-[version_number}.dmg
* '''Using CVS''': You can also use CVS to download older releases and incremental releases of the Moodle generic packages, e.g. Moodle 1.5.4 - see the [[CVS_for_Administrators | CVS documentation]].

== How to enable and check PHP error logs==
PHP can be set up to log errors in a variety of different ways: two of these involve the use of the php.ini file and the ini_set command.
* '''Using the php.ini file''': The log settings are contained in the php.ini file stored on the server. If you don't know where that is, edit your Moodle ''config.php'' and add the following as the second line

phpinfo();

:then reload the web page. Look for the entry '''Configuration File (php.ini) Path'''.

:When you have located php.ini open it in your favorite text editor. Find the '''Error handling and logging''' section of the php.ini file. Make sure that both '''display_errors = On''', '''display_startup_errors = On''' and '''log_errors = On''' are present and uncommented. Check the value of '''error_log''' - this tells you the location of the file errors are logged to. If it is commented out then errors will be sent to the web server error log file. Remember, if you make any changes to this file you will need to restart the web server (or just reboot the server).

* '''Using ini_set commands''': If you are using Moodle 1.8 or higher, the previous steps are not enough. In those versions error logging parameters are dependant on certain administrative settings that you specify in the debugging section. The problem is that if you can't access the administrative pages, you can't set the debugging options. So the only way to modify them is by adding the following lines to your config.php file, just before the last line (the one containing a single'?>' only):

ini_set ('display_errors', 'on');
ini_set ('log_errors', 'on');
ini_set ('display_startup_errors', 'on');

:This will enable the same settings specified above even if Moodle sets them otherwise.
:'''Important''': Remember to put them just before the last line of config.php.

==Any text I add with an apostrophe (') or a quote (") causes errors or comes up with a slash added==

Problems caused by apostrophes are caused by incorrect "magic quotes" settings. Moodle requires the following settings in the php.ini file (which are usually the default):

magic_quotes_gpc = On
magic_quotes_runtime = Off

Please see [[Installing Moodle]] for more details.

==Email copies are not being sent from my forums==

You ''must'' set up cron properly if you want Moodle to send out automatic email from forums, assignments etc. This same process also performs a number of clean-up tasks such as deleting old unconfirmed users, unenrolling old students and so on.

Basically, you need to set up a process to regularly call the script <code><nowiki>http://yoursite/admin/cron.php</nowiki></code>. Please refer to the [[Cron|cron instructions]].

'''Tip:''' Try the default setting in Moodle variables page. Leave the smtphost blank. This will be acceptable for the majority of users.

'''Tip:''' Make sure that allowuseremailcharset in Administration > Configuration > Variables > Mail is set to No. Setting this to Yes might cause this problem in some versions of Moodle.

==Error: database connection failed==

If you get errors like "database connection failed" or "could not connect to the database you specified", here are some possible reasons and some possible solutions.

* Your '''database server''' isn't installed or running. To check this for MySQL try typing the following command line
$telnet database_host_name 3306
:You should get a cryptic response which includes the version number of the MySQL server.
* If you are attempting to run '''two instances of Moodle on different ports''', use the ip address of the host (not localhost) in the $CFG->dbhost setting, e.g. $CFG->dbhost = 127.0.0.1:3308.
* You don't have the '''PHP mysql or postgresql extensions''' installed (please refer to FAQ re. whether PHP is installed).
* You haven't created a '''Moodle database and assigned a user''' with the correct privileges to access it.
* The '''Moodle database settings''' are incorrect. The database name, database user or database user password in your Moodle configuration file ''config.php'' are incorrect. Use phpMyAdmin to set up and check your MySQL installation.
* Check that there are '''no apostrophes or non-alphabetic letters''' in your MySQL username or password.
* You are using MySQL version 4.1 or higher but the PHP MySQL extension is pre-4.1 (check in your phpinfo output). In this case the '''default password hashing algorithm''' is incompatible with that available in the PHP mysql extension versions 4.x.x. Use these MySQL commands to change the passwords to the old format:

mysql>SET PASSWORD FOR 'root'@'localhost' = OLD_PASSWORD('password');
mysql>SET PASSWORD FOR 'moodleuser'@'localhost' = OLD_PASSWORD('password');

:Also, consider upgrading your PHP MySQL extension. See [http://dev.mysql.com/doc/mysql/en/old-client.html this MySQL document] for further information on how to deal with this problem.
* You are using Fedora core 3 or some other Linux system with '''SELinux installed''' and enabled. See the following URL for information on how to disable SELinux: http://fedora.redhat.com/projects/selinux/
* Mac OSX users -- if you are running MySQL on a Mac OSX, try changing '''$CFG->dbhost''' from 'localhost' to '127.0.0.1'
'''See also''': MySQL page on [http://dev.mysql.com/doc/refman/5.0/en/common-errors.html common errors] which lists several possible scenarios for connection failure, with advice on how to fix the problems.

==I can't log in - I just stay stuck on the login screen==

The most common cause for this is that your own computer (not your Moodle server) has a firewall that is stripping referrer information from the browser. Here are some instructions for fixing [http://service1.symantec.com/SUPPORT/nip.nsf/46f26a2d6dafb0a788256bc7005c3fa3/b9b47ad7eddd343b88256c6b006a85a8?OpenDocument&src=bar_sch_nam Norton firewall products].

The server admin can also fix this for everyone by changing the ''secureforms'' variable to 'No' in the security section of Administration >> Configuration >> [[admin/config|Variables]].

Another possible cause of this problem is that sessions are not configured properly on the server. You can test this by calling the script <nowiki>http://yourserver/moodle/lib/session-test.php</nowiki>.

If you are still having problems, read the [[Can_not_log_in | Cannot log in]] page.

==I can't log in with message "Please verify that the current setting of session.save_path is correct" ==

This error occurs when PHP is having problems saving its session files. You may also see these other error messages displayed on the screen or in your log files:

Warning: Unknown: open(some-path/sessions/sess_acbf942a7399db3489ffa910e35d5242, O_RDWR)
failed: Permission denied (13) in Unknown on line 0

Warning: Unknown(): open(some-path/sessions/sess_acbf942a7399db3489ffa910e35d5242, O_RDWR)
failed: No space left on device (28) in Unknown on line 0

Warning: Unknown: Failed to write session data (files). Please verify that the current
setting of session.save_path is correct (some-path/sessions) in Unknown on line 0

To temporarily bypass these errors, '''use database sessions''' by editing your [[Configuration_file | moodle configuration file]] and adding this line:

$CFG->dbsessions = true;

Database sessions may overload your mysql database and are not ideal in a shared hosting environment, so if this solves the problem, you can start fixing the problem as follows:
* Check '''access rights'''. The session.save_path should be accessible by the apache user. Try this command:

chown -R apache:apache some-path/sessions

:This assumes that 'apache' is the name of the user your webserver runs under - it could also be 'nobody'.
* Check the '''permissions''' to the directory that PHP is trying to save to (session.save_path = some-path/sessions). Set the permissions initially to 0777 (everyone read, write, execute) with this command:

chmod -R 0777 some-path/sessions

:If this fixes the problem, reduce the permissions (700 is recommended).

'''See also''': Session problems can be specific to your server environment. As an example, see [http://moodle.org/mod/forum/discuss.php?d=55925#254596 this forum discussion] about session problems with Lycos hosting.

==I log in but the login link doesn't change. I am logged in and can navigate freely.==

Make sure the URL in your <code>$CFG->wwwroot</code> setting is exactly the same as the one you are actually using to access the site.

==I keep getting this error: A server error that affects your login session was detected.==

Please refer to the Using Moodle forum discussion [http://moodle.org/mod/forum/discuss.php?d=73716 A server error that affects your login session was detected. Please login again or restart your browser.].

==I keep getting this error: Failed opening required '/web/moodle/lib/setup.php'==

In your ''config.php'', the setting that you use for the dirroot variable must be the complete path from the root of your server's hard drive.

Sometimes people only use the path from their home directory, or relative to the root of the web server directory.

==My pages show fatal errors such as : Parse error, call to undefined function: get_string()==

If you see errors like:

Parse error: parse error, unexpected T_VARIABLE in /path/to/moodle/config.php on line 94
Fatal error: Call to undefined function: get_string() in /path/to/moodle/mod/resource/lib.php
on line 11

then you have probably left out a semi-colon or closing quote from a line in ''config.php'' (previous to line 94).

Another possibility is that you edited ''config.php'' in a program like Word and saved it as a HTML web page, instead of using a plain text editor like Notepad.

Another thing to check, particularly if you are using 3rd party modules or plugins, is whether any of the php scripts use short open tags (<? ?>) instead of proper ones (<?php ?>). Short tags are bad for various reasons, so first contact the author of that extension to tell them about the problem. Then either replace short tags with conventional ones, or set this line in php.ini:

short_open_tag = On

You should never find short tags in core moodle code. If you do, please file a bug in the bug tracker.

==Serious Error! Could not set up the site!==

Please refer to the Using Moodle forum discussion [http://moodle.org/mod/forum/discuss.php?d=32071 Serious Error! Could not set up the site!].

==Uploaded files give "File not found"==

For example: Not Found: The requested URL /moodle/file.php/2/myfile.jpg was not found on this server.

Your web server needs to be configured to allow the part of the URL after a script name to be passed directly to the script. This is usually enabled in Apache 1, but is usually disabled by default in Apache 2. To turn it on, add this line to your ''httpd.conf'', or to a ''.htaccess'' file in your local directory (see [[Installing Moodle]] for more details):

'''AcceptPathInfo''' on

Note, this will ONLY work for Apache versions 2.x.

If you are not using Apache 2 and you still have this problem (unlikely) then you can switch Moodle to use an alternative method. The disadvantages are a slight loss of performance for your users and you won't be able to use relative links within HTML resources.

To use this alternative method, you should change the ''slasharguments'' variable in the Operating System section of Administration >> Configuration >> [[admin/config|Variables]]. You should now be able to access your uploaded files.

==When I go to the admin page, I get told to make dirroot blank!==

If you see errors like this:

Please fix your settings in config.php:
You have: $CFG->dirroot = "/home/users/fred/public_html/moodle";
but it should be: $CFG->dirroot = "";

then you have encountered a small bug that occurs on some servers. The problem is with the error-checking mechanism, not with your actual path. To fix it, find this line (line 66) in the file ''admin/index.php'':

if ($dirroot != $CFG->dirroot) {

and change it to this:

if (!empty($dirroot) and $dirroot != $CFG->dirroot) {

==When trying to add a resource I receive error messages==

Assuming you are using Apache, then it's quite likely that your setting in ''config.php'' for <code>$CFG->wwwroot</code> is different from the actual URL you are using to access the site. Also try turning off the ''secureforms'' variable in the security section of Administration >> Configuration >> [[admin/config|Variables]].

==Why are all my pages blank?==

Check the dirroot variable in ''config.php''. You must use complete, absolute pathnames e.g.

$CFG->dirroot = "d:\inetpub\sites\www.yoursite.com\web\moodle";

Another reason might be that PHP has not been configured to support MySQL. This is common on Redhat and OpenBSD installations. In this case, an error is generated, but since error displays are often disabled by default, all that is seen on the browser is a blank screen. To enable PHP error displays, set these lines in your ''php.ini'' file and reload the web page.

display_errors = On
display_startup_errors = On

To determine if MySQL support is your problem, insert this as the second line in your ''config.php'' file

phpinfo();

then reload the web page. Examine the output closely to see if MySQL is supported. If not look for a package you are missing.

== Why is a particular page blank or incomplete? ==

*'''Check your web server log files!!'''
:If a particular page is blank or incomplete (it doesn't display the footer), before you do anything else [[Installation_FAQ#How_to_enable_and_check_PHP_error_logs | check your error logs]]. Having established that PHP error logging is working, reproduce the error. Immediately check the error log file right at the end. Hopefully you will see a PHP error message at or very near the end of the file. This may solve your problem directly or makes it a lot easier to diagnose the problem in the Moodle forums.

*If you are '''upgrading to a new version of Moodle''', check that you do not have an old version of a non-standard block or module installed. Remove any such blocks or modules installed remove them using the admin settings page and start the install process again.

*If you '''do not see any blocks listed''', turn editing on and remove any blocks that you have added to that page and try reloading.

==Installation hangs when setting-up database tables==
*Sometimes the installation will hang when setting up tables. This will be an abrupt hang with half the page displayed in the browser and/or other outputs removed, e.g. the “Scroll to continue” link is displayed but no “Continue” button is there. If this is the case, it is usually a mysql error and not a php error. Check that there is no limit placed on your mysql database, e.g. a "questions" limit.

*If the install is on a webhost, adding the following line to the .htaccess file in the moodle directory has been known to solve the problem.
AddType x-mapp-php5 .php

*Try also renaming the .htaccess file so that it is disabled.

*You may also want to look and see if you've customized any of your code. Look at the last successful table, and then look at the block, mod, or other code that is referenced by that table. For example, if your install hangs and continues to say that the forum tables were successful as the last message, look at /mod/forum/ for any custom code. If you have customized code, backup those files and replace with the correct files. You can then restart the install by renaming config.php or reinstalling your database from the backup. If your install is successful, you can make your code changes back into the stock Moodle code.

*It may also be that the "memory_limit" in your php.ini is set too low. Please check your php.ini file and allocate at least 40MB, as recommended, as the maximum amount of memory a script may consume (ver 1.8).

==Why can't I upload a new image into my profile?==

If you don't see anything on your user profile pages to let you upload user images then it's usually because GD is not enabled on your server. GD is a library that allows image processing.

1. Make sure '''GD has been included in your PHP installation'''. You can check this by going into Administration >> Configuration >> [[Variables]] and looking for the gdversion setting. This setting is chosen automatically every time you visit that page. If it shows GD version 1 or version 2 then everything should be fine. Save that configuration page and go back to your user profile.

2. If Moodle thinks GD is not installed, then you will need to '''install the GD library'''.
*On Unix you may need to re-compile PHP with arguments something like this:

./configure --with-apxs=/usr/local/apache/bin/apxs --with-xml --with-gd
--with-jpeg-dir=/usr/local --with-png-dir=/usr --with-ttf --enable-gd-native-ttf
--enable-magic-quotes --with-mysql --enable-sockets --enable-track-vars
--enable-versioning --with-zlib

* On Windows this is usually a matter of "turning on" the extension in PHP by editing your php.ini file. To do this remove the semicolon for the php_gd2.dll extension - check that this file is actually present in your php extensions folder first (search your php.ini for extension_dir to determine where this points to on your hard disk). You should then have a line that looks like this:
extension=php_gd2.dll

:Windows users should see the [[Installing AMP|installation instructions]] for further help.

3. Remember to '''restart your webserver''' (if possible) and re-visit the Moodle configuration page after making any changes to PHP so it can pick up the correct version of GD.

'''See also''': Using Moodle forum discussion [http://moodle.org/mod/forum/discuss.php?d=44271 Profile pictures] for additional information.

==Why do I keep getting error messages about "headers already sent"?==

If you see errors like this:

Warning: Cannot add header information - headers already sent by
(output started at /webs/moodle/config.php:87) in /webs/moodle/lib/moodlelib.php
on line 1322

Warning: Cannot add header information - headers already sent by
(output started at /webs/moodle/config.php:87) in /webs/moodle/lib/moodlelib.php
on line 1323

Warning: Cannot add header information - headers already sent by
(output started at /webs/moodle/config.php:87) in /webs/moodle/login/index.php
on line 54

you have blank lines or spaces after the final <code>?></code> in your ''config.php'' file. Sometimes text editors add these - for example Notepad on Windows - so you may have to try a different text editor to remove these spaces or blank lines completely.

== Why doesn't my Moodle site display the time and date correctly? ==

Each language requires a specific language code (called a '''locale''' code) to allow dates to be displayed correctly. The language packs contain default standard codes, but sometimes these don't work on Windows servers.

You can find the correct locale codes for Windows on these two pages: [http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_crt_language_strings.asp Language codes] and [http://msdn.microsoft.com/library/default.asp?url=/library/en-us/vclib/html/_crt_country_strings.asp Country/region] codes (e.g. "esp_esp" for spanish)

These new locale codes can be entered on the Administration >> Configuration >> [[admin/config|Variables]] page, where they override the ones in the currently chosen language pack.

==I receive this error "500:Internal Server Error"==
You'll get this error message if there is a syntax error in your .htaccess or httpd.conf files. You will also see this error if your server does not support .htaccess files, especially if it is running PHPsuexec. Also, you may have a directive in your .htaccess or httpd.conf files which are not compatible with your web server version.

==How do I uninstall Moodle?==
'''Moodle package installation''': If you have downloaded a Moodle package, simply uninstall using your system commands. On Windows PCs, you should access the Control Panel -> Add/Remove Programs. Select the package name and click Change or Remove Programs.

'''Webhost/manual installation''': If you have installed Moodle manually or have installed onto a webhost, follow these steps:
*Delete the moodle database using this mysql command (or delete using your mysql client, e.g. PHPMyAdmin):
<pre>sql>DROP DATABASE moodle;</pre>
:In the above example replace 'moodle' with the name of the moodle database you created when installing.
*Delete the moodledata directory. If you, or your users, have uploaded materials into this directory take a copy of these before deleting this directory.
*Delete the moodle directory itself. This will delete all of the moodle PHP script files.

==How do I upgrade Moodle? Do I just overwrite the files?==
Do not overwrite files, it may cause strange errors. You should read the [[Upgrade]] documentation before proceeding.

==Migrating Moodle to a new site or server==
Migrating Moodle means that you have to move the current installation to a new server, and so may have to change IP addresses or DNS entries. To do this you will need to change the $CFG->wwwroot value in the config.php on the new server. You will also have to change any absolute links stored in the database backup file (before restoring the file on the new server) either using the admin/replace.php script, your text editor or another "search and replace" tool, e.g. sed. For more details see the [[Moodle_migration | Moodle Migration]] page.

==Fatal error allowed memory size exhausted. How do I increase my php memory limit?==
You will sometimes see an error message something like this:
Fatal error: Allowed memory size of 67108864 bytes exhausted
(tried to allocate xx bytes) in /var/www/moodle/yyyy.php
This error means that the php memory_limit value is not enough for the php script. The memory_limit value is the "allowed memory size" - 64M in the example above (67108864 bytes / 1024 = 65536 KB. 65536 KB / 1024 = 64 MB). You will need to increase the php memory_limit value until this message is not shown anymore. There are two methods of doing this.
*On a hosted installation, add the following line to your .htaccess file (or create one in the moodle directory if it does not already exist):
php_value memory_limit <value>M
Example: php_value memory_limit 40M
*If you have your own server with shell access, edit your php.ini file (make sure it's the correct one by checking in your phpinfo output) as follows:
memory_limit <value>M
Example: memory_limit 40M
Remember that you need to restart your web server to make changes to php.ini effective. An alternative is to disable the memory_limit by using the command ''memory_limit 0''.

----
==Why does my new installation display correctly on the server, but when I view it from a different machine, styles and images are missing?==
In the installation instructions, one of the suggested settings for 'webroot' is 'localhost'. This is fine if all you want to do is some local testing of your new Moodle installation. If, however, you want to view your new installation from another machine on the same local area network, or view your site on the internet, you will have to change this setting:
*For local testing, 'localhost' is fine for the webroot.
*If you want to test your site from other machines on the same local area network (LAN), then you will have to use the private ip address of the serving machine, (e.g. 192.168.1.2/moodle) or the network name of the serving computer (e.g. network_name_of_serving_machine/moodle) as the web root. Depening on your LAN setup, it may be better to use the network name of the computer rather than its (private) ip address, because the ip address can and will change from time to time. If you don't want to use the network name, then you will have to speak to your network administrator and have them assign a permanent ip address to the serving machine.
*Finally, if you want to test your new installation across the internet, you will have to use either a domain name or a permanent (public) ip address/moodle as your web root.

[[Category:FAQ]]
[[Category:Installation]]

[[es:FAQ Instalación]]
[[fr:FAQ d'installation]]
[[nl:Installatie FAQ]]
[[ja:インストールFAQ]]
[[ru:Установка FAQ]]

Complete install packages

2007-07-30T19:33:03Z

Matm: /* Start the Moodle installation */

Complete install packages are available from [http://download.moodle.org/ Moodle Downloads], located on a tab for each of the operating systems. The packages are designed for new installations on a server or standalone computer. Please note the standard distributions only contain the Moodle code.

This document provides instructions for using the Windows packages. Separate instructions are available for [[Complete Install Packages for Mac OS X|Mac OS X packages]].

For installation on a Windows 2000 or Windows 2003 server it is good practice to perform a manual install (see the manual installation section in [[Windows_installation|Windows Installation]]).

After installing the Windows package, note that there are other downloads (e.g. additional modules and plugins) that may involve more customization of configuration files.

The complete install packages allow Moodle to be installed, along with the prerequisites that includes a web server, database and scripting language (Apache, MySQL and PHP in this case). Several versions of the complete install package are available. The instructions on the download page provide guidance on which version is likely to be most suitable.

Note: The latest complete install package version components, may not be backwardly compatible. Always check version compatibility of each component if you intend to develop materials on a later version of Moodle than the version installed on your "main" Moodle site. In short, complete install packages are designed for first time install on a "clean" machine.

==System requirements==
+ 256 MB RAM (minimum), 512 MB RAM (recommended)
+ 160 MB free Fixed Disk (more space will be needed depending on user uploads)
+ Windows 98/ME (minimum)
+ Windows NT/2000/XP (recommended)

==Install complete package ==
===First install Apache, MySQL, PHP===
Step 1: [http://download.moodle.org/ Download] the packed-zip file from Moodle.

Step 2: Unpack the file you downloaded to a drive or partition of your choice and let the default create C:\moodle or W:\moodle or something like this. It is a good practice at this point to rename the C:\moodle to something like C:\Web or C:\Xampplite. There will be a folder under this called \moodle.

Step 3: Start the "setup_xampp.bat" in the top folder. This will configure Xammplite. Note: XAMPP makes no entries in the windows registry and no settings for the system variables.

Step 4: Now you are ready to start your webserver. Use the Xampp_start file. Once the Xampp_start is open, don't close it, use Xampp_stop for that purpose. Xampp program(s) control both Apache and MySQL. Alternatively you can individually start and stop Apache and MySQL with the bat files found in the top folder such as C:\xampplite .

Step 5: Start your browser and type <nowiki> http://127.0.0.1 or http://localhost </nowiki>in the address bar. You will either start your first time Moodle installation or if it is already install enter your homepage.

===Start the Moodle installation===

*In your web browser, type the path to the folder containing the Moodle files in te address bar – in this example it’s <nowiki>http://localhost/moodle, in the above example it would be http://localhost</nowiki>.

*The initial install page is displayed.

[[image:Xampp24.gif|thumb|center|600px]]

*Choose your preferred language (English is used in this example) and click the “Next” button.

*A diagnostic report is displayed – hopefully it will look like this, if not you may need to address some issues.

[[image:Xampp25.gif|thumb|center|600px]]

*Click the “Next” button to continue.

*The paths for your Moodle installation are shown – accept the ones that are shown on your screen.

[[image:Xampp26.gif|thumb|center|400px]]

*Click the “Next” button to continue.

*What you enter in the "Host Server" field depends on what you intend to use the new Moodle installation for.
If you are just going to use it for local testing, then use 'localhost'.
If you are going to test the new installation on a LAN, and will be accessing it from other machines on that LAN, then put the private IP address or network name of the serving machine, followed by a forward slash and moodle: Host Server - 192.168.1.1/moodle
If you are going to test the installation on the internet, then you will need to put the public ip address followed by a forward slash and moodle: your_ip_address/moodle or you can put your domain name here instead.

*In the next fields, we enter the database settings. The fields are populated with some suggested values.

*We strongly recommend you place a user name and password in this screen. (Don't forget them).

*DO NOT USE THE “ROOT” USER WITHOUT A PASSWORD FOR PRODUCTION INSTALLATIONS AS THIS CREATES A SECURITY VULNERABILITY
[[image:Xampp27.gif|thumb|center|600px]]

*When the fields have been populated, click the “Next” button to continue.

*Provided the Moodle folder is writable, a message confirming the configuration has been completed will be displayed.

[[image:Xampp29.gif|thumb|center|400px]]

*Click the “ Continue” button to proceed.

The Moodle copyright / licence notices are displayed.

[[image:Xampp30.gif|thumb|center|400px]]

*Click the “Yes” button to continue. In most cases this will be followed by a series of screens that have a continue button on the bottom. In Moodle 1.8 look for the "Unattended" checkoff box. This will press the continue button at the bottom of the screen for you. This process stops with Admin user profile settings which needs to be filled out, then goes to the site setting page.

===Congratulations===
This finishes the installation of a complete package. Type http://localhost in your browser and Moodle will open.

Your next task will be to configure Moodle. See [[Administrator_documentation#Configuration]] or [[Administrator_documentation]]. Don't worry, it is easy to change any of the settings once Moodle is running.

== Installing Apache and MySQL as services==
(This is only for NT4 | Windows 2000 | Windows XP operating systems)

\xampplite\apache\apache_installservice.bat =
==> Install Apache 2 as service

\xampplite\apache\apache_uninstallservice.bat =
==> Uninstall Apache 2 as service

\xampplite\mysql\mysql_installservice.bat =
==> Install MySQL as service

\xampplite\mysql\mysql_uninstallservice.bat =
==> Uninstall MySQL as service

==> After all Service (un)installations, please restart your system!

== Security matters (A MUST READ!)==

As mentioned before, XAMPP is not meant for production use but only for developers
in a development environment. The way XAMPP is configured is to be open as possible
and allowing the developer anything he/she wants. For development environments this
is great but in a production environment it could be fatal. Here a list of missing security
in XAMPP:

The MySQL administrator (root) has no password.
The MySQL daemon is accessible via network.
phpMyAdmin is accessible via network.
Examples are accessible via network.

To fix most of the security weaknesses simply call the following URL:

http://localhost/security/

The root password for MySQL + phpMyAdmin and also a XAMPP directory protection can be established here.

==Apache Notes==

You should use the apache_start and apache_stop bat files to start and stop apache from running.

==MySQL notes==

(1) The MySQL server can be started by double-clicking (executing)
mysql_start.bat. This file can be found in the same folder you installed
xampp in, most likely this will be C:\xampplite\.
The exact path to this file is X:\xampplite\mysql_start.bat, where
"X" indicates the letter of the drive you unpacked xampp into.
This batch file starts the MySQL server in console mode. The first
intialization might take a few minutes.

Do not close the DOS window or you'll crash the server!
To stop the server, please use mysql_shutdown.bat, which is located in the same
directory.

(2) To use the MySQL Daemon with "innodb" for better performance,
please edit the "my" (or "my.cnf") file in the /xampplite/mysql/bin
directory or for services the c:\my.cnf for windows NT/2000/XP.
In there, activate the "innodb_data_file_path=ibdata1:30M"
statement. Attention, "innodb" is not recommended for 95/98/ME.

To use MySQL as Service for NT/2000/XP, simply copy the "my"
/ "my.cnf" file to C:\my, or C:\my.cnf. Please note that this
file has to be placed in C:\ (root), other locations are not permitted. Then
execute the "mysql_installservice.bat" in the mysql folder.

(3) MySQL starts with standard values for the user id and the password. The preset
user id is "root", the password is "" (= no password). To access MySQL via PHP
with the preset values, you'll have to use the following syntax:
mysql_connect("localhost","root","");
If you want to set a password for MySQL access, please use of mysqladmin.
To set the passwort "secret" for the user "root", type the following:
\xampplite\mysql\bin\mysqladmin -u root password secret

After changing the password you'll have to reconfigure phpMyAdmin to use the
new password, otherwise it won't be able to access the databases. To do that,
open the file config.inc.php in \xampplite\phpmyadmin\ and edit the
following lines:

$cfg['Servers'][$i]['user'] = 'root'; // MySQL user
$cfg['Servers'][$i]['auth_type'] = 'http'; // HTTP authentificate

So first the 'root' password is queried by the MySQL server, before phpMyAdmin
may access.

Have a lot of fun! Viel Spaß! Bonne Chance!

==See also==
*[[Administrator_documentation]] for links that help configure Moodle.
*[[Installation guide - Moodle for Windows on a USB Memory Stick]]
*Return to [[Windows installation]]
*[https://docs.moodle.org/en/Windows_installation_using_XAMPP#Troubleshooting Troubleshooting]if you are running Skype.
*[[Installing_AMP]] lots of XAMPP stuff. XAMPP stands for XP, Apache, MySQL,PHP and Perl. XAMPPlite does not include Perl. MAMP stands for Mac, Apache, MySQL and PHP.
*[[Complete Install Packages for Mac OS X]]

[[es:Paquetes_para_Instalaci%C3%B3n_Completa]]

[[Category:Administrator]]
[[Category:Installation]]

Installing AMP

2006-02-17T23:33:48Z

Matm: /* Windows */

Moodle is written in a scripting language called PHP and stores most of its data in a database. The recommendrd database is MySQL. Before installing Moodle you must have a working PHP installation and a working database to turn your computer into a functional web server platform. These packages can be tricky to set up for average computer users, so this page has been written to try to make this process as simple as possible for different platforms:

== Hosting Service ==

Unfortunately hosting services vary quite a lot in the way they work. Some will even install Moodle for you.

Most will offer a web-based control panel to control your site, create databases and set up cron. Some may also offer terminal access via ssh, so that you can use the command shell to do things.

You should work your way through the [[Installing Moodle|Installation guide]] and take each step at a time. Ask your hosting provider if you get stuck.

== Mac OS X ==

The easiest way to do this is use the Apache server that Apple provides, and add PHP and MySQL using Marc Liyanage's packages. Both of the pages below come with good instructions that we won't duplicate here:

* '''PHP''': Download from here: http://www.entropy.ch/software/macosx/php/
* '''MySQL''': Download here: http://www.entropy.ch/software/macosx/mysql/

Once these are installed the standard [[Installing Moodle|Installation guide]] should be fairly straightforward.

== Red Hat Linux ==

You should install all available RPM packages for Apache, PHP and MySQL. One package that people frequently forget is the php-mysql package which is necessary for PHP to talk to MySQL.

Once these are installed the standard [[Installing Moodle|Installation guide]] should be fairly straightforward.

A more detailed walkthrough is here: [[RedHat Linux installation]]

== Windows ==

The easiest way to do this is use one of the complete install packages available from the Downloads page at Moodle.org. [http://download.moodle.org/?lang=en]

Alternatively, use a package like EasyPHP that bundles all the software you need into a single Windows application. Note that the EasyPHP 1.7 uses the following somewhat older versions:

# apache 1.3.27 (currnet relases is 2.2.0)
# php 4.3.3 (current stable release is 5.1.3)
# mysql 4.0.15 (current release is 5.0.18)
# phpmyadmin 2.5.3

It should be noted that these are not the current releases. Also many menus for EasyPHP are still in French. Here are the steps from start to finish for this approach, XAMPP or Windows 2003 with IIS: [[Windows installation]].

==Testing PHP==
Once you have installed your web server and PHP you should be able to create a file (for example phpinfo.php in the document root) with the following in it:

<?phpinfo()?>

You should be able to open this file in a web browser by going to to the URL '''localhost/phpinfo''' and see a web page that has PHP status information in it such as [[phpinfo|this]].

==See also==

*[[Installing Moodle]]
*[[Installation FAQ]]
*[[Upgrading Moodle]]
*[[Debian GNU/Linux installation]]

[[Category:Core]]
[[Category:Administrator]]
[[Category:Installation]]

Installing AMP

2006-02-17T23:31:30Z

Matm: /* Windows */

Moodle is written in a scripting language called PHP and stores most of its data in a database. The recommendrd database is MySQL. Before installing Moodle you must have a working PHP installation and a working database to turn your computer into a functional web server platform. These packages can be tricky to set up for average computer users, so this page has been written to try to make this process as simple as possible for different platforms:

== Hosting Service ==

Unfortunately hosting services vary quite a lot in the way they work. Some will even install Moodle for you.

Most will offer a web-based control panel to control your site, create databases and set up cron. Some may also offer terminal access via ssh, so that you can use the command shell to do things.

You should work your way through the [[Installing Moodle|Installation guide]] and take each step at a time. Ask your hosting provider if you get stuck.

== Mac OS X ==

The easiest way to do this is use the Apache server that Apple provides, and add PHP and MySQL using Marc Liyanage's packages. Both of the pages below come with good instructions that we won't duplicate here:

* '''PHP''': Download from here: http://www.entropy.ch/software/macosx/php/
* '''MySQL''': Download here: http://www.entropy.ch/software/macosx/mysql/

Once these are installed the standard [[Installing Moodle|Installation guide]] should be fairly straightforward.

== Red Hat Linux ==

You should install all available RPM packages for Apache, PHP and MySQL. One package that people frequently forget is the php-mysql package which is necessary for PHP to talk to MySQL.

Once these are installed the standard [[Installing Moodle|Installation guide]] should be fairly straightforward.

A more detailed walkthrough is here: [[RedHat Linux installation]]

== Windows ==

The easiest way to do this is use one of the complete install packages available from the Downloads page at Moodle.org. [http://download.moodle.org/?lang=en]

Alternatively, use a package like EasyPHP that bundles all the software you need into a single Windows application. Note that the EasyPHP 1.7 uses the following somewhat older versions:

# apache 1.3.27 (currnet relases is 2.2.0)
# php 4.3.3 (current stable release is 5.1.3)
# mysql 4.0.15 (current release is 5.0.18)
# phpmyadmin 2.5.3

It should be noted that these are not the current releases. Also many menus for EasyPHP are still in French. Here are the steps from start to finish for this approach or XAMPP: [[Windows installation]].

==Testing PHP==
Once you have installed your web server and PHP you should be able to create a file (for example phpinfo.php in the document root) with the following in it:

<?phpinfo()?>

You should be able to open this file in a web browser by going to to the URL '''localhost/phpinfo''' and see a web page that has PHP status information in it such as [[phpinfo|this]].

==See also==

*[[Installing Moodle]]
*[[Installation FAQ]]
*[[Upgrading Moodle]]
*[[Debian GNU/Linux installation]]

[[Category:Core]]
[[Category:Administrator]]
[[Category:Installation]]