Preg question type

Note: Even though this plugin is currently (February 2020) listed as available for Moodle branches 2.3 to 3.1 only, the plugin Stats tab shows that half of the more than 30 sites that have this plugin installed, are Moodle branches from 3.2 to 3.8. This officially not-supported branches can have a working regex plugin, but they will not have the 'Authoring tools' available. The plugin developers are working in order to update this plugin and fix the JavaScript problem that causes this issue. If you enable a DEVELOPER level of debugging, You may see some warnings that do not represent any danger to your server. They should be fixed in future releases.

Preg is a question type that uses regular expressions (regexes) to check student's responses (though you can use it without regexes for its hinting features). Regular expressions give vast capabilities and flexibility to both teachers when making questions and students when writing answers to them. The first section should guide you to using this docs; please use it with discretion. More details about regex syntax can be found at http://www.nusphere.com/kb/phpmanual/reference.pcre.pattern.syntax.htm. There are many good regex manuals; they will not be repeated here.

Ways to use Preg questions and this docs

I don't (want to) know anything about regular expressions but next word (character) hinting seems useful

Then you can use Preg question type just as Short-answer, but with advanced hinting, without any knowledge about regular expressions. To do this, you need to choose:

Notation => Moodle shortanswer
Engine => Finite state automata
Exact matching => Yes

After that, you can just copy answers from you short-answer questions. You may want to read the section about hinting to understand more about hinting settings. See the Preg question type examples for several illustrated, step-by-step, K-12 and University level usage examples.

I have a vague knowledge of regular expressions, but I want to use pattern matching

If writing regular expressions is hard for you, but you want to use their strength as patterns, authoring tools may help you a lot to create your questions. The tools show you the meaning of your regex in different ways: internal structure of the expression (syntax tree), visual path of matching (explaining graph) and a text description. They also allow you to test you regex against several strings and see if it works as expected. Experiment and play with your regexes; see corresponding changes in the authoring tools, and eventually you'll get the regex you want.

Read the section on authoring tools, then (probably after some experimenting with tools on your own) read the section about understanding regular expressions (this is optional, but may be interesting and help a lot). You should also read a section about question working to better understand various settings and how they affects you questions.

I can make some effort to learn regular expressions well and be able to do anything they allow

Well, you don't know regexes but want to understand them and create complex expressions easily. Then, instad of blunt trying, you should better spend some time and effort reading and understanding this section. Then read slightly about authoring tools and use them to experiment creating regexes. With these tools you can see if you really understand them well and that they behave as expected. The syntax tree may be especially useful when you try to get the right meaning of precedence and arity. After you understand the principles of regexes well, read the sections about question working and regular expression reference (to know your possibilities; you needn't bother understanding or remembering them all - just look there periodically for something new to learn). Now you should be able to write regexes without much use of the authoring tools, except for the testing tool to test your expressions.

I know regular expressions well enought to write them on my own without further guidance

You should read about question working to understand various settings and question behaviour under them. You also may be interested in regex testing in the authoring tools section. Finally, regular expression reference may be of some use for you.

Supported Moodle branches and translations

Even though this plugin is currently (April 2018) listed as available for Moodle branches 2.3 to 3.1 only, its stats page shows that half of the more than 50 sites that have this plugin installed, are Moodle branches from 3.2 to 3.4. This officially not-supported branches can have a working regex plugin, but they will not have the authoring tools available. The plugin developers are working in order to update this plugin and fix the JavaScript problem that causes this issue. If you enable a DEVELOPER level of debugging, You may see some warnings that do not represent any danger to your server. They will be fixed in future releases:

Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP

As all Moodle plugins, the preg plugin is initially available in English. Currently there are Russian and Mexican Spanish translations available, but you can make other translations in AMOS.

Installation

Download/install from the Moodle plugins database.
Be sure to install all the plugin dependencies.

If you want to use the authoring tools , you will need to install the Graphviz Graph Visualization Software and set the 'path to dot'.

To find the path to doc in a Mac, open a terminal window and type:

which dot

- Then type that path (eg, /opt/local/bin/dot) in 'Site administration / ▶︎ Server / ▶︎ System paths ' Path to dot

.

How Preg questions work

Basically, this question type is an extended version of the Short-Answer question type. It extends its features in several different ways (you could use them in almost any combination):

Pattern matching - using regular expressions you can create powerful patterns describing possible students answers
Hinting - when students are stuck doing the question, you may allow them to ask for a next correct word (lexeme) or a character (with possible penalty).

Settings affecting question work

Sets the case sensitivity for all regular expressions you specify as answers. Note that you can also set the case sensitivity for regex parts.

Exact matching affects the question in the following way:

Yes: the entire student's response, from the first to the last letter, should match your regular expression
No: student's response can just contain a part that matches your regex: for example, if the correct answer is "whole" then "the whole enchilada","the whole shebag", the whole nine yards, "the whole world" and "the whole truth" will all be correct student responses.

You still can set some of your regexes to match the whole student's response using special regex syntax.

Notations specify the "language" of your answers.

Regular expression: a usual notation for regular expression. Precisely it is Perl-compatible regex dialect. You may write regex on multiple strings for better reading - line breaks will be ignored.
Regular expression (extended): useful for really complex regexes. It is similar to the PHP 'x' modifier. It will ignore any unescaped whitespaces in you regexes, that are not inside character classes (use \s instead) - so that you may freely format you regexes with spaces. It will also ignore line breaks with one useful exception: everything after '#' character untill the end of string is treated as commentary (# should not be escaped and should not be inside a character class).
Moodle shortanswer: use it to avoid regex syntax at all: just copy answers from you short-answer questions. The '*' wildcard is supported. By choosing the FA engine you can get access to the hinting features. You can skip all that is said on regexes there, but be sure to read the hinting section to understand various settings you can alter to configure you question hinting behaviour. See the Preg question type examples for useful hints and working examples.

Matching engine specifies the program module that performs the regex matching. There is no 'best' matching engine - it depends on the features you want to use. Engines have different stability and offer different features to use.

PHP preg extension: should be used when you don't need hinting and other engines are rejecting you expressions as too difficult, or if you encounter bugs in them. It is based on the native PHP preg_ functions. It supports 100% perl-compatible regex features, it is very stable and thoroughly tested. But it doesn't support partial matching, so (unless we storm PHP developers to add support of partial matching) there is no hinting. However it supports subpattern capturing. Choose it when you need complex regex features that other engines don't support.
Finite state automata (FA): can be used to perform hinting for your students. FA engine is a custom PHP code, it allows many (but not all) regex features and is thoroughly tested (it passes all tests from AT&T testregex suite and most tests from PCRE testinput1, testinput4 suite for the features it supports, which means quite much), but still may contain bugs in rare cases. Unsupported features for now are lookaround assertions and some types of conditional subpatterns.

Hinting

Hinting is supported by the FA engine in adaptive and interactive behaviours.

Partial matching

Hinting starts with partial matching. By partially correct response we understand a string that starts with correct characters (matching your regex) but on some character the match breaks. Assume you entered the regex

 "are blue, white(,| and) red"

and a student answered

 "they are blue, vhite and red"

In this situation the partial match is

 "are blue, "

Note that the regex is unanchored ("Exact match" is set to "No") so the match may not start with the first character of the student's response (like in the example above: "they " is skipped). While using just partial matching the student will see the correct and incorrect parts:

 they are blue, vhite and red

General hinting rules

Preg question type doesn't add hinted characters to the student's response (unlike the REGEXP Regular Expression Short-Answer question type), showing it separately instead for a number of reasons:

It is student's responsibility whether he wants to add hinted character to the his response (and some more possibly).
It slightly facilitates thinking about a hint, since when the response is modified it is too easy to repeatedly press hint, which is not usually a desirable behaviour.

When possible, hinting chooses a character that leads to the shortest path to complete the match. Consider this response to the previous regular expression:

 are blue, white; red

There are two possible hint characters: "," or " " (leading to the " and" path). The question will choose "," because it leads to the shortest path to complete the match, while " " leads to the path 3 characters longer.

It is possible that not all regular expressions will give 100% grade. Consider you added an expression for students with bad memory:

 are white(,| and) red

with 60% grade and feedback about forgetting blue. You may not want hinting to lead student to the response

  are white, red

if he entered

  are white, oh I forgot the other colors.

Hint grade border controls this. Only regular expressions with the grade greater or equal than the hint grade border will be used for partial matching and hinting. If you set hint grade border to 1, only 100% grade regular expression will be used for hinting, if you set it to 0,5 regular expressions with grades from 50% to 100% will be used for hinting and 0%-49% would not. Regular expressions not used for hinting work only when they have a full match with the student response.

Next character hinting

When next character hinting is available, student will have the hint next character button by pressing which he receives one next correct character, highlighted by background coloring:

 they are blue, wvhite and red

You should typically set the hint penalty more than usual question penalty, because they are applied separately: usual penalty for an attempt without hinting, while hint penalty for an attempt with hinting.

Next lexeme (word) hinting

Lexeme means an atomic part of a language. For natural language a word, a number, a punctuation mark (or group of marks like '?!' or '...') are lexemes. For a programming language it can be a keyword, a variable name, a constant, an operator. Note that spaces are usually not considered to be lexemes, but separators between them, since they don't have any particular meaning.

Next lexeme hint will show student either completion of the current lexeme (if partial match ends inside it) or next one (if student complete the current lexeme). Like

  are blue

or

  are blue,

or

  are blue, white

Preg question type, since the 2.3 release, allows usage of next lexeme hinting using the formal languages block. You should choose the language in which you expect a response for you question, since lexeme borders are different for different languages. For now it supports these languages (there will be more):

simple English: English language scanner recognize words, numbers and punctuation marks;
C/C++ language: a programming language C (or C++);
printf language: a special language for formatting strings in C/C++ programming language, you will have it disabled probably.

The site administrator can control what languages are available to the teachers, to avoid confusion. See the settings of the block "Formal languages" in the plugin settings menu.

Note: The simple English language option can be used with other human written languages, but you would be wise to check that your regexes work as you expect before the students sit for a quiz.Currently (march 2018), it has been tested with Mexican Spanish (that uses a decimal point) and Russian (that uses a decimal comma).

Note that "lexeme" typically isn't the word you would like your students to see on the hinting button. Each language define their own word for it. You can enter another word in the question description, if you don't like default ones.

Subpattern capturing and feedback

Any pair of parentheses in a regex are considered as a subpattern and when matching the engine remembers (captures) not only the whole match, but its parts corresponding to all subpatterns. Subpatterns can be nested. If a subpattern is repeated (i.e. have quantifier), only last match of all repeats will be captured. If you want to change order of evaluation without defining a subpattern to capture (which will speed up processing), you should use (?: ) instead of just ( ). Lookaround assertions don't create subpatterns.

Subpatterns are counted from left to right by opening parentheses. Precisely 0 is the whole regex, 1 is first subpattern etc. You can insert them in the answer's feedback using simple placeholders: {$0} will be replaced by the whole match, {$1} by the first subpattern value etc. That can improve the quality of you feedbacks. Placeholders won't work in the general feedback because different answers can have different number of subpatterns.

Note: The following text assumes that the site language uses the point character as the Decimal separator.

Let's look at a regex defining a decimal number with optional integral part:

[+\-]?([0-9]+)?\.([0-9]+)

It has two subpatterns: first capturing integral part, second - fractional part of the number. If you wrote the feedback:

The number is: {$0} Integral part is {$1} and fractional part is {$2}

Then a student entered

123.34

He will see

The number is: 123.34 Integral part is 123 and fractional part is 34

If no integral part is given, {$1} will be replaced by empty string. There is no way (for now) to erase "Integral part is" under that circumstances - the placeholder syntax may become complex and prone to errors.

Looking for missing and misplaced things

The Regular Expression Short-Answer question type (REGEXP) created by Joseph Rezeau has a missing words feature, allowing to define an answer that will work when something is absent in the answer (and give an appropriate feedback to the student).

Similar effect can be achieved with negative assertions combined with anchoring the matching start. The regular expression to look for the missing word necessary would be

 ^(?!.*\bnecessary\b.*)

where

(?!.*\bnecessary\b.*) is a negative lookahead assertion, that allows matching only if there is no word necessary ahead of some point in the string;
^ is an assertion too, that anchores the match to the start of the response (otherwise there would be places in response after the word "necessary", where matching is possible even if the word is present).

In case if the description is difficult to you, just surround regexp to be missing with ^(?! and ). Don't try '--' syntax, that is specific to the Regular Expression Short-Answer question type (REGEXP) created by Joseph Rezeau!

You can also have a rough search for misplaced words (it will actually work only if anything else is correct) using syntax like this:

  (?!<I\s+)\bam\b(?!\s+victor)

This expression catches misplaced "am" in the sentence "I am victor" by first looking for "am" doesn't have "I" before it ("(?!<I\s+)" part) and then "victor" after it ("(?!\s+victor)" part). "\s+" allows any number of spaces between words. If you want to catch the first (last) word (punctuation mark, etc) - then you should place simple assertions for start/end of string ("^" or "$") instead of words in related assertions. For instance to look for misplaced "I" you should write something like

  (?!<^)\bI\b(?!\s+am)

which looks for "I" that is not preceded by start of the string and not followed by "am".

Note, that if you have several answers to catch missing and misplaced things, only one will actually work for any given student response.

Since the Preg 2.3 release you can combine hints and catching missing words. But you should be sure that the answers that look for missing things (and other to give specific feedback) have a fraction (grade) lower, that hint grade border (see #Hinting). You actually don't want to generate hints for these answers, as they don't define a correct situation, so it's not problem but a feature.

Templates

Preg 2.8 introduces a new feature called Templates.

A template is a more convenient and semantic way to write frequently used patterns. Template is a regular expression comment that is changed on special regex before execution. Some templates can be parametrized, any regular expression can be used as parameter value.

Simple template => (?###template_name)
Parametrized template => (?###template_name<)param1(?###,)param2(?###,)...paramN(?###>)

Templates can be used for making regex shorter and easier to understand. For particulary complex regexes (usually ones with parentheses) you may consider using extended notation to be able to use line breaks and spaces for better formatting regex.

For now, templates are hard-coded in the Preg question type, but there are plans to add support for custom user templates in the next releases.

At the moment the following templates are available:

Template name	Parameters	Description	Example
word	None	One or more 'word' characters (letters, digits and underscore).	((?###word)\s+)+ will match any number of words with any number of spaces between them
integer	None	Optional sign + or -, followed by one or more digits	(?##integer) will match any integer value
parens_req	The text you want to see in parentheses	Something in at least one pair of correctly closed round parentheses	(?###parens_req<)a(?###>) will match (a) ((a)) and (((((a)))))
parens_opt	The text you want to see in parentheses	Something optionally placed in any number of pairs of correctly closed round parentheses	(?###parens_opt<)a(?###>) will match a (a) and (((a)))
brackets_req	The text you want to see in brackets	Something in at least one pair of correctly closed square brackets	(?###brackets_req<)(?###word)(?###>) will match [abc] [[cat]]
brackets_opt	The text you want to see in brackets	Something optionally placed in any number of pairs of correctly closed square brackets	(?###brackets_opt<)(?###word)(?###>) will match cat [dog] [[[Fido]]]
custom_parens_req	1. Pattern for the opening parenthesis 2. Text inside custom parentheses 3. Pattern for the closing parenthesis	This template is similar to the parens_req, but allows you to specify custom parentheses (possibly by more than one character)	(?###custom_parens_req<)<(?###,)a(?###,)>(?###>) will match <a> <<<a>>>
custom_parens_opt	1. Pattern for the opening parenthesis 2. Text inside custom parentheses 3. Pattern for the closing parenthesis	This template is similar to the parens_req, but allows you to specify custom parentheses (possibly by more than one character)	(?###custom_parens_opt<)/\(?###,)(?###word)(?###,)\/(?###>) will match /something/ ///word///

One templates can be used inside other as parameters. For example you can write

 (?###parens_opt<)(?###word)(?###>)

It will match strings "a", "(a)", "(((((long_word_in_many_parens)))))" and so on.

Authoring tools

Authoring tools are there to help you write, test and understand you regexes. The tools can show you the meaning of written regex (and its parts), and test it.

The authoring tools are activated by pressing the "edit" icon (cogwheel) near the regex field.

Note: When you click on the cogwheel in order to open the authoring tools, the popup window title is 'Regex constructor'

If you want to fully use the Authoring Tools in the Regex constructor, you need to have installed the Open Source Graphviz Graph Visualization Software, and set the 'path to dot' (in Windows it is likely C:\Program Files (x86)\Graphviz2.38\bin\dot.exe) specified in your Moodle server in ' Site administration ► Server ► System paths ► Path to dot ':

The Graphviz Graph Visualization Software is necessary in order to draw the Syntax Tree and the Explaining Graph tools.

There are four tools available in the Regex constructor:

syntax tree: shows you the inner structure of regular expressions
explaining graph: shows you how your expression will work in a graphical way
description: formulates the meaning of your expression in English
testing tool: allows you to enter strings and see how they match your regex

Installation note and known technical issues

To have syntax tree and explaining graph authoring tools working, you (or your site admin) have to install the Graphviz Graph Visualization Software on the server and fill the 'pathtodot' setting on you Moodle installation at ' Site administration ► Server ► System paths ► Path to dot '. The 'path to dot' in Windows is likely C:\Program Files (x86)\Graphviz2.38\bin\dot.exe.

Graphviz is used to draw pictures for you. Be sure to use Graphviz 2.36 or newer, as earlier versions had a bug in svg output which led to incorrect pictures.

The syntax tree and explaining graph may not work correctly in old Opera versions - for some reason the images are not updated on user actions. Fortunately, there's a newer version 16 for Windows, which works with authoring tools pretty well. On Linux you will have to use something else.

Regular expression area

Here you can edit your regular expression. Clicking on "Show" sends the regex to all tools - syntax tree, explaining graph, description and testing results will be updated. "Save" closes the authoring tools form and saves the regex and test strings in the main question editing form. "Cancel" closes the authoring tools form and discards all changes made there.

You can select part of regular expression there, and corresponding parts of syntax tree, explaining graph, description and matched part of the strings will be highlighted. It is possible to select part of regex text, that doesnt correspond with a logically completed part of regular expression. In that case you selection will be widened to the nearest logically completed part.

Matching options

There you can change options affecting matching - matching engine, regex notation, exact matching, and case sensitivity.

Matching engine will change the code performing matching - you could use Testing tool to see if it suits your needs.
Regular expression notation will change the way regexes are written - all instruments will show you the difference how this notation is interpreted.
Case sensitivity will affect basic case sensitivity of expression, the results you can see in the explaining graph - case insensitive nodes are gray, case sensitive - white.
Exact matching will add new parts to your regexp to ensure the entire student's response will match it. These added parts will be shown on gray background in the tools - see the picture below.

Panning and zooming of pictures

Syntax tree and Explaining graph tools generate pictures that can be too large. So, starting from Preg 2.6, these tool allow you easy pan and zoom features.

To Pan image, press left mouse button on the free area (not on the nodes - it will select them), and drag the mouse around without releasing the pressed button. On the Explaining graph you should set Rectangle selection mode off, in order to pan; since in the rectangle selection mode, pressing the mouse button starts drawing a rectangle.

To Zoom the image, use the mouse wheel while the mouse pointer is over an image.

Syntax tree

As explained above, regular expressions, like all expressions, are trees of operators and operands. Syntax tree shows the inner structure of expression graphically: what is inside what. This will be the most useful tool if you know how to understand regular expressions or if you are learning to do this.

If you don't understand operators and precedence conception well, it may have little meaning to you. But it is still useful to find out, where you need parentheses: compare the trees for ab+ (a) and (ab)+ (b) on the picture below.

The tree will show you names and numbers of all subpatterns, so you can check their numerations - and back references to it.

The part of the expression that you have selected is shown in a green rectangle. You can select nodes of the tree by pressing on them when the 'Collapsing mode' check box is unchecked.

Starting from Preg 2.6, the sytax tree tool have a 'Collapsing mode' option, since a syntax tree can be quite large and you usually need only a part of it. When 'Collapsing mode' is on, pressing on a tree node collapses all it's child nodes into one single ellipsis (see image below). Pressing a collapsed node again will un-collapse it. Switching off collapsing mode doesn't un-collapse nodes, but it will allow you to return to the usual Selection mode. On the following picture you can see two collapsed nodes with tooltip, showing the collapsed part of a regular expression over one of them.

Explaining graph

The graph shows how regular expression works. Its nodes are matched characters, its edges show paths throught the nodes from the beginning to the end.

Oval nodes represent individual characters, character sequences (so that graph isn't extremly big) or single special character classes (in which case they change line colour). Complex character classes are shown as rectangles. Simple assertions are checked between nodes, so they are written on the edges.

$graph for regex ^\dabc[!,0-9]$$

Dotted rectangles shows you repeated parts of you expression.

$graph for regex \d*$

Solid line rectangles show you subpatterns. When expression is matched, it remembers which part of the string matched each subpattern. You could insert it in the feedback or use in backreference in expression. If you do not need to remember subpatterns, you may use (?: ) instead of ( ) parentheses, that will speed up matching.

Green rectangle shows you selected part of expression. Switching on "Rectangle selection mode" you can select part of the graph using rubber rectangle and see corresponding part of regex selected on all instruments (including regular expression text).

Description

Description try to formulate a sentence, describing you how expression is supposed to work. Selected part of the expression will be shown by yellow background color.

Testing tool

You can enter a set of strings there, one per line. These strings will be matched against your expression. You'll see coloured strings, showing which parts of your strings matched the expression, so you can test if it works as you expected. You will also see green check marks for the strings that match entire regular expressions (and will be graded for that regex) and red crosses for the strings that don't give full match. PHP preg matcher can't show partial matches, so it only shows full matches or nothing (to not mislead you that entire string is wrong).

If you selected a part of regex, you will be able to see what part of strings matches that part (usually in yellow color, but that may depend on you theme). FA matcher will show that for any part of regex, PHP preg matcher - only for capturing subpatterns.

The strings for testing will be saved in database, if you save regex (they will be lost if you close window with "cancel" button) and (later) question.

Understanding regular expressions

Understanding expressions in general

Regular expressions - as any expressions - are just a bunch of operators with their operands. Don't worry - you all learned to master arithmetic expressions from chilhood and regular ones are just as easy - if you look at them from the right angle. Learn (or recall) only 4 new words - and you are a master of regexes with very wide possibilities. Let's go?

Look at a simple math expression: x+y*2. There are two operators: '+' and '*'. The operands of '*' are 'y' and '2'. The operands of '+' are 'x' and the result of 'y*2'. Easy?

Thinking about that expression deeper we can find that there is a definite order of evaluation, governed by operator's precedence. The '*' has a precedence over '+', so it is evaluated first. You can change the evaluation order by using parentheses: (x+y)*2 will evaluate '+' first and multiply the result by 2. Still easy?

One more thing we should learn about operators is their arity - this is just the number of operands required. In the example above '+' and '*' are binary operators - they both take two operands. Most of arithmetic operators are binary, but the minus has also the unary (single operand) form, like in this equation: y=-x. Note that the unary and binary minuses work differently.

Now any expression are just a lego game, where you set a sequence of operators with correct number of operands for each (arity), taking heed of their evaluation order by using their precedence and parentheses. Arithmetic expressions are for evaluating numbers. Regular expressions are for finding patterns in strings, so they naturally use other operands and operators - but they are governed by the same rules of precedence and arity.

Regular expressions

Regular expressions is a powerful mechanism for searching in strings using patterns. So their operands are characters or a sets of characters, that is allowed in particular position. A is a regular expressions that matches a single character 'A'. The operators in regular expressions define a way to combine individual characters in the pattern: sequence (concatenation operator), alternative and repeating (it is called quantifier). The concatenation is such a simple operator, that it doesn't have any character for it at all - just write some characters in sequence, and they'll be concatenated. But it is still have precedence, so that the question can see, did you want to repeat a single character or a sequence of them. Alternative is written as vertical bar. There are many form of quantifiers - most commonly used are question mark (repeat zero or one times), asterisk (zero or more times) and plus (one or more times). You may specify mininimum and maximum number of repeats in curly braces - this is a quantifier too.

The special characters that define operators should be escaped when used as operands - preceded by a backslash. Mathematical expressions never have escaping problems since their operands (numbers, variables) are constructed from different characters than operators (+,- etc), but when constructing a pattern for matching you should be able to use any character as an operand.

Character classes allows you to specify several possible characters for one place. They can be defined in many different ways: by enumeration of characters in square brackets [as3], by ranges in square brackets [a-z], by special sequences (\d means any digit, \W anything except a letter, digit and underscore, [[:alpha:]] any letter etc). An important type of operand is a simple assertions: they allow you to test some conditions - start of the string ^, end of the string $ or word border \b.

You could find a list and more examples of operands and operators in reference section.

Precedence and order of evaluation

A quantifier has precedence over concatenation and concatenation has precedence over alternative. Let's look what it means:

quantifiers over concatenation means that quantifiers are executed first and will repeat only a single character if used without parentheses:
- "many times*" matches "manytime" followed by zero or more "s";
- "(many times)*" matches "many times" zero or more times - changing the previous regex by using parentheses allows us define a string repetition;
concatenation over alternative means that you can define multi-character alternatives without parentheses (for single character alternatives it's better to use character classes, not the alternative operator):
- "first|second|third" matches "first" or "second" or "third";
- "(first |second |)part" matches "first part" or "second part" or just "part" - typical use of an empty alternative (note that space is in alternative to not require it before just "part");
quantifier over alternative means that you should use parentheses to repeat an alternative set:
- "first|second*" matches "first" or "secon" followed by zero or more "d" like "secondddddd";
- "(first|second)*" matches "first" or "second", repeated zero or more time in any order, like "firstsecondfirstfirst". Note that quantifiers repeat the whole alternative, not a definite selection from it, i.e.:
- "(1|2){2}" matches "11" or "12" or "21" or "22", not just "11" or "22";
- "1{2}|2{2}" matches "11" or "22" only.

An internal structure of regular expression can be viewed well on the syntax tree (authoring tool). The operators that executed first are placed lower on the tree (or to the right on horizontal view), the operator that executed last is the root of the tree. You can compare tree and explaining graphs for the examples above in authoring tools if this section doesn't seems too clear to you. Remember, that "execution" of regular expression operator means linking them in the string: sequental, alternative linking, or repeating.

Anchoring

Anchoring is used to set restrictions on the matching process by using simple assertions:

if a regular expression starts with the ^ the match should start at the start of the student's response;
if a regular expression ends with the $ the match should end at the end of the student's reponse;
otherwise a regex match can be found anywhere inside a student's response.

Note that simple assertions are concatenated with regex and concatenation has precedence over alternative, this makes it's usage slightly tricky:

"^start|end$" will match "start" from the start of the string or "end" at the end of it;
"^(start|end)$" using brackets to match exactly with "start" or "end";
"^start$|^end$" is another way to get exact match (all top-level alternatives are anchored).

If you set the exact matching options to "yes" (which is the default value), the question will add ^ and $ in each regular expression for you (it will not affect subpattern usage). However, you may prefer to use some non-anchored regexes to catch common errors and give feedback and use manually anchored expressions for grading.

Examples of the use of regular expresions as better substitutes for the standard Moodle Short-Answer question type

The feature of character or word hinting makes the preg question type a very powerful and user-friendly tool for formative quizzes:

Vocabulary practice:
- Solar system example
- Medical vocabulary example

Sentence memorization:
- Article of the law example
- Matching the beginning of a large paragraph example

Regular expressions reference

Operands

Here's an incomplete list of operands that define character sets.

Simple characters (with no special meaning) match themselves.
Escaped special characters match corresponding special characters. Escaping means preceding special characters by the backslash "\". For example, the regex "\|" matches the string "|", the regex "a\*b\[" matches the string "a*b[". Backslash is a special character too and should be escaped: "\\" matches "\".
- full list of characters needs escaping \ ^ $ . [ ] | ( ) ? * + { }
- NOTE! when you are unsure whether to escape some character, it is safe to place "\" before any character except letters and digits. Do not escape letters and digits unless you know what you are doing - they get special meaning when escaped and lose it when not.
- If you have too many characters that need escaping in some fragment, you can use \Q ... \E sequence instead. Anything between \Q and \E is treated literally as characters:
  - "\Q^(abc)$\E." matches "^(abc)$" followed by any character - there are NO simple assertions and subpatterns;
  - "\Q^(abc)$." matches "^(abc)$." because there is no "\E" and all characters after "\Q" are treated as literals till the end of the regex.
Dot meta-character (".") matches any possible character (except newline, but students can't enter it anywhere), escape it "\." if you need to match a single dot. Loses it's special meaning inside character class.
Character classes match any character defined in them. Character classes are defined by square brackets. The particular ways to define a character class are:
- "[ab,!]" matches "a", "b", "," or "!";
- "[a-szC-F0-9]" contains ranges (defined by a hyphen between 2 characters) "a-z", "C-F" and "0-9" mixed with the single character "z", it matches any character from "a" to "s", "z", from "C to "F" and from "0" to "9";
- "[^a-z-]" starts with the "^" that means a negative character set: it matches any character except from "a" to "z" and "-" (note that the second hyphen is not placed between 2 characters so defines itself);
- "[\-\]\\]" contains escaping inside a character set: it matches "-", "]" and "\", other characters loose their special meaning inside a character set and can be be not escaped, but if you want to include "^" in a character set it shouldn't be first there;
Escape sequences for common character sets (can be used both inside or outside character classes):
- "\w" for any word character (letter, underscore or digit) and "\W" for any non-word character;
- "\s" for any space character and "\S" for any non-space character;
- "\d" for any digit and "\D" for any non-digit.
Unicode properties are special escape-sequences "\p{xx}" (positive) or "\P{xx}" (negative) for matching specific unicode characters which could be used both inside or outside character classes (the complete list of "xx" variations can be found at found at http://www.nusphere.com/kb/phpmanual/reference.pcre.pattern.syntax.htm):
- "\p{Ll}" matches any lowercase letter;
- "\P{Lu}" matches any non-uppercase letter.
POSIX character classes are used for the same purpose as unicode properties (and complete list of them can be found on the Internet too), but may not work with non-ASCII characters. They are allowed only inside character classes:
- "[[:alnum:]]" matches any alpha-numeric character;
- "[[:^digit:]]" matches any non-digit chararcter.
Simple assertions - they are not characters, but conditions to test, they don't consume characters while matching, unlike other operands (have those meaning only outside character classes):
- "^" matches in the start of the string, fails otherwise;
- "$" matches in the end of the string, fails otherwise;
- "\b" matches on a word boundary, i.e. either between word (\w) and non-word (\W) characters, or in the start (end) of the string if it starts (ends) with a word character;
- "\B" matches not on a word boundary, negative to "\b".

Still, a pattern that matches only one character isn't very useful. So here come the operators that allow us to define an expression that matches strings of several characters.

Operators

Here's a list of the common regex operators:

Concatenation - so simple binary operator that doesn't require any special character to be defined. It is still an operator and has it's precedence, which is important if you want to understand where to use brackets. Concatenation allows you to write several operands in sequence:
- "ab" matches "ab";
- "a[0-9]" matches "a" followed by any digit, for example, "a5"
Alternative - a binary operator that lets you define a set of alternatives:
- "a|b" matches "a" or "b";
- "ab|cd|de" matches "ab" or "cd" or "de";
- "ab|cd|" matches "ab" or "cd" or emptiness (useful as a part in more complex expressions);
- "(aa|bb)c" matches "aac" or "bbc" - using parentheses to outline alternative set;
- "(aa|bb|)c" matches "aac" or "bbc" or "c" - typical usage of the emptiness;
Quantifiers - an unary operator that lets you define repetition of something used as its operand:
- "x*" matches "x" zero or more times;
- "x+" matches "x" one or more times;
- "x?" matches "x" zero or one times;
- "x{2,4}" matches "x" from 2 to 4 times;
- "x{2,}" matches "x" two or more times;
- "x{,2}" matches "x" from 0 to 2 times;
- "x{2}" matches "x" exactly 2 times;
- "(ab)*" matches "ab" zero or more times, i.e. if you want to use a quantifier on more than one character, you should use parentheses;
- "(a|b){2}" matches "aa" or "ab" or "ba" or "bb", i.e. it is a repeated alternative, not a repetition of "a" or "b".

Subpatterns and backreferences

Subpatterns are operators that remember substrings captured by the regex. The simplest way to define a subpattern is to use parentheses: the regex "a(bc)d" contains a subpattern "bc". Subpatterns are numerated from 0 for the whole regex and counted by opening parentheses. That "(bc)" subpattern is the 1st. If we write, say, "a(b(c)(d))e" - there are subpatterns "bcd" which is 1st, "c" which is 2nd and "d" which is 3rd. Subpatterns are usually used with backreferences which, too, have numbers. Backreferences are operands that match the same strings which are matched by the subpatterns with the same numbers. The simplеst syntax for backreferences is a slash followed by a number: "\1" means a backreference to the 1st subpattern. The regular expression "([ab])\1" matches strings "aa" and "bb", but neither "ab" nor "ba" because the backreference should match the same character as the subpattern did. Constider a little example: declaration and initialization of an integer variable in C programming language:

"int ([_\w][_\w\d]*); \1 = -?\d+;" matches, for example, "int _var; _var = -10;". Of course, there can be any number of spaces between "int", variable name etc, so a more correct regex will look like:
"\s*int\s+([_\w][_\w\d]*)\s*;\s*\1\s*=\s*-?\d+\s*;\s*" - this will match, say, " int var2 ; var2=123 ; ". Looks a bit frightning, but it is easier to write this regex once than to try understand it after.

Finally, instead of just numbers, subpatterns and backreferences can have names via a little more complicated syntax:

"(?<name1>...)" means a subpattern with name "name1";
"(?'name2'...)" means a subpattern with name "name2";
"(?P<name3>...)" means a subpattern with name "name3";
"\k<name4>" means a backreference to the subpattern named "name4";
"\k'name5'" means a backreference to the subpattern named "name5";
"\g{name6}" means a backreference to the subpattern named "name6";
"\k{name7}" means a backreference to the subpattern named "name7";
"(?P=name8)" means a backreference to the subpattern named "name8".

This is very useful when you work with complicated regexes and often modify it by adding or removing subpatterns - names stay the same.

Duplicate subpattern numbers and names

There is a useful syntax when combining subpatterns with alternation. If you create a group "(?|...)" than every alternative inside that group will have the same subpattern numeration. Consider the regex "(?|(a(b))|(c(d)))" - there are 2 alternatives with 2 subpatterns in each. Subpatterns "ab" and "cd" are 1st ones, "b" and "d" are 2nd ones.

Subpattern calls

Another way to use a subpattern is to call it. When hitting a subpattern call, the matching engine goes to the beginning of the target subpattern, and then starts to match it over again, until its end (not the end of the whole regex). If a subpattern call is placed outside the subpattern it refers to, it is almost equivalent to copy-pasting the subpattern, except its number (it will stay the same).

The most common usage of subpattern calls is the problem of matching a string in parentheses, allowing for unlimited nested parentheses. Without recursive subpattern calls, it is impossible to handle an arbitrary nesting depth. Note that regex for arbitrary nested parentheses is quite complex and actively using them may get you regexes quite obscure. Preg question type provide several templates to make regexes with parenthesis more readable and easier to write.

The syntax of a subpattern call is:

(?R) recursive call fo the whole pattern
(?n) call subpattern by absolute number
(?+n) call subpattern by relative number
(?-n) call subpattern by relative number
(?&name) call subpattern by name
(?P>name) call subpattern by name
\g<name> call subpattern by name
\g'name' call subpattern by name
\g<n> call subpattern by absolute number
\g'n' call subpattern by absolute number

The first one is explicitly recursive. The rest of the variants cause recursion if placed inside the subpattern they refer to, for example: "a(b(?1)?c)d" contains recursion, "a(bc)(?1)d" does not.

When using the finite state automata engine, subpattern calls behave slightly different from PCRE in that the called subpatterns are NOT treated as atomic groups. Generally the behaviour implemented in Preg question type is more intuitive and helpful. Please take a look at PCRE docs for more information.

Conditional subpatterns

Conditional subpatterns allow to write "if-then-else" alike constructions. Basically a conditional subpattern consists of a condition, a positive branch and an optional negative branch. If there is no explicit negative branch, it is implied to be empty, like (?:). General syntax is: "(?(condition)yes-pattern)" or "(?(condition)yes-pattern|no-pattern)".

The more specific options are:

(?(n)... absolute reference condition - is the n'th subpattern captured?
(?(+n)... relative reference condition
(?(-n)... relative reference condition
(?(<name>)... named reference condition - is the subpattern with the given name captured?
(?('name')... named reference condition
(?(name)... named reference condition
(?(R)... overall recursion condition - if there is no subpattern named 'R', the condition is true if a recursive call to the whole pattern or any subpattern has been made
(?(Rn)... specific group recursion condition - the condition is true if the most recent recursion is into the n'th subpattern
(?(R&name)... specific recursion condition - the condition is true if the most recent recursion is into the subpattern with the given name
(?(DEFINE)... define subpattern for reference
(?(assert)... complex assertion condition - the condition is true if the assert (positive/negative lookahead/lookbehind) matches

At "top level", all these recursion test conditions are false.

The latter type of conditional subpatterns is not yet supported by the FA engine.

Complex assertions

Assertions about some part of the string don't actually go into matching text, but affect the matching occurrence:

positive lookahead assertion "a+(?=b)" matches any number of "a" ending with "b" without including "b" in the match;
negative lookahead assertion "a+(?!b)" matches any number of "a" that is not followed by "b";
positive lookbehind assertion "(?<=b)a+" matches any number of "a" preceeded by "b";
negative lookbehind assertion "(?<!b)a+" matches any number of "a" that is not preceeded by "b".

Local case-sensitivity modifiers

Starting from Preg 2.1 you can set case-(in)sensitivity for parts of your regular expressions by using the standard syntax of Perl-compatible regular expressions:

"(?i)" will turn case-sensitivity off;
"(?-i)" will turn case-sensitivity on.

This affects general case-sensitivity, which is choosen on the question level. So you can make some answers case-sensitive and some not, or even do this for the parts of answers. For example you can set question as "use case" and have a 50% answer starting with "(?i)" to grade lesser when the case doesn't match, but everything else is correct.

When placed in parentheses, local modifiers work up to the closest ")". When placed on the top level (not inside parentheses) they work up to the end of the expression, i.e. with case sensitivity on for the question:

"abc(de(?i)gh)xyz" will have the bold part case-insensitive;
"abc(de)(?i)ghxyz" will have the bold part case-insensitive.

Error reporting

Native PHP preg extension functions only report if there is an error in regular expression or not, so PHP preg extension engine can't tell you much about the error.

FA' engine uses a custom regular expression parser, so it supports advanced error reporting. The are several classes of potential errors:

more than two top-level alternatives in a conditional subpattern "(?(?=f)first|second|third)";
unopened closing parenthesis "abc)";
unclosed opening parenthesis of any sort (subpatterns, assertions, etc) "(?:qwerty";
quantifier without an operand, i.e. at the start of (sub)expression with nothing to repeat "+" or "a(+)";
unclosed brackets of character classes "[a-fA-F\d";
setting and unsetting the same modifier at the same time "(?i-i)";
unknown unicode properties "\p{Squirrel}";
unknown posix classes "[[:hamster:]]";
unknown (*...) sequence "(*QWERTY)";
incorrect character set range "[z-a]";
incorrect quantifier ranges "{5,3}";
\ at end of pattern "ab\";
\c at end of pattern "ab\c";
invalid escape sequence;
POSIX class ouside of a character set "[:digit:]";
reference to unexisting subpattern (abc)\2;
unknown, wrong or unsupported modifier "(?z)";
missing ) after comment "(?#comment";
missing conditional subpattern name ending;
missing ) after (?C;
missing subpattern name ending;
missing backreference name ending;
missing backreference name beginning;
missing ) after control sequence;
wrong conditional subpattern number, digits expected;
assertion or condition expected "(?()a|b)";
character code too big "\x{ffffffff}";
character code disallowed "\x{d800}";
invalid condition (?(0);
too big number in (?C...) "(?C256)";
two named subpatterns have the same name "(?<name>a)(?<name>b)";
backreference to the whole expression "abc\g{0}";
different subpattern names for subpatterns of the same number "(?|(?<name1>a)|(?<name2>b))";
subpattern name expected "(?<>abc)";
\c should be followed by an ascii character "\cй";
\L, \l, \N{name}, \U, and \u are unsupported;
unrecognized character after (?<.

Authors

Idea, design, question type and behaviours code, hinting, error reporting, regular expression testing (authoring tool) - Oleg Sychev.
Regex parsing, FA regex matching engine, matchers testing, backup&restore, unicode support, templates - Valeriy Streltsov.
Assertions support for FA matcher - Elena Lepilkina.
Explaining graph (authoring tool) - Vladimir Ivanov.
Syntax tree (authoring tool) - Grigory Terekhov.
Regex description (authoring tool) - looking for maintainer.
Assertions support - Elena Lepilkina.

We would gladly accept testers and contributors (see the development plans section) - there is still more work to be done than we have time. Thanks to:

Joseph Rezeau for being devoted tester of Preg question type releases and being the original author of many ideas that have been implemented in Preg question type;
Tim Hunt - for his polite and useful answers and commentaries that helped writing this question, also for joint work on extra_question_fields and extra_answer_fields code, that is useful to many question type developers;
Bondarenko Vitaly - for conversion of a vast set of regular expression matching tests.
Dmitriy Pahomov - for been first author of Regex description (authoring tool)

You, too, could aways help us a lot - regardless of the way you use Preg and your capabilities.

The ways to give back

This project is free software, so it's hard to get any feedback. You shouldn't expect to get software which ideally suits you needs without telling anyone about these needs, or encouragement, or some non-difficult support to the authors. Sometimes as little as writing where you work and how you use (or what prevents you from using) Preg question type may help a lot.

This software is considered a scientific project and such things could be really useful and appreciated:

an evidence that the results of our work (i.e. Preg questoin type) are really useful to people and were used in production environment;
a cooperative work to research it's effectiveness for various applications - basically you need to write about how you use Preg and make some survey with you teachers and/or students about it - but it can include co-authoring a conference thesis or a journal article;
cooperating in writing article or help publishing it in English-language journals (information and help in grants for further work is welcome too).

If you consider any way of helping, do not hesitate to write me about it and ask any questions about details. You may receive individual help during such work too (for example, doing cooperative research I may give you tips how to improve you regexes, etc).

I am a high school teacher, researcher and programmer who must do much on his main paid job and have not much free time to spend on developing this question type. If you could help me in some ways, I may be able to spend more time and effort doing this though. Some examples:

publishing a thesis or paper describing your usage of the Preg question I could give reference for would improve rating of the project there and my rating as a researcher/developer, so please publish and let me know the reference if you feel grateful for this software;
if you would take some more work and organise publishing a paper (or at least thesis) with me as co-author, that would help even more - please inform me immediately if you consider this;
if publishing is hard, you could just write me what your organisation is and how you use preg - that'll help and I would be able to better determine what should be done next;
join the testing efforts - there are many settings in the question, and regexes can be quite complex, so it's hard to do all testing by developers themselves.

Development plans

There is no definite shedule or order of the development for those features - it depends on the available time and developers. Many features require complex code to achieve the results. If you want to help us with a specific feature, please contact the question type maintainer (Oleg Sychev) using http://moodle.org messaging.

Templates editor, allowing users to create custom templates
Support for complex assertions
Support for approximate matching to catch typos in answers
Improve a set of authoring tools to make writing regular expressions easier
Add more languages for next lexeme hinting
Develop more help and examples (in progress) for the people that don't know much about regular expressions.

Problems ?

Please post in the Mathematics forum.

Learn more about regular expressions

Regular Expressions Tutorial A complete introduction to the topic.
Regular Expression Test Page Test your regular expressions on a variety of "answers".
preg syntax manual

Ongoing issues that will soon be addressed in the upcoming releases

English language strings improvements

Where it says:

Input your regex here. You'll see corresponding syntax tree, explaining graph and description. Click "Update" to commit changes in the regex.

- it should be

 Input your regex here. You'll see corresponding syntax tree, explaining graph and description. Click "Save changes" to commit changes in the regex

- since "Save changes" is the name of the button that the user must press.

Where it says

If the difference between engines is too hard to you,

- ' it should be

If the difference between engines is too hard for you,

Where it says

Here you can see syntax tree of you expression. Pressing the node of tree marks corresponding subtree, subgraph and corresponding part of description. You can also collapse / expand the nodes by turning on convolution.'

- it should be

Here you can see the syntax tree of you expression. Pressing the node of tree marks the corresponding subtree, subgraph and corresponding part of the description. You can also collapse / expand the nodes by turning on Collapsing mode.

Where it says

Here you can see explaining graph. Pressing the node of the tree marks corresponding subgraph with dark green rectangle. You can also select multiple nodes at once rectangle, enabling mode selection rectangle.

- it should be

Here you can see an explaining graph. Pressing the node of the tree marks the corresponding subgraph with a dark green rectangle. You can also select multiple nodes at once inside a rectangle, by enabling  the Rectangle selection node checkbox.

Where it says

Here you can see description of regular expression. Pressing the node of the tree marks corresponding subgraph marks corresponding part of description with yellow color.

- it should say

Here you can see a description of your regular expression. Pressing the node of the tree highlights the corresponding subgraph and also marks the corresponding part of the description with a yellow color.

Where it says

Here you can input some strings (one per line) to test your regex. After clicking "Check the string(s)" you'll see results on the right: matched parts are green, unmatched parts are red. Icon shows you whether given string matched entire regex (green check mark) or not (reg cross).

- it should say

Here you can input some strings (one per line) to test your regex. After clicking "Check the string(s)", you'll see results on the right: matched parts are green, unmatched parts are red. An icon shows you whether the given string matched the entire regex (green check mark) or not (red cross).

Where it says

lexem

- it should say

lexeme

- See lexeme in the wikipedia.

Where (pathtodotincorrect | qtype_preg) it says

Can't draw {$a->name}: path to dot of graphviz is incorrect or dot can not be executed. Please ask your administrator to check if <a href="http://www.graphviz.com">graphviz</a> is installed and 'pathtodot' option is correct at Administration > Server > System Paths

- it should say

Can't draw {$a->name}: path to dot of graphviz is incorrect or dot can not be executed. Please ask your administrator to check if <a href="http://www.graphviz.com">graphviz</a> is installed and 'pathtodot' option is correct at Administration > Server > System Paths (The 'path to dot' in Windows is likely C:\Program Files (x86)\Graphviz2.38\bin\dot.exe ).

Where (pathtodotempty | qtype_preg) it says

Can't draw {$a->name}: path to dot of graphviz is empty. Please ask your administrator to install <a href="http://www.graphviz.com">graphviz</a> and specify path to it using the 'pathtodot' option at Administration > Server > System Paths

- it should say

Can't draw {$a->name}: path to dot of graphviz is empty. Please ask your administrator to install <a href="http://www.graphviz.com">graphviz</a> and specify path to it using the 'pathtodot' option at Administration > Server > System Paths (The 'path to dot' in Windows is likely C:\Program Files (x86)\Graphviz2.38\bin\dot.exe ).

English language strings inside files different from preg.php

Where it says

simple english

- it should be

simple English

PHP warnings that will be fixed in following versions

Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; qtype_preg_text_and_button has a deprecated constructor in ...\server\moodle\question\type\preg\authoring_tools\preg_text_and_button.php on line 36

Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; MoodleQuickForm_preg_textarea has a deprecated constructor in ...\server\moodle\question\type\preg\authoring_tools\preg_textarea.php on line 37

Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; qtype_preg_collapsible_info_block has a deprecated constructor in ...\server\moodle\question\type\preg\authoring_tools\preg_collapsible_info_block.php on line 36

See Fixing PHP4 constructors for PHP7 for a posible fix for this error.

Moodle Code checker

2016 March 16th: Installing the Moodle code checker plugin and setting the path to check to 'question/type/preg' resulted in : 170986 error(s) and 4517 warning(s).

Link to discussion

The link to discussion currently pointing to https://bitbucket.org/oasychev/moodle-plugins is not user-friendly. It should point to a specific thread about the regex plugin in the Moodle forums at https://moodle.org/mod/forum/view.php?id=752.

Improving the English grammar and adding more images in this Documentation page

This will:
- improve readability
- reduce the risk of errors for newcomers who want to use this plugin
- reduce frustration for non-programmers users of this plugin
- ease the work of Moodle Docs translators

Issues currently detected in this current Docs page are:

This page is too long. It may look better if split into several pages (overview, understanding regular expressions, ...).

- The examples for short-answer replacements have been made. They look OK :)

The trees produced for the examples above for 'ab+ (a)' and '(ab)+ (b)' do not match the pictures currently produced in a Moodle 3.1.11 local server. What are the regex that will be used for these examples?

'ab+ (a)' 
'(ab)+ (b)'

.

Preg question type
Type	question type
Set	https://moodle.org/plugins/browse.php?list=set&id=28
Downloads	https://moodle.org/plugins/qtype_preg
Issues	https://bitbucket.org/oasychev/moodle-plugins/issues?status=new&status=open
Discussion	https://bitbucket.org/oasychev/moodle-plugins
Maintainer(s)	Oleg Sychev

Documentation