Shortanswer question development

Following discussion about regular expressions question types in Moodle Tim Hunt suggested a development of short answer question that allows the teacher to use a number of rules with different types (matches string, not matches string, contains the words, matches regexp an so on). The idea is good, though it needs further discussion.

Abilities

Initial list of types for rules

Feel free to place suggestions there:

• Exact match string
• Partial match string (can be implemented via next option)
• Contains the all/one/several substrings
• Exact match regular expression
• Partial match regular expression

Logical operators

Many teachers would want to combine several conditions in one answer condition using logical operators (for example contains some words and not contains another). We definitely should allow NOT and AND operators. The OR can be omitted as several blanks effectively work as OR (this would simplify interface greatly).

Interface

The actual problem is to get clever interface there, so the question creation would not be too cumbersome in simple cases.

Considering using logical operation to group complex expressions in one answer I think that using simple formal language (as in embedded question now) is better alternative than having many fields for each graded subasnwers.--Oleg Sychev 10:02, 12 November 2008 (CST)

You must type much in shortanswer questions anyway, and the less you'll get distracted by mouse (i.e. dropdowns) the better. Wiki is an example, that people easily adopts simple pseudocodes. We may get some buttons to simplify editing for newbies later (as in MediaWiki now), of someone will help me with it thought.

Right now I think about two possibilities:

1. An 'answer' field will remain, but now it is used mainly to enter the correct answer in user-readable form (and backward compatibility), it may be empty; if it isnt't empty and rules is empty it will be matching against students response just as it is now, however there are better options. The 'rules' textarea field will be added to describe rules for matching questions with types mentioned above using simple pseudocode. If the both rules and answer field are filled, the answer must match with rules. That'll costs us a new field in table, and (probably) some problems with backups and import/export compatibility, which is an area in Moodle where I'm not competent enought - but I think it'll be quite easy to any experieced Moodle developer to help me there.
2. We may just replace 'answer' field with textarea, leaving first string in it for an answer (as it is now and for display to the user) and other for rules. However this'll create some strange things with an empty first strings, where it may be easy to make an error (thought validator will find it).

I would like you opinion about what path to take.

Syntax

The place for syntax proposals.

Compatibility issues

Singular shortanswer questions compatibility

Most existing questions must work as before, and the rest must be automatically upgraded, it's clear enough.

There is no actual problems there, just a handful of choices:

1. The first string without rule type must be handled just as it is now. Two choices await us there:
   1. we may start a rule description with some special character (or sequence). Any string not starting with that character will work just as it's now. This character must be escaped in old syntax question if it will be the first (well, which character is the rarest at the beginning? Dot? Or ^, or some bracket?). Upgrader can take care of this in old questions, the only possible problems will be new questions added without awareness of the new syntax - these problems are sort of inevitable, the users must learned to escape * somewhere when wildcards were introduced, and in present case it'll be more easy as validator will detects most errors of this type;
   2. we may retain 100% compatibility if in new syntax the first line must be empty, but this will be quite strange for the new uses, is this worth it?
2. we may state that any string must starts from the rule type, and create a rule type that work just as now. Than upgrader just prepend that type before any string on existing questions. This'll would be more confusing to people first, but will avoid rare errors with peoples, that don't read help messages about changes in syntax much, mentioned above - they will be aware of it from the start.

I would like community (and Tim's) opinion on what path this must take.

I think that a text analysis shortanswer question type should be a new question type because any additional syntax than the * character will interfere potentially with already created shortanswer questions.

Moodle need to be upward compatible.

We could even create more than one text analysis shortanswer question type in relation to the code or computer code (regexp, basic ) used to analyse the student responses.

Let the user choose the code ( or text analysis shortanswer question type) that he knows best.

I.e. regexp is the kind of code created to limit the typing in those days where code was enter on a teletype.

Efficient for a computer but not the kind of computer I have in my head...

I cannot use it without a verification program. Pierre Pichet 11:50, 12 November 2008 (CST)

Pierre, I hate incompatibily a lot as you can know already. I thought of kind of system, that will be compatible with most of existing questions (and the rest can be automatically upgraded). I'm not fond of having a lot of question types - even with standard questiontypes adding question dropdown is somewhat crowded. Also, there is nothing exceptional in wanting to use several 'codes' in one question (one to catch right answers, another to catch typical error for example - Joseph Rezeau actually tring to do this with his regexes).

Dropping OR allows us create a very simple interface. We replace text field with textarea, every string in it is a rule (fortunately shortanswer doesn't allows line breaks in answers), the rules in one answer are connected with AND. Each rule starts with short (or full - it may replace the short form automaticaly) form of the rule's type (that can be preceeded by 'NOT' or ! for negation), that ends with delimiter, followed by it's parameters.

The sort of interface that allows user write something like this:

{contains:all:}red:green:blue
{!contains:1/}purple/white/black
{not regexp:part}[0-9]

which will means the string which contains all of the words red,green and blue; doesn't contains purple, white or black and numbers too (defined using regexp).

We can keep compatibility in two ways:

1. expect simple string to be of "Exact Match with wildcards" type (that is how it work's now) and start any rule with some rare special character - as { above - (the character must be escaped if the string still starts with it thought - that is if we can't find a character that absolutely can't be a part of an answer now except line break). This will be more familiar to the user (which can still use old syntax), but may leads to mistakes by omission escaping in rare cases (thought validator can easily catch them most of the time).
2. automaticaly prepend every string with type of the rule for "Exact Match with wildcards" during upgrade. This may caught user by surprise first, but then he'll knows the syntax and won't mistakes just because he didn't know about it.

I consider also about dropping wildcard syntax using regex instead (it's not hard to automatically replace strings with wildcards with regexes). It was useful without regexes, but with them it is only a strange additional option. --Oleg Sychev 18:57, 12 November 2008 (CST)

You involve yourself in Moodle with all the energy of your age and I don't want 
to loose the possibilities this open for Moodle.

However Moodle is used by many important schools or universities to surely more 
than 1 million students (40 000 at my own university) and quiz (and questions) are 
involved in the grading process.

For schools the grades are like money for banks, everything should be done to maintain their value.
Short answer has been used in very different situations and we cannot change easily the way it works.
 
There are current limitations to short answer response analysis that can be improved.
They are more important than the simplification of the actual interface that seems to bother you.
I encourage you to work on response analysis more than just setting a list of equivalent response 
in a single line or a text area.
Pierre Pichet 19:38, 12 November 2008 (CST)

Well, Pierre, actually I'm sort of resting there :-), not doing any complicated work. You probably must state you concerns more clear as I can't understand them. The proposed changes don't hurt anything - in almost every situation (say, any answer except starting with { sign, which can be changed) it will work just as before, and the rest will be automatically upgraded. The only possible problem would be the teachers trying to enter new questions starting with { after upgrade and doesn't know that it now must be escaped. But (apart from clearly states it in instructions on question editing page) most of these situations will be handled by validator. I also can't quite understand what you mean by "working more on response analysis" and "setting a list of equivalent options" too. Could you be more clear on the matter?

I am a teacher from the somewhat large University and I know what is the support of big questions bank, don't waste you energy there.--Oleg Sychev 20:20, 12 November 2008 (CST)

Embedded answers compatibility issues

Pierre, you could actually help me there, as you knows embedded answers question type well.

Implementation

The options should be implemented as subclasses of abstract class to make adding new options easier. Does we need to have them as actual plugins (with versions, language packs, installation and so on) or more simple subplugins?

Does we need to store all options (with logical operators) with given subanswer as a whole, or create a new table with one entry per options? Right now I'm against new tables. They add many problems with moving questions to another category and so on. --Oleg Sychev 10:02, 12 November 2008 (CST)