Auto-feedback shortanswer question

Warning: This page is no longer in use. The information contained on the page should NOT be seen as relevant or reliable.

Goal

Develop a question type, that was similar to shortanswer (except that no initial wildcard support is planned, thought it can be added later) which will be able to analyze student answer and automatically generate feedback messages describing errors, which could be useful to the student without revealing correct answer (if necessary).

Such question is very useful in adaptive mode (and in non-adaptive mode too for courses with minimal teacher involvement) whenever you try to teach some language (eithern natural or artificial). If someone knows about works or papers that trying to do similar things please let me know there or on page comments.

Question still will be based (at least from the beginning) on entering a number of correct answers.

Question will be able to handle correct answer with several identical words, but that is discouraged (unless absolutely necessary) beacuse it will lower accuracy of feedback.

External specification draft

Creating a question

Teacher input one or several correct answers (with possible feedback describing particular answer). These answers are tokenised (this process will be called scanning later) to break them down to smaller parts (lexems): words, numbers, separators, operators etc.

Since I couldn't start this work from writing universal scanner similar to the flex etc, it would be good to have pluginable scanners for the first time (e.g. natural language scanner, C++ scanner etc).

Teacher than asked for a names for all lexems, that could be shown to the user without revealing correct answer. Examples of tokenised answer with named lexems:

Function header in C language: void function(int abc, char def)

void: type of returned value
function: function name
(: bracket (or opening bracket for function arguments)
int: first argument type
abc: first argument name
,: argument list separator
char: second argument type
def: second argument name
): bracket (or closing bracket for function arguments)

English language sentence: The cat ate the mouse.

The: definite article
cat: subject
ate: verb
the: definite article
mouse: complement
.: sentence ending point

Could anyone write a natural language example please? I don't know well grammatical categories of English language in English to do so.

There will be probably some settings to determine how a grade should be calculated in case of errors, but they could be defined later.

Types of errors to report

misspelled word (number, another complex lexem)
missing separator
invalid separator
misplaced word (number, etc)
missing word (number, etc)
unwanted word (number, separator, etc)

Examples of error reporting

Correct answer: void function(int abc, char def)

Student's response: funcion (int, chardef)void

Feedback from the question:

please check function name, you are probably misspelled it
first argument name is missing
missing separator between second argument type and second argument name
type of returned value is probably misplaced

Correct answer: The cat ate the mouse.

Student's response: The mouse eat a cat.