# Question Engine 2:Numerical tolerances

Analysis of the various tolerances for numerical and calculated question type answer see (MDL-31837)

In grading a numerical response, the (student) numerical value is compared to the answer numerical value.

This comparison allow a tolerance that is associated with the answer and that can be expressed in various ways.

For calculated question answers the tolerance can be of 3 different types: relative, nominal or geometric.

# Limits of real numbers in PHP

The following is from http://www.php.net/manual/en/language.types.float.php

The size of a float is platform-dependent, although a maximum of ~1.8e308 with a precision of roughly 14 decimal digits is a common value (the 64 bit IEEE format).

Warning Floating point precision

Floating point numbers have limited precision. Although it depends on the system, PHP typically uses the IEEE 754 double precision format, which will give a maximum relative error due to rounding in the order of 1.11e-16. Non elementary arithmetic operations may give larger errors, and, of course, error propagation must be considered when several operations are compounded.

Additionally, rational numbers that are exactly representable as floating point numbers in base 10, like 0.1 or 0.7, do not have an exact representation as floating point numbers in base 2, which is used internally, no matter the size of the mantissa. Hence, they cannot be converted into their internal binary counterparts without a small loss of precision. This can lead to confusing results: for example, floor((0.1+0.7)*10) will usually return 7 instead of the expected 8, since the internal representation will be something like 7.9999999999999991118....

So never trust floating number results to the last digit, and do not compare floating point numbers directly for equality. If higher precision is necessary, the arbitrary precision math functions and gmp functions are available.

Since the internal representation is in base 2, 0 and 1 will have the same exponent so the precision ( i.e. init_get('precision') should have the same value.?

....

# 1,9 version

```
function get_tolerance_interval(&$answer) {
// No tolerance
if (empty($answer->tolerance)) {
$answer->tolerance = 0;
}
// Calculate the interval of correct responses (min/max)
if (!isset($answer->tolerancetype)) {
$answer->tolerancetype = 2; // nominal
}
// We need to add a tiny fraction depending on the set precision to make the
// comparison work correctly. Otherwise seemingly equal values can yield
// false. (fixes bug #3225)
$tolerance = (float)$answer->tolerance + ("1.0e-".ini_get('precision'));
switch ($answer->tolerancetype) {
case '1': case 'relative':
/// Recalculate the tolerance and fall through
/// to the nominal case:
$tolerance = $answer->answer * $tolerance;
// Do not fall through to the nominal case because the tiny fraction is a factor of the answer
$tolerance = abs($tolerance); // important - otherwise min and max are swapped
$max = $answer->answer + $tolerance;
$min = $answer->answer - $tolerance;
break;
case '2': case 'nominal':
$tolerance = abs($tolerance); // important - otherwise min and max are swapped
// $answer->tolerance 0 or something else
if ((float)$answer->tolerance == 0.0 && abs((float)$answer->answer) <= $tolerance ){
$tolerance = (float) ("1.0e-".ini_get('precision')) * abs((float)$answer->answer) ; //tiny fraction
} else if ((float)$answer->tolerance != 0.0 && abs((float)$answer->tolerance) < abs((float)$answer->answer) && abs((float)$answer->answer) <= $tolerance){
$tolerance = (1+("1.0e-".ini_get('precision')) )* abs((float) $answer->tolerance) ;//tiny fraction
}
$max = $answer->answer + $tolerance;
$min = $answer->answer - $tolerance;
break;
case '3': case 'geometric':
$quotient = 1 + abs($tolerance);
$max = $answer->answer * $quotient;
$min = $answer->answer / $quotient;
break;
default:
error("Unknown tolerance type $answer->tolerancetype");
}
$answer->min = $min;
$answer->max = $max;
return true;
}
```

# 2,0 actual code

```
class qtype_numerical_answer extends question_answer {
/** @var float allowable margin of error. */
public $tolerance;
/** @var integer|string see {@link get_tolerance_interval()} for the meaning of this value. */
public $tolerancetype = 2;
public function __construct($id, $answer, $fraction, $feedback, $feedbackformat, $tolerance) {
parent::__construct($id, $answer, $fraction, $feedback, $feedbackformat);
$this->tolerance = abs($tolerance);
}
public function get_tolerance_interval() {
if ($this->answer === '*') {
throw new coding_exception('Cannot work out tolerance interval for answer *.');
}
// We need to add a tiny fraction depending on the set precision to make
// the comparison work correctly, otherwise seemingly equal values can
// yield false. See MDL-3225.
$tolerance = (float) $this->tolerance + pow(10, -1 * ini_get('precision'));
switch ($this->tolerancetype) {
case 1: case 'relative':
$range = abs($this->answer) * $tolerance;
return array($this->answer - $range, $this->answer + $range);
case 2: case 'nominal':
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) *
max(1, abs($this->answer));
return array($this->answer - $tolerance, $this->answer + $tolerance);
case 3: case 'geometric':
$quotient = 1 + abs($tolerance);
return array($this->answer / $quotient, $this->answer * $quotient);
default:
throw new coding_exception('Unknown tolerance type ' . $this->tolerancetype);
}
}
public function within_tolerance($value) {
if ($this->answer === '*') {
return true;
}
list($min, $max) = $this->get_tolerance_interval();
return $min <= $value && $value <= $max;
}
}
```

# 2,0 vs 1,9 differences

## case relative

The math treatment is equivalent as

$this->tolerance = abs($tolerance);

## case geometric

The math treatment is equivalent as

$this->tolerance = abs($tolerance);

## case nominal

The use of

pow(10, -1 * ini_get('precision')) * max(1, abs($this->answer));

The 'tiny fraction' part of the tolerance is not related to the answer when the answer is less than 1.

We should get back to the 1,9 equivalent code first by removing the max function

```
case 2: case 'nominal':
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer);
return array($this->answer - $tolerance, $this->answer + $tolerance);
```

The 1,9 code has a special treatment of the answer == 0 .

Should we keep the code ?

### Answer is 0

What is the precision of 0 ? It cannot be 0 , it only can be the precision of the digital part of the number i.e. pow(10, -1 * ini_get('precision')). As a mather of fact the exponent part of 0 is the same one as 1 so its precision is the same. We should not forget that the "turning point" (positive to negative) of real number exponent value is 1 not 0.

So to get the "tiny number" we should multiply the pow(10, -1 * ini_get('precision')) by $this->answer except when $this->answer == 0 when it should be 1.

### Tolerance is 0

The 0 value should be used as is, the php should only retain the "tiny part" as a final result;

$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

### Tolerance < pow(10, -1 * ini_get('precision'))

The tolerance i.e 1e-26 should become the number that control the process and define the "tiny number". Such a case could result when the 0 answer comes from something like 0,3333333e-24-1/3e24 i.e. So

$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

### Tolerance >pow(10, -1 * ini_get('precision'))

all the other cases

$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;

### Answer is NOT 0

The "tiny part" should mostly be calculated as pow(10, -1 * ini_get('precision'))*$answer

### Tolerance is 0

The 0 value should be used as is, the php should only retain the "tiny part" as a final result;

$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

### Tolerance < pow(10, -1 * ini_get('precision'))*$this->answer

The tolerance i.e 1e-26 should become the number that control the process and define the "tiny number". Such a case could result when the 0 answer comes from something like 2,3333333e-24-1/3e24 i.e. So

$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

### Tolerance >=pow(10, -1 * ini_get('precision'))*$this->answer

all the other cases

$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))** $this->answer ;

# Code flow proposal

This code reflects the discussion although a more concise version could be built. The proposal is somehow different from the 1,9 version as it includes the precision of the tolerance in the range.

```
if ($this->answer == 0.0){
if($this->tolerance == 0.0){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
} else if( $this->tolerance < pow(10, -1 * ini_get('precision')){ // the 0
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;
}else {
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
}
} else {
if($this->tolerance === 0.0){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer) ;
} else if( $this->tolerance < pow(10, -1 * ini_get('precision')* abs($this->answer)){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * $this->tolerance ;
}else {
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer);
}
}
```

## Tim's idea

I am wondering whether it works just change

```
max(1, abs($this->answer));
```

in the 2.0 code to

```
max($this->tolerance, abs($this->answer), pow(10, -1 * ini_get('precision')));
```

The code might be nicer if we define $epsilon = pow(10, -1 * ini_get('precision')) at the start of the function.

Finally, I would like to know what tests you propose to add to question/type/numerical/tests/answer_test.php in order to demonstrate that everything is working properly. In particular, what tests would you like to add that will fail with the current code, but which will pass once we have fixed this bug?

--Tim Hunt 00:53, 16 August 2012 (WST)

## Pierre's comment

Thanks for your code expert feedback...

I am using a simple calculated question on 2,0 or a calculated question on 1,9 to test the various combinations with different answers and tolerances.

The range (Min- Max ) is available directly.

At first tests your proposal seems to be working.

I will report here the results of various tests on 1,9, actual 2, your proposal and the more linear code.

This should help us to choose which tests to use.

Pierre Pichet 12:38, 16 August 2012 (WST)

# Tests results

## Testing Min-Max

Types

- R :relative
- N :nominal
- G : geometric

Answer | Tolerance | Type | 1,9 Min | 1,9 Max | Code Min | Code Max | 2,0 Min | 2,0 Max | Tim Min | Tim Max |
---|---|---|---|---|---|---|---|---|---|---|

0 | 0 | N | 0 | 0 | -1.0E-14 | 1.0E-14 | -1.0E-14 | 1.0E-14 | -1.0E-28 | 1.0E-28 |

0 | 1e-24 | N | 0 | 0 | -1.0E-24 | 1.0E-24 | -1.0000000001E-14 | +1.0000000001E-14 | -1.0001E-24 | 1.0001E-24 |

1e-20 | 0 | N | 9.9999999999999E-21 | 1.0E-20 | 9.9999999999999E-21 | 1.0E-20 | -9.99999E-15 | 1.000001E-14 | 9.9999999999999E-21 | 1.0E-20 |

1e-20 | 1e-24 | N | 9.999E-21 | 1.0001E-20 | 9.999E-21 | 1.0001E-20 | -9.999990001E-15 | 1.0000010001E-14 | 9.9989999E-21 | 1.00010001E-20 |

0 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 1 |

0 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 1 |

0 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 1 |

0 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 2 | cell 3 | cell 1 | cell 1 |

The tests on master, Tim proposal and code proposal are done on the same installation just changing the branchs (master) created for the tests so everything else is constant.

The Code proposal give similar results to the 1,9 version with a more precise handling of 0 as answer.

TODO : Review the Code proposal , use as suggested $epsilon, ....

## HOW PHP handle 0

Here the results from different 0 answers using Tim's code

0.0 = 0 Correct answer : 0 inside limits of true value 0.0 Min: -1.0001E-24 --- Max: 1.0001E-24 0+0.00000000000000000000001 = 1.00000000e-23 Correct answer : 1.00000000e-23 inside limits of true value 0+0.00000000000000000000001 Min: 9.9999E-24 --- Max: 1.00001E-23

1-1.000000000000000001 = 0 Correct answer : 0 inside limits of true value 1-1.000000000000000001 Min: -1.0E-28 --- Max: 1.0E-28

1-1.0000000000000001 = 0 Correct answer : 0 inside limits of true value 1-1.0000000000000001 Min: -1.0E-28 --- Max: 1.0E-28

1-1.000000000000001 = -1.11022302e-15 ERROR Correct answer : -1.11022302e-15 outside limits of true value 1-1.000000000000001 Min: -1.1102230246253E-15 --- Max: -1.1102230246251E-15

In the first one when PHP decode the answer, it detect the first 0 than detect the second term as 1e-23 which is not zero.

When PHP add the 2 numbers, the answer is 1.00000000e-23

In the second answer he convert

1 to 1.000000000000

and

1.0000000000000001 to 1.000000000000 i.e. to its epsilon limit

which when substacted will give 0.0

This could occur only on calculated questions as there is no equation in numerical.

## TESTING GRADING

In order to test how the code handle responses I create a question with 3 answers

- A dummy answer {x} to have at least one parameter used in an answer.
- Answer 0 Nominal Tolerance 0 Grade 100%
- Answer 0 Nominal Tolerance 1e-20

# Final Code proposal

```
if ($this->answer == 0.0){
if($this->tolerance == 0.0){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
} else if( $this->tolerance < pow(10, -1 * ini_get('precision')){ // the 0
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;
}else {
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
}
} else {
if($this->tolerance === 0.0){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer) ;
} else if( $this->tolerance < pow(10, -1 * ini_get('precision')* abs($this->answer)){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * $this->tolerance ;
}else {
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer);
}
}
```

This "linear" code can be rewritten differently :

```
if ($this->answer == 0.0){
if( $this->tolerance < pow(10, -1 * ini_get('precision')){ // the 0
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;
}else {
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
}
} else {
if( $this->tolerance < pow(10, -1 * ini_get('precision')* abs($this->answer)){
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * $this->tolerance ;
}else {
$tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer);
}
}
```

If we add

$epsilon = pow(10, -1 * ini_get('precision'));

then