Difference between revisions of "Question Engine 2:Numerical tolerances"

Jump to: navigation, search
Line 1: Line 1:
 
Analysis of the various tolerances for numerical and calculated question type answer see (MDL-31837)
 
Analysis of the various tolerances for numerical and calculated question type answer see (MDL-31837)
 
This is a first version [[User:Pierre Pichet|Pierre Pichet]] 20:27, 15 August 2012 (WST)
 
  
 
In grading a numerical response, the (student) numerical value is compared to the answer numerical value.
 
In grading a numerical response, the (student) numerical value is compared to the answer numerical value.

Revision as of 04:39, 16 August 2012

Analysis of the various tolerances for numerical and calculated question type answer see (MDL-31837)

In grading a numerical response, the (student) numerical value is compared to the answer numerical value.

This comparison allow a tolerance that is associated with the answer and that can be expressed in various ways.

For calculated question answers the tolerance can be of 3 different types: relative, nominal or geometric.

Limits of real numbers in PHP

The following is from http://www.php.net/manual/en/language.types.float.php

The size of a float is platform-dependent, although a maximum of ~1.8e308 with a precision of roughly 14 decimal digits is a common value (the 64 bit IEEE format).

Warning
Floating point precision

Floating point numbers have limited precision. Although it depends on the system, PHP typically uses the IEEE 754 double precision format, which will give a maximum relative error due to rounding in the order of 1.11e-16. Non elementary arithmetic operations may give larger errors, and, of course, error propagation must be considered when several operations are compounded.

Additionally, rational numbers that are exactly representable as floating point numbers in base 10, like 0.1 or 0.7, do not have an exact representation as floating point numbers in base 2, which is used internally, no matter the size of the mantissa. Hence, they cannot be converted into their internal binary counterparts without a small loss of precision. This can lead to confusing results: for example, floor((0.1+0.7)*10) will usually return 7 instead of the expected 8, since the internal representation will be something like 7.9999999999999991118....

So never trust floating number results to the last digit, and do not compare floating point numbers directly for equality. If higher precision is necessary, the arbitrary precision math functions and gmp functions are available.

Since the internal representation is in base 2, 0 and 1 will have the same exponent so the precision ( i.e. init_get('precision') should have the same value.?

....

1,9 version

function get_tolerance_interval(&$answer) {
        // No tolerance
        if (empty($answer->tolerance)) {
            $answer->tolerance = 0;
        }
 
        // Calculate the interval of correct responses (min/max)
        if (!isset($answer->tolerancetype)) {
            $answer->tolerancetype = 2; // nominal
        }
 
        // We need to add a tiny fraction depending on the set precision to make the
        // comparison work correctly. Otherwise seemingly equal values can yield
        // false. (fixes bug #3225)
        $tolerance = (float)$answer->tolerance + ("1.0e-".ini_get('precision'));
        switch ($answer->tolerancetype) {
            case '1': case 'relative':
                /// Recalculate the tolerance and fall through
                /// to the nominal case:
                $tolerance = $answer->answer * $tolerance;
                // Do not fall through to the nominal case because the tiny fraction is a factor of the answer
                 $tolerance = abs($tolerance); // important - otherwise min and max are swapped
                $max = $answer->answer + $tolerance;
                $min = $answer->answer - $tolerance;
                break;
            case '2': case 'nominal':
                $tolerance = abs($tolerance); // important - otherwise min and max are swapped
                // $answer->tolerance 0 or something else
                if ((float)$answer->tolerance == 0.0  &&  abs((float)$answer->answer) <= $tolerance ){
                    $tolerance = (float) ("1.0e-".ini_get('precision')) * abs((float)$answer->answer) ; //tiny fraction
                } else if ((float)$answer->tolerance != 0.0 && abs((float)$answer->tolerance) < abs((float)$answer->answer) &&  abs((float)$answer->answer) <= $tolerance){
                    $tolerance = (1+("1.0e-".ini_get('precision')) )* abs((float) $answer->tolerance) ;//tiny fraction
               }
 
                $max = $answer->answer + $tolerance;
                $min = $answer->answer - $tolerance;
                break;
            case '3': case 'geometric':
                $quotient = 1 + abs($tolerance);
                $max = $answer->answer * $quotient;
                $min = $answer->answer / $quotient;
                break;
            default:
                error("Unknown tolerance type $answer->tolerancetype");
        }
 
        $answer->min = $min;
        $answer->max = $max;
        return true;
    }

2,0 actual code

class qtype_numerical_answer extends question_answer {
    /** @var float allowable margin of error. */
    public $tolerance;
    /** @var integer|string see {@link get_tolerance_interval()} for the meaning of this value. */
    public $tolerancetype = 2;
 
    public function __construct($id, $answer, $fraction, $feedback, $feedbackformat, $tolerance) {
        parent::__construct($id, $answer, $fraction, $feedback, $feedbackformat);
        $this->tolerance = abs($tolerance);
    }
 
    public function get_tolerance_interval() {
        if ($this->answer === '*') {
            throw new coding_exception('Cannot work out tolerance interval for answer *.');
        }
 
        // We need to add a tiny fraction depending on the set precision to make
        // the comparison work correctly, otherwise seemingly equal values can
        // yield false. See MDL-3225.
        $tolerance = (float) $this->tolerance + pow(10, -1 * ini_get('precision'));
 
        switch ($this->tolerancetype) {
            case 1: case 'relative':
                $range = abs($this->answer) * $tolerance;
                return array($this->answer - $range, $this->answer + $range);
 
            case 2: case 'nominal':
                $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) *
                        max(1, abs($this->answer));
                return array($this->answer - $tolerance, $this->answer + $tolerance);
 
            case 3: case 'geometric':
                $quotient = 1 + abs($tolerance);
                return array($this->answer / $quotient, $this->answer * $quotient);
 
            default:
                throw new coding_exception('Unknown tolerance type ' . $this->tolerancetype);
        }
    }
 
    public function within_tolerance($value) {
        if ($this->answer === '*') {
            return true;
        }
        list($min, $max) = $this->get_tolerance_interval();
        return $min <= $value && $value <= $max;
    }
}

2,0 vs 1,9 differences

case relative

The math treatment is equivalent as

$this->tolerance = abs($tolerance);

case geometric

The math treatment is equivalent as

$this->tolerance = abs($tolerance);

case nominal

The use of

pow(10, -1 * ini_get('precision')) * max(1, abs($this->answer));

The 'tiny fraction' part of the tolerance is not related to the answer when the answer is less than 1.

We should get back to the 1,9 equivalent code first by removing the max function

case 2: case 'nominal':
                $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer);
                return array($this->answer - $tolerance, $this->answer + $tolerance);

The 1,9 code has a special treatment of the answer == 0 .

Should we keep the code ?

Answer is 0

What is the precision of 0 ? It cannot be 0 , it only can be the precision of the digital part of the number i.e. pow(10, -1 * ini_get('precision')). As a mather of fact the exponent part of 0 is the same one as 1 so its precision is the same. We should not forget that the "turning point" (positive to negative) of real number exponent value is 1 not 0.

So to get the "tiny number" we should multiply the pow(10, -1 * ini_get('precision')) by $this->answer except when $this->answer == 0 when it should be 1.

Tolerance is 0

The 0 value should be used as is, the php should only retain the "tiny part" as a final result;

 $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

Tolerance < pow(10, -1 * ini_get('precision'))

The tolerance i.e 1e-26 should become the number that control the process and define the "tiny number". Such a case could result when the 0 answer comes from something like 0,3333333e-24-1/3e24 i.e. So

 $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

Tolerance >pow(10, -1 * ini_get('precision'))

all the other cases

 $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;

Answer is NOT 0

The "tiny part" should mostly be calculated as pow(10, -1 * ini_get('precision'))*$answer

Tolerance is 0

The 0 value should be used as is, the php should only retain the "tiny part" as a final result;

 $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

Tolerance < pow(10, -1 * ini_get('precision'))*$this->answer

The tolerance i.e 1e-26 should become the number that control the process and define the "tiny number". Such a case could result when the 0 answer comes from something like 2,3333333e-24-1/3e24 i.e. So

 $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ;

Tolerance >=pow(10, -1 * ini_get('precision'))*$this->answer

all the other cases

 $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))** $this->answer ;

Code flow proposal

This code reflects the discussion although a more concise version could be built. The proposal is somehow different from the 1,9 version as it includes the precision of the tolerance in the range.

if ($this->answer == 0.0){
         if($this->tolerance == 0.0){
               $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
         } else if( $this->tolerance <  pow(10, -1 * ini_get('precision')){ // the 0
               $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision'))* $this->tolerance ; 
         }else {
              $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) ;
         }
   } else {
         if($this->tolerance === 0.0){
               $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * $this->answer ;
         } else if( $this->tolerance <  pow(10, -1 * ini_get('precision')* abs($this->answer)){ 
               $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * $this->tolerance ; 
         }else {
              $tolerance = $this->tolerance + pow(10, -1 * ini_get('precision')) * abs($this->answer);
         }     
   }

Tim's idea

I am wondering whether it works just change

max(1, abs($this->answer));

in the 2.0 code to

max($this->tolerance, abs($this->answer), pow(10, -1 * ini_get('precision')));

The code might be nicer if we define $epsilon = pow(10, -1 * ini_get('precision')) at the start of the function.

Finally, I would like to know what tests you propose to add to question/type/numerical/tests/answer_test.php in order to demonstrate that everything is working properly. In particular, what tests would you like to add that will fail with the current code, but which will pass once we have fixed this bug?

--Tim Hunt 00:53, 16 August 2012 (WST)

Pierre's comment

Thanks for your code expert feedback...

I am using a simple calculated question on 2,0 or a calculated question on 1,9 to test the various combinations with different answers and tolerances.

The range (Min- Max ) is available directly.

At first tests your proposal seems to be working.

I will report here the results of various tests on 1,9, actual 2, your proposal and the more linear code.

This should help us to choose which tests to use.

Pierre Pichet 12:38, 16 August 2012 (WST)