Quiz statistics calculations

Moodle 2.0

General issues

Quizzes that allow multiple attempts

For quizzes that allow multiple attempts, by default the report should only include data from the first attempt by each student. (The data for subsequent attempts probably does not satisfy the assumptions that underlie item analysis statistics.) However, there should be an option 'Include data from all attempts', which should have a disclaimer that this may be statistically invalid either near it on screen, or possibly in the help file. (For small data sets, it may be better to include all data.)

Using the first attempt also avoids problems caused by each attempt builds on last.

Within the analysis, when multiple attempts per student are included, each attempt is treated as an independent attempt. (That is, we pretend each attempt was by a different student.)

Adaptive mode

Adaptive mode does not pose a problem. Item Analysis just supposes that each item in the test returns a score, and these scores are added up to get the test score. Therefore Item Analysis does not care about adaptive/non-adaptive mode. However, just to be clear, for an adaptive question, the score used in the calculation is the final score for the item, including penalties.

Certainty based marking

Similarly, should CBM, and/or negative scoring for multiple choice questions be implemented, we just use the final item score in the calculations, making sure that the formulae are not assuming that the minimum item score is zero.

Incomplete attempts

There is an issue about what you do when not all students have answered all questions. Depending on how you handle these missing items, you distort the statistics in different ways.

There are basically two reasons why a student may not have answered a particular question:

• they may have chosen to omit it, or
• they may have run out of time, if the test is timed. In this case, omitted questions tend to be towards the end of the test.

Available approaches for handling this are:

1. treat omitted items as having a score of 0.
2. exclude any attempt with missing scores from the analysis.
3. Analyse each question separately - and when analysing a particular question, just include students who answered that question in the analysis.

I think we should implement 1 for now. This is how Moodle currently works - a blank answer is marked and receives a score of zero, and so is indistinguishable from a wrong answer. Item analysis is most important for summative tests, and the result of that question being in the test is that it gave that student a contribution of 0 marks towards their final score.

Random questions

In a quiz with random questions not all students will have attempted the same set of questions. To account for this, the analysis below distinguishes between positions in the test, and test items.

Notation used in the calculations

Note: The only way I can think of to write this document with all the maths is to type TeX notation like $$x_i$$ even though MoodleDocs does not currently render it nicely. Maybe one day it will. In the mean time, if you want to see things nicely formatted, you will have to copy them to a Moodle with a working TeX filter, or copy and paste the following magic into your browser's Address bar and hit enter (or get the [http://www.mathtran.org/wiki/index.php/MathTran_bookmarklet MathTran bookmarklet):

javascript:(function(){function searchWithinNode(node,re){var pos,imgnnode,middlebit,endbitskip=0;if(node.nodeType==3){pos=node.data.search(re);if(pos>=0){middlebit=node.splitText(pos);endbit=middlebit.splitText(RegExp.lastMatch.length);imgnode=document.createElement(%22img%22);imgnode.src=%22http://www.mathtran.org/cgi-bin/mathtran?tex=%22 + encodeURI(middlebit.data).replace(%22+%22,%22%252B%22);middlebit.parentNode.replaceChild(imgnode,middlebit);}}else if(node.nodeType==1&& node.childNodes){for (var child=0; child < node.childNodes.length; ++child){searchWithinNode(node.childNodes[child], re);}}}searchWithinNode(document.body, /\$\$(.*?)\$\$/);})();

We have a lot of students $$s \in S$$, who have completed at least one attempt on the quiz.

The test has a number of positions $$p \in P$$.

The test is assembled from a number of items $$i \in I$$.

Because of random questions, different students may have recieved different items in different positions, so $$i(p, s)$$ is the item student $$s$$ received in position $$p$$.

Let $$I_s$$ be the set of items that student $$s$$ saw. Let $$S_i$$ be the set of students who attempted item $$i$$.

Each position has a maximum and minimum possible contribution to the test score, $$x_p(min)$$ and $$x_p(max)$$. At the moment in Moodle, $$x_p(min)$$ is always zero, but we cannot assume that will continue to be the case. $$x_p(max)$$ is database column quiz_question_instances.grade.

Then, each student achieved an actual score $$x_p(s)$$ on the item in position $$p$$. So $$x_p(min) \le x_p(s) \le x_p(max)$$.

$$x_p(s)$$ should be measured on the same scale as the final score for the quiz. That is, scaled by quiz_question_instances.grade, but that is already how grades are stored in mdl_question_states.

We can also think of the student's score on a particular item $$x_i(s)$$. However, in this case, the score should be measured out of the Default question grade for this question. Also, there is a $$x_i(max)$$ ( = Default question grade) and $$x_i(min)$$ (currently, zero, but if we allow negative marking, that will change).

$$x_{i(p, s)}(s) = x_p(s)\frac{x_i(max)}{x_p(max)}$$.

Each student has a total score

$$\displaystyle T_s = \sum_{p \in P} x_p(s)$$

Similarly, there are the maximum and minimum possible test scores

$$\displaystyle T_{max} = \sum_{p \in P} x_p(max)$$

and

$$\displaystyle T_{min} = \sum_{p \in P} x_p(min)$$

Intermediate calculations

To simplify the form of the formulas below, we need some intermediate calculated quantities, derived from the ones above.

Student's rest of test score for a position: $$X_p(s) = T_s - x_p(s)$$.

For any quantity that depends on position (for example $$x_p$$ or $$X_p$$), it's average is denoted with an overbar, and is an average over all students, so

$$\displaystyle \bar{x}_p = \frac{1}{S} \sum_{s \in S} x_p(s)$$.

When a quantity is a property of an item, a bar denotes an average over all students who attempted that item, so

$$\displaystyle \bar{x}_i = \frac{1}{S_i} \sum_{s \in S_i} x_i(s)$$.

Similarly we have the variance of a quantity depending on position:

$$\displaystyle V(x_p) = \frac{1}{S - 1} \sum_{s \in S} (x_p(s) - \bar{x}_p)^2$$

and for a quantity depending on items:

$$\displaystyle V(x_i) = \frac{1}{S_i - 1} \sum_{s \in S_i} (x_i(s) - \bar{x}_i)^2$$.

Finally, we need co-variances of two quantites, for example:

$$\displaystyle C(x_p, X_p) = \frac{1}{S - 1} \sum_{s \in S} (x_p(s) - \bar{x}_p)(X_p(s) - \bar{X}_p)$$

$$\displaystyle C(x_i, X_i) = \frac{1}{S_i - 1} \sum_{s \in S_i} (x_i(s) - \bar{x}_i)(X_i(s) - \bar{X}_i)$$.

Position statistics

Facility index

This is the average score on the item, expressed as a percentage:

$$\displaystyle F_p = 100\frac{\bar{x}_p - x_p(min)}{x_p(max) - x_p(min)}$$.

The higher the facility index, the easier the question is (for this cohort of students).

Standard deviation

Again expressed on a percentage scale:

$$\displaystyle SD_p = 100\frac{\sqrt{V(x_p)}}{x_p(max) - x_p(min)}$$.

Discrimination index

This is the product moment correlation coefficient between $$x_p$$ and $$X_p$$, expressed on a percentage scale. That is,

$$\displaystyle D_p = 100r(x_p, X_p) = 100\frac{C(x_p, X_p)}{\sqrt{V(x_P) V(X_p)}}$$.

The idea is that for a good question (or at least a question that fits in with the other questions in the test), students who have scored highly on the other parts of the test should also have scored highly on this question, so the score for the question and the score for the test as a whole should be well correlated.

The weakness of this statistic is that, unless the facility index is 50%, it is impossible for the discrimination index to be 100%, or, to put it another way, if $$F_p$$ is close to 0% or 100%, $$D_p$$ will always be very small. That makes interpreting this statistic difficult.

Discriminative efficiency

This gets around that weakness in the discrimination index by expressing $$C(x_p, X_p)$$ as a percentage of the maximum value it could have taken given the scores the students got on this question, and the test as a whole. That is:

$$\displaystyle DE_p = 100\frac{C(x_p, X_p)}{C_{max}(x_p, X_p)}$$

where $$C_{max}(x_p, X_p)$$ is defined as follows:

When you compute $$C(x_p, X_p)$$, you do the sum

$$\displaystyle C(x_p, X_p) = \frac{1}{S - 1} \sum_{s \in S} (x_p(s) - \bar{x}_p)(X_p(s) - \bar{X}_p)$$

which involves a term for each student combining their question score and rest of test score. That is, you start with an array of $$x_p(s)$$ with an array of corresponding $$X_p(s)$$, one for each $$s$$. To compute $$C_{max}(x_p, X_p)$$, you just sort these two arrays before applying the above formula. That is, for the purpose of computing $$C_{max}$$, you pretend that the first student scored the lowest $$x_p$$ and the lowest $$X_p$$, the second student scored the second lowest $$x_p$$ and the second lowest $$X_p$$, and so on to the last student, who scored the highest $$x_p$$ and $$X_p$$.

Intended question weight

How much this question was supposed to contribute to determining the overall test score.

$$\displaystyle IQW_p = 100\frac{x_p(max) - x_p(min)}{T_{max} - T_{min}}$$.

Effective question weight

This is an estimate of what proportion of the variance in the students' test scores is due this question.

$$\displaystyle EQW_p = 100\frac{\sqrt{C(x_p, T)}}{\sum_{p \in P}\sqrt{C(x_p, T)}}$$.

Item statistics

Number of attempts

The number of students who got this question as part of a quiz attempt. This is just $$S_i$$.

Random guess score

This is the score that the student would have got by guessing randomly. It depends on the question type. For types like shortanswer, it is 0 - or the score associated with answer '*', if there is one.

For multiple choice questions (including matching, truefalse, etc.) it is the average score over all the possible choices.

(There should probably be a method in the question type class to compute this.)

Facility index

$$\displaystyle F_i = 100\frac{\bar{x}_i - x_i(min)}{x_i(max) - x_i(min)}$$.

Standard deviation

$$\displaystyle SD_i = 100\frac{\sqrt{V(x_i)}}{x_i(max) - x_i(min)}$$.

Discrimination index

$$\displaystyle D_i = 100r(x_i, T) = 100\frac{C(x_i, T)}{\sqrt{V(x_i) V(T)}}$$.

(It is not possible to define $$X_i$$ because it is conceivable that the same item may have been chosen in different positions with different weights, so we substitute $$T$$ instead.)

Discriminative efficiency

$$\displaystyle DE_i = 100\frac{C(x_i, T)}{C_{max}(x_i, T)}$$.

Test statistics

Number of Attempts

This is $$S$$.

Note that depending on the options, we may be counting one or all attempts per student.

Mean Score

Test mean $$\displaystyle = \bar{T} = \frac{1}{S} \sum_{s \in S} T_s$$

Median Score

Sort all the $$T_s$$, and take the middle one if S is odd, or the average of the two middle ones if S is even.

Standard Deviation

Test standard deviation $$\displaystyle = SD = \sqrt{V(t)} = \sqrt{\frac{1}{S - 1} \sum_{s \in S} (T_s - \bar{T})^2}$$.

Skewness and Kurtosis

Probably of limited interest, but included for completeness. Skewness is a measure of the asymmetry in a distribution. Kurtosis - imagine a normal distribution. Kurtosis tells you if your distribution has more of a bulge, but thinner tails, or vice-versa.

First calculate:

$$\displaystyle m_2 = \frac{1}{S} \sum_{s \in S} (T_s - \bar{T})^2$$

$$\displaystyle m_3 = \frac{1}{S} \sum_{s \in S} (T_s - \bar{T})^3$$

$$\displaystyle m_4 = \frac{1}{S} \sum_{s \in S} (T_s - \bar{T})^4$$

Then compute

$$\displaystyle k_2 = \frac{S}{S - 1} m_2 = V(T)$$

$$\displaystyle k_3 = \frac{S^2}{(S - 1)(S - 2)} m_3$$

$$\displaystyle k_4 = \frac{S^3}{(S - 1)(S - 2)(S - 3)} \left((S + 1)m_4 - 3(S - 1)m_2^2\right)$$

Then

Skewness $$\displaystyle = \frac{k_3}{k_2^{2/3}}$$

Kurtosis $$\displaystyle = \frac{k_4}{k_2^2}$$

Coefficient of Internal Consistency

This is on a percentage scale

$$\displaystyle CIC = 100 \frac{P}{P - 1} \left(1 - \frac{1}{V(T)}\sum_{p \in P} V(x_p) \right)$$

Error Ratio

Also a percentage.

$$\displaystyle ER = 100 \sqrt{1 - \frac{CIC}{100}}$$

Standard Error

$$\displaystyle SE = \frac{ER}{100}SD$$

These last three are to do with estimating how reliable the test scores are. If you take the view that the score the student got on the test on the day is a combination of their actual ability and a random error (how lucky they were on the day of the test), the the standard error is an estimate of the luck factor. So if SE ~= 10, and the student scored 60, then you can be quite confident that their actual ability is between 50 and 70.