MoodleDocs - User contributions [en]

Talk:The OU PMatch algorithm

2011-08-08T08:42:41Z

Pgb:

I hope this is the best place to discuss the PMatch question type...
I am currently testing it and have tons of questions and remarks.
* Question 1
How can I match a correct answer '''''including a comma''''', e.g. "The French flag is white, blue and red."?

''match (The French flag blue white and red.)'' gives error message: ''Missing closing bracket in code fragment "match (The French flag is blue".''

--[[User:Joseph Rézeau|Joseph Rézeau]] 20:23, 4 August 2011 (WST)

First let's start by asking the context that requires the comma. You clearly need 'white' 'red' and 'blue'. So
1 are you bothered about the order? Is red blue white OK?
2 are you bothered that the punctuation is correct? Do you want first colour then comma then second colour then and then third colour?
--[[User:Phil Butcher|Phil Butcher]] 19:48, 5 August 2011 (WST)Phil Butcher

And then it dawned on me ... I've not signed this off at the OU yet, so what documentation, if any, have you seen? Everything I've written in on OU websites only - but of course Tim Hunt may have added some of that to the question type. But I don't know what. Maybe it will be best to wait until it is fully finished (the sub/superscript option is what's missing).
--[[User:Phil Butcher|Phil Butcher]] 19:48, 5 August 2011 (WST)Phil Butcher

Talk:The OU PMatch algorithm

2011-08-05T11:48:53Z

Pgb:

Quiz report statistics

2010-09-27T16:23:20Z

Pgb: /* Test statistics */

'''These statistics are designed for use with summative tests where students have just one attempt and complete that attempt.'''

== Test statistics ==

''Average grade'': For discriminating, deferred feedback, tests aim for between 50% and 75%. Values outside
these limits need thinking about. Interactive tests with multiple tries invariably lead to higher averages.

''Median grade'': Half the students score less than this figure.

''Standard deviation'': A measure of the spread of scores about the mean. Aim for values between 12% and
18%. A smaller value suggests that scores are too bunched up.

''Skewness'': A measure of the asymmetry of the distribution of scores. Zero implies a perfectly symmetrical
distribution, positive values a ‘tail’ to the right and negative values a ‘tail’ to the left.

[[Image:Skewness.jpg]]

Aim for a value of -1.0. If it is too negative, it may indicate lack of discrimination between students who do better than average. Similarly, a large positive value (greater than 1.0) may indicate a lack of discrimination near the pass fail border.

''Kurtosis'': Kurtosis is a measure of the flatness of the distribution.

A normal, bell shaped, distribution has a kurtosis of zero. The greater the kurtosis, the more peaked is the
distribution, without much of a tail on either side.

Aim for a value in the range 0-1. A value greater than 1 may indicate that the test is not discriminating
very well between very good or very bad students and those who are average.

''Coefficient of internal consistency (CIC)'': It is impossible to get internal consistency much above 90%.
Anything above 75% is satisfactory. If the value is below 64%, the test as a whole is unsatisfactory and
remedial measures should be considered.

A low value indicates either that some of the questions are not very good at discriminating between
students of different ability and hence that the differences between total scores owe a good deal to chance
or that some of the questions are testing a different quality from the rest and that these two qualities do not
correlate well – i.e. the test as a whole is inhomogeneous.

''Error ratio (ER)'': This is related to CIC according to the following table: it estimates the percentage of the
standard deviation which is due to chance effects rather than to genuine differences of ability between
students. Values of ER in excess of 50% cannot be regarded as satisfactory: they imply that less than half
the standard deviation is due to differences in ability and the rest to chance effects.

{| class="nicetable"
|-
! CIC
! ER
|-
| 100
| 0
|-
| 99
| 10
|-
| 96
| 20
|-
| 91
| 30
|-
| 84
| 40
|-
| 75
| 50
|-
| 64
| 60
|-
| 51
| 700
|}

''Standard error (SE)'': This is SD x ER/100. It estimates how much of the SD is due to chance effects and is a
measure of the uncertainty in any given student’s score. If the same student took an equivalent test, his
or her score could be expected to lie within ±SE of the previous score. The smaller the value of SE the
better the test, but it is difficult to get it below 5% or 6%. A value of 8% corresponds to half a grade
difference on a typical scale – if the SE exceeds this, it is likely that a substantial proportion of the
students will be wrongly graded in the sense that the grades awarded do not accurately indicate their true
abilities.

== Question statistics ==

''Facility index (F)'': The mean score of students on the item.

{| class="nicetable"
|-
! F
! Interpretation
|-
| 5 or less
| Extremely difficult or something wrong with the question.
|-
| 6-10
| Very difficult.
|-
| 11-20
| Difficult.
|-
| 20-34
| Moderately difficult.
|-
| 35-64
| About right for the average student.
|-
| 66-80
| Fairly easy.
|-
| 81-89
| Easy.
|-
| 90-94
| Very easy.
|-
| 95-100
| Extremely easy.
|}

''Standard deviation (SD)'': A measure of the spread of scores about the mean and hence the extent to which
the question might discriminate. If F is very high or very low it is impossible for the spread to be large.
Note however that a good SD does not automatically ensure good discrimination. A value of SD less than
about a third of the question maximum (i.e. 33%) in the table is not generally satisfactory.

''Random guess score (RGS)'': This is the mean score students would be expected to get for a random guess at
the question. Random guess scores are only available for questions that use some form of multiple choice.
All random guess scores are for deferred feedback only and assume the simplest situation e.g. for multiple
response questions students will be told how many answers are correct.

Values above 40% are unsatisfactory – and show that True/False questions must be used sparsely in summative tests.

''Intended weight'': The question weight expressed as a percentage of the overall test score.

''Effective weight'': An estimate of the weight the question actually has in contributing to the overall spread of
scores. The effective weights should add to 100% - but read on.

The intended weight and effective weight are intended to be compared. If the effective weight is greater
than the intended weight it shows the question has a greater share in the spread of scores than may have
been intended. If it is less than the intended weight it shows that it is not having as much effect in
spreading out the scores as was intended.

The calculation of the effective weight relies on taking the square root of the covariance of the question
scores with overall performance. If a question’s scores vary in the opposite way to the overall score, this
would indicate that this is a very odd question which is testing something different from the rest. And the
computer cannot calculate the effective weights of such questions resulting in warning message boxes
being displayed.

''Discrimination index'': This is the correlation between the weighted scores on the question and those on the
rest of the test. It indicates how effective the question is at sorting out able students from those who are
less able. The results should be interpreted as follows

{| class="nicetable"
|-
! Index
! Interpretation
|-
| 50 and above
| Very good discrimination
|-
| 30 – 50
| Adequate discrimination
|-
| 20 - 29
| Weak discrimination
|-
| 0 - 19
| Very weak discrimination
|-
| -ve
| Question probably invalid
|}

''Discrimination efficiency'': This statistic attempts to estimate how good the discrimination index is relative
to the difficulty of the question.

An item which is very easy or very difficult cannot discriminate between students of different ability,
because most of them get the same score on that question. Maximum discrimination requires a facility
index in the range 30% - 70% (although such a value is no guarantee of a high discrimination index).

The discrimination efficiency will very rarely approach 100%, but values in excess of 50% should be
achievable. Lower values indicate that the question is not nearly as effective at discriminating between
students of different ability as it might be and therefore is not a particularly good question.

--[[User:Phil Butcher|Phil Butcher]] 15:53, 27 September 2010 (UTC)

Quiz report statistics

2010-09-27T16:21:13Z

Pgb: /* Test statistics */

'''These statistics are designed for use with summative tests where students have just one attempt and complete that attempt.'''

== Test statistics ==

''Average grade'': For discriminating, deferred feedback, tests aim for between 50% and 75%. Values outside
these limits need thinking about. Interactive tests with multiple tries invariably lead to higher averages.

''Median grade'': Half the students score less than this figure.

''Standard deviation'': A measure of the spread of scores about the mean. Aim for values between 12% and
18%. A smaller value suggests that scores are too bunched up.

''Skewness'': A measure of the asymmetry of the distribution of scores. Zero implies a perfectly symmetrical
distribution, positive values a ‘tail’ to the right and negative values a ‘tail’ to the left.

[[Image:Skewness.jpg]]

Aim for a value of -1.0. If it is too negative, it may indicate lack of discrimination between students who do better than average. Similarly, a large positive value (greater than 1.0) may indicate a lack of discrimination near the pass fail border.

''Kurtosis'': Kurtosis is a measure of the flatness of the distribution.

A normal, bell shaped, distribution has a kurtosis of zero. The greater the kurtosis, the more peaked is the
distribution, without much of a tail on either side.

Aim for a value in the range 0-1. A value greater than 1 may indicate that the test is not discriminating
very well between very good or very bad students and those who are average.

''Coefficient of internal consistency (CIC)'': It is impossible to get internal consistency much above 90%.
Anything above 75% is satisfactory. If the value is below 64%, the test as a whole is unsatisfactory and
remedial measures should be considered.

A low value indicates either that some of the questions are not very good at discriminating between
students of different ability and hence that the differences between total scores owe a good deal to chance
or that some of the questions are testing a different quality from the rest and that these two qualities do not
correlate well – i.e. the test as a whole is inhomogeneous.

''Error ratio (ER)'': This is related to CIC according to the following table: it estimates the percentage of the
standard deviation which is due to chance effects rather than to genuine differences of ability between
students. Values of ER in excess of 50% cannot be regarded as satisfactory: they imply that less than half
the standard deviation is due to differences in ability and the rest to chance effects.

{| class="nicetable"
|-
! CIC
! ER
|-
| 100
| 0
|-
| 99
| 10
|-
| 96
| 20
|-
| 91
| 30
|-
| 84
| 40
|-
| 75
| 50
|-
| 64
| 60
|-
| 51
| 700
|}

''Standard error (SE)'': This is SD x ER/100. It estimates how much of the SD is due to chance effects and is a
measure of the uncertainty in any given student’s score. If the same student took an equivalent test, his
or her score could be expected to lie within ±SE of the previous score. The smaller the value of SE the
better the test, but it is difficult to get it below 5% or 6%. A value of 8% corresponds to half a grade
difference on the University Scale – if the SE exceeds this, it is likely that a substantial proportion of the
students will be wrongly graded in the sense that the grades awarded do not accurately indicate their true
abilities.

== Question statistics ==

''Facility index (F)'': The mean score of students on the item.

{| class="nicetable"
|-
! F
! Interpretation
|-
| 5 or less
| Extremely difficult or something wrong with the question.
|-
| 6-10
| Very difficult.
|-
| 11-20
| Difficult.
|-
| 20-34
| Moderately difficult.
|-
| 35-64
| About right for the average student.
|-
| 66-80
| Fairly easy.
|-
| 81-89
| Easy.
|-
| 90-94
| Very easy.
|-
| 95-100
| Extremely easy.
|}

''Standard deviation (SD)'': A measure of the spread of scores about the mean and hence the extent to which
the question might discriminate. If F is very high or very low it is impossible for the spread to be large.
Note however that a good SD does not automatically ensure good discrimination. A value of SD less than
about a third of the question maximum (i.e. 33%) in the table is not generally satisfactory.

''Random guess score (RGS)'': This is the mean score students would be expected to get for a random guess at
the question. Random guess scores are only available for questions that use some form of multiple choice.
All random guess scores are for deferred feedback only and assume the simplest situation e.g. for multiple
response questions students will be told how many answers are correct.

Values above 40% are unsatisfactory – and show that True/False questions must be used sparsely in summative tests.

''Intended weight'': The question weight expressed as a percentage of the overall test score.

''Effective weight'': An estimate of the weight the question actually has in contributing to the overall spread of
scores. The effective weights should add to 100% - but read on.

The intended weight and effective weight are intended to be compared. If the effective weight is greater
than the intended weight it shows the question has a greater share in the spread of scores than may have
been intended. If it is less than the intended weight it shows that it is not having as much effect in
spreading out the scores as was intended.

The calculation of the effective weight relies on taking the square root of the covariance of the question
scores with overall performance. If a question’s scores vary in the opposite way to the overall score, this
would indicate that this is a very odd question which is testing something different from the rest. And the
computer cannot calculate the effective weights of such questions resulting in warning message boxes
being displayed.

''Discrimination index'': This is the correlation between the weighted scores on the question and those on the
rest of the test. It indicates how effective the question is at sorting out able students from those who are
less able. The results should be interpreted as follows

{| class="nicetable"
|-
! Index
! Interpretation
|-
| 50 and above
| Very good discrimination
|-
| 30 – 50
| Adequate discrimination
|-
| 20 - 29
| Weak discrimination
|-
| 0 - 19
| Very weak discrimination
|-
| -ve
| Question probably invalid
|}

''Discrimination efficiency'': This statistic attempts to estimate how good the discrimination index is relative
to the difficulty of the question.

An item which is very easy or very difficult cannot discriminate between students of different ability,
because most of them get the same score on that question. Maximum discrimination requires a facility
index in the range 30% - 70% (although such a value is no guarantee of a high discrimination index).

The discrimination efficiency will very rarely approach 100%, but values in excess of 50% should be
achievable. Lower values indicate that the question is not nearly as effective at discriminating between
students of different ability as it might be and therefore is not a particularly good question.

--[[User:Phil Butcher|Phil Butcher]] 15:53, 27 September 2010 (UTC)

Quiz report statistics

2010-09-27T16:18:46Z

Pgb: /* Test statistics */

'''These statistics are designed for use with summative tests where students have just one attempt and complete that attempt.'''

== Test statistics ==

''Average grade'': For discriminating, deferred feedback, tests aim for between 50% and 75%. Values outside
these limits need thinking about. Interactive tests with multiple tries invariably lead to higher averages.

''Median grade'': Half the students score less than this figure.

''Standard deviation'': A measure of the spread of scores about the mean. Aim for values between 12% and
18%. A smaller value suggests that scores are too bunched up.

''Skewness'': A measure of the asymmetry of the distribution of scores. Zero implies a perfectly symmetrical
distribution, positive values a ‘tail’ to the right and negative values a ‘tail’ to the left.

[[Image:skewness.jpg]]

Aim for a value of -1.0. If it is too negative, it may indicate lack of discrimination between students who do better than average. Similarly, a large positive value (greater than 1.0) may indicate a lack of discrimination near the pass fail border.

''Kurtosis'': Kurtosis is a measure of the flatness of the distribution.

A normal, bell shaped, distribution has a kurtosis of zero. The greater the kurtosis, the more peaked is the
distribution, without much of a tail on either side.

Aim for a value in the range 0-1. A value greater than 1 may indicate that the test is not discriminating
very well between very good or very bad students and those who are average.

''Coefficient of internal consistency (CIC)'': It is impossible to get internal consistency much above 90%.
Anything above 75% is satisfactory. If the value is below 64%, the test as a whole is unsatisfactory and
remedial measures should be considered.

A low value indicates either that some of the questions are not very good at discriminating between
students of different ability and hence that the differences between total scores owe a good deal to chance
or that some of the questions are testing a different quality from the rest and that these two qualities do not
correlate well – i.e. the test as a whole is inhomogeneous.

''Error ratio (ER)'': This is related to CIC according to the following table: it estimates the percentage of the
standard deviation which is due to chance effects rather than to genuine differences of ability between
students. Values of ER in excess of 50% cannot be regarded as satisfactory: they imply that less than half
the standard deviation is due to differences in ability and the rest to chance effects.

{| class="nicetable"
|-
! CIC
! ER
|-
| 100
| 0
|-
| 99
| 10
|-
| 96
| 20
|-
| 91
| 30
|-
| 84
| 40
|-
| 75
| 50
|-
| 64
| 60
|-
| 51
| 700
|}

''Standard error (SE)'': This is SD x ER/100. It estimates how much of the SD is due to chance effects and is a
measure of the uncertainty in any given student’s score. If the same student took an equivalent test, his
or her score could be expected to lie within ±SE of the previous score. The smaller the value of SE the
better the test, but it is difficult to get it below 5% or 6%. A value of 8% corresponds to half a grade
difference on the University Scale – if the SE exceeds this, it is likely that a substantial proportion of the
students will be wrongly graded in the sense that the grades awarded do not accurately indicate their true
abilities.

== Question statistics ==

''Facility index (F)'': The mean score of students on the item.

{| class="nicetable"
|-
! F
! Interpretation
|-
| 5 or less
| Extremely difficult or something wrong with the question.
|-
| 6-10
| Very difficult.
|-
| 11-20
| Difficult.
|-
| 20-34
| Moderately difficult.
|-
| 35-64
| About right for the average student.
|-
| 66-80
| Fairly easy.
|-
| 81-89
| Easy.
|-
| 90-94
| Very easy.
|-
| 95-100
| Extremely easy.
|}

''Standard deviation (SD)'': A measure of the spread of scores about the mean and hence the extent to which
the question might discriminate. If F is very high or very low it is impossible for the spread to be large.
Note however that a good SD does not automatically ensure good discrimination. A value of SD less than
about a third of the question maximum (i.e. 33%) in the table is not generally satisfactory.

''Random guess score (RGS)'': This is the mean score students would be expected to get for a random guess at
the question. Random guess scores are only available for questions that use some form of multiple choice.
All random guess scores are for deferred feedback only and assume the simplest situation e.g. for multiple
response questions students will be told how many answers are correct.

Values above 40% are unsatisfactory – and show that True/False questions must be used sparsely in summative tests.

''Intended weight'': The question weight expressed as a percentage of the overall test score.

''Effective weight'': An estimate of the weight the question actually has in contributing to the overall spread of
scores. The effective weights should add to 100% - but read on.

The intended weight and effective weight are intended to be compared. If the effective weight is greater
than the intended weight it shows the question has a greater share in the spread of scores than may have
been intended. If it is less than the intended weight it shows that it is not having as much effect in
spreading out the scores as was intended.

The calculation of the effective weight relies on taking the square root of the covariance of the question
scores with overall performance. If a question’s scores vary in the opposite way to the overall score, this
would indicate that this is a very odd question which is testing something different from the rest. And the
computer cannot calculate the effective weights of such questions resulting in warning message boxes
being displayed.

''Discrimination index'': This is the correlation between the weighted scores on the question and those on the
rest of the test. It indicates how effective the question is at sorting out able students from those who are
less able. The results should be interpreted as follows

{| class="nicetable"
|-
! Index
! Interpretation
|-
| 50 and above
| Very good discrimination
|-
| 30 – 50
| Adequate discrimination
|-
| 20 - 29
| Weak discrimination
|-
| 0 - 19
| Very weak discrimination
|-
| -ve
| Question probably invalid
|}

''Discrimination efficiency'': This statistic attempts to estimate how good the discrimination index is relative
to the difficulty of the question.

An item which is very easy or very difficult cannot discriminate between students of different ability,
because most of them get the same score on that question. Maximum discrimination requires a facility
index in the range 30% - 70% (although such a value is no guarantee of a high discrimination index).

The discrimination efficiency will very rarely approach 100%, but values in excess of 50% should be
achievable. Lower values indicate that the question is not nearly as effective at discriminating between
students of different ability as it might be and therefore is not a particularly good question.

--[[User:Phil Butcher|Phil Butcher]] 15:53, 27 September 2010 (UTC)

Quiz report statistics

2010-09-27T16:10:02Z

Pgb:

'''These statistics are designed for use with summative tests where students have just one attempt and complete that attempt.'''

== Test statistics ==

''Average grade'': For discriminating, deferred feedback, tests aim for between 50% and 75%. Values outside
these limits need thinking about. Interactive tests with multiple tries invariably lead to higher averages.

''Median grade'': Half the students score less than this figure.

''Standard deviation'': A measure of the spread of scores about the mean. Aim for values between 12% and
18%. A smaller value suggests that scores are too bunched up.

''Skewness'': A measure of the asymmetry of the distribution of scores. Zero implies a perfectly symmetrical
distribution, positive values a ‘tail’ to the right and negative values a ‘tail’ to the left.

Aim for a value of -1.0. If it is too negative, it may indicate lack of discrimination between students who do better than average. Similarly, a large positive value (greater than 1.0) may indicate a lack of discrimination near the pass fail border.

''Kurtosis'': Kurtosis is a measure of the flatness of the distribution.

A normal, bell shaped, distribution has a kurtosis of zero. The greater the kurtosis, the more peaked is the
distribution, without much of a tail on either side.

Aim for a value in the range 0-1. A value greater than 1 may indicate that the test is not discriminating
very well between very good or very bad students and those who are average.

''Coefficient of internal consistency (CIC)'': It is impossible to get internal consistency much above 90%.
Anything above 75% is satisfactory. If the value is below 64%, the test as a whole is unsatisfactory and
remedial measures should be considered.

A low value indicates either that some of the questions are not very good at discriminating between
students of different ability and hence that the differences between total scores owe a good deal to chance
or that some of the questions are testing a different quality from the rest and that these two qualities do not
correlate well – i.e. the test as a whole is inhomogeneous.

''Error ratio (ER)'': This is related to CIC according to the following table: it estimates the percentage of the
standard deviation which is due to chance effects rather than to genuine differences of ability between
students. Values of ER in excess of 50% cannot be regarded as satisfactory: they imply that less than half
the standard deviation is due to differences in ability and the rest to chance effects.

{| class="nicetable"
|-
! CIC
! ER
|-
| 100
| 0
|-
| 99
| 10
|-
| 96
| 20
|-
| 91
| 30
|-
| 84
| 40
|-
| 75
| 50
|-
| 64
| 60
|-
| 51
| 700
|}

''Standard error (SE)'': This is SD x ER/100. It estimates how much of the SD is due to chance effects and is a
measure of the uncertainty in any given student’s score. If the same student took an equivalent test, his
or her score could be expected to lie within ±SE of the previous score. The smaller the value of SE the
better the test, but it is difficult to get it below 5% or 6%. A value of 8% corresponds to half a grade
difference on the University Scale – if the SE exceeds this, it is likely that a substantial proportion of the
students will be wrongly graded in the sense that the grades awarded do not accurately indicate their true
abilities.

== Question statistics ==

''Facility index (F)'': The mean score of students on the item.

{| class="nicetable"
|-
! F
! Interpretation
|-
| 5 or less
| Extremely difficult or something wrong with the question.
|-
| 6-10
| Very difficult.
|-
| 11-20
| Difficult.
|-
| 20-34
| Moderately difficult.
|-
| 35-64
| About right for the average student.
|-
| 66-80
| Fairly easy.
|-
| 81-89
| Easy.
|-
| 90-94
| Very easy.
|-
| 95-100
| Extremely easy.
|}

''Standard deviation (SD)'': A measure of the spread of scores about the mean and hence the extent to which
the question might discriminate. If F is very high or very low it is impossible for the spread to be large.
Note however that a good SD does not automatically ensure good discrimination. A value of SD less than
about a third of the question maximum (i.e. 33%) in the table is not generally satisfactory.

''Random guess score (RGS)'': This is the mean score students would be expected to get for a random guess at
the question. Random guess scores are only available for questions that use some form of multiple choice.
All random guess scores are for deferred feedback only and assume the simplest situation e.g. for multiple
response questions students will be told how many answers are correct.

Values above 40% are unsatisfactory – and show that True/False questions must be used sparsely in summative tests.

''Intended weight'': The question weight expressed as a percentage of the overall test score.

''Effective weight'': An estimate of the weight the question actually has in contributing to the overall spread of
scores. The effective weights should add to 100% - but read on.

The intended weight and effective weight are intended to be compared. If the effective weight is greater
than the intended weight it shows the question has a greater share in the spread of scores than may have
been intended. If it is less than the intended weight it shows that it is not having as much effect in
spreading out the scores as was intended.

The calculation of the effective weight relies on taking the square root of the covariance of the question
scores with overall performance. If a question’s scores vary in the opposite way to the overall score, this
would indicate that this is a very odd question which is testing something different from the rest. And the
computer cannot calculate the effective weights of such questions resulting in warning message boxes
being displayed.

''Discrimination index'': This is the correlation between the weighted scores on the question and those on the
rest of the test. It indicates how effective the question is at sorting out able students from those who are
less able. The results should be interpreted as follows

{| class="nicetable"
|-
! Index
! Interpretation
|-
| 50 and above
| Very good discrimination
|-
| 30 – 50
| Adequate discrimination
|-
| 20 - 29
| Weak discrimination
|-
| 0 - 19
| Very weak discrimination
|-
| -ve
| Question probably invalid
|}

''Discrimination efficiency'': This statistic attempts to estimate how good the discrimination index is relative
to the difficulty of the question.

An item which is very easy or very difficult cannot discriminate between students of different ability,
because most of them get the same score on that question. Maximum discrimination requires a facility
index in the range 30% - 70% (although such a value is no guarantee of a high discrimination index).

The discrimination efficiency will very rarely approach 100%, but values in excess of 50% should be
achievable. Lower values indicate that the question is not nearly as effective at discriminating between
students of different ability as it might be and therefore is not a particularly good question.

--[[User:Phil Butcher|Phil Butcher]] 15:53, 27 September 2010 (UTC)

Quiz report statistics

2010-09-27T15:53:14Z

Pgb: New page: '''These statistics are designed for use with summative tests where students have just one attempt and complete that attempt.''' == Test statistics == ''Average grade'': For discriminat...

'''These statistics are designed for use with summative tests where students have just one attempt and complete that attempt.'''

== Test statistics ==

''Average grade'': For discriminating, deferred feedback, tests aim for between 50% and 75%. Values outside
these limits need thinking about. Interactive tests with multiple tries invariably lead to higher averages.

''Median grade'': Half the students score less than this figure.

''Standard deviation'': A measure of the spread of scores about the mean. Aim for values between 12% and
18%. A smaller value suggests that scores are too bunched up.

''Skewness'': A measure of the asymmetry of the distribution of scores. Zero implies a perfectly symmetrical
distribution, positive values a ‘tail’ to the right and negative values a ‘tail’ to the left.

Aim for a value of -1.0. If it is too negative, it may indicate lack of discrimination between students who do better than average. Similarly, a large positive value (greater than 1.0) may indicate a lack of discrimination near the pass fail border.

''Kurtosis'': Kurtosis is a measure of the flatness of the distribution.

A normal, bell shaped, distribution has a kurtosis of zero. The greater the kurtosis, the more peaked is the
distribution, without much of a tail on either side.

Aim for a value in the range 0-1. A value greater than 1 may indicate that the test is not discriminating
very well between very good or very bad students and those who are average.

''Coefficient of internal consistency (CIC)'': It is impossible to get internal consistency much above 90%.
Anything above 75% is satisfactory. If the value is below 64%, the test as a whole is unsatisfactory and
remedial measures should be considered.

A low value indicates either that some of the questions are not very good at discriminating between
students of different ability and hence that the differences between total scores owe a good deal to chance
or that some of the questions are testing a different quality from the rest and that these two qualities do not
correlate well – i.e. the test as a whole is inhomogeneous.

''Error ratio (ER)'': This is related to CIC according to the following table: it estimates the percentage of the
standard deviation which is due to chance effects rather than to genuine differences of ability between
students. Values of ER in excess of 50% cannot be regarded as satisfactory: they imply that less than half
the standard deviation is due to differences in ability and the rest to chance effects.

CIC 100 99 96 91 84 75 64 51

ER 0 10 20 30 40 50 60 70

''Standard error (SE)'': This is SD x ER/100. It estimates how much of the SD is due to chance effects and is a
measure of the uncertainty in any given student’s score. If the same student took an equivalent test, his
or her score could be expected to lie within ±SE of the previous score. The smaller the value of SE the
better the test, but it is difficult to get it below 5% or 6%. A value of 8% corresponds to half a grade
difference on the University Scale – if the SE exceeds this, it is likely that a substantial proportion of the
students will be wrongly graded in the sense that the grades awarded do not accurately indicate their true
abilities.

== Question statistics ==

''Facility index (F)'': The mean score of students on the item.

F Interpretation

5 or less Extremely difficult or something wrong with the question.

6-10 Very difficult.

11-20 Difficult.

20-34 Moderately difficult.

35-64 About right for the average student.

66-80 Fairly easy.

81-89 Easy.

90-94 Very easy.

95-100 Extremely easy.

''Standard deviation (SD)'': A measure of the spread of scores about the mean and hence the extent to which
the question might discriminate. If F is very high or very low it is impossible for the spread to be large.
Note however that a good SD does not automatically ensure good discrimination. A value of SD less than
about a third of the question maximum (i.e. 33%) in the table is not generally satisfactory.

''Random guess score (RGS)'': This is the mean score students would be expected to get for a random guess at
the question. Random guess scores are only available for questions that use some form of multiple choice.
All random guess scores are for deferred feedback only and assume the simplest situation e.g. for multiple
response questions students will be told how many answers are correct.

Values above 40% are unsatisfactory – and show that True/False questions must be used sparsely in summative tests.

''Intended weight'': The question weight expressed as a percentage of the overall test score.

''Effective weight'': An estimate of the weight the question actually has in contributing to the overall spread of
scores. The effective weights should add to 100% - but read on.

The intended weight and effective weight are intended to be compared. If the effective weight is greater
than the intended weight it shows the question has a greater share in the spread of scores than may have
been intended. If it is less than the intended weight it shows that it is not having as much effect in
spreading out the scores as was intended.

The calculation of the effective weight relies on taking the square root of the covariance of the question
scores with overall performance. If a question’s scores vary in the opposite way to the overall score, this
would indicate that this is a very odd question which is testing something different from the rest. And the
computer cannot calculate the effective weights of such questions resulting in warning message boxes
being displayed.

''Discrimination index'': This is the correlation between the weighted scores on the question and those on the
rest of the test. It indicates how effective the question is at sorting out able students from those who are
less able. The results should be interpreted as follows

50 and above Very good discrimination

30 – 50 Adequate discrimination

20 - 29 Weak discrimination

0 - 19 Very weak discrimination

-ve Question probably invalid

''Discrimination efficiency'': This statistic attempts to estimate how good the discrimination index is relative
to the difficulty of the question.

An item which is very easy or very difficult cannot discriminate between students of different ability,
because most of them get the same score on that question. Maximum discrimination requires a facility
index in the range 30% - 70% (although such a value is no guarantee of a high discrimination index).

The discrimination efficiency will very rarely approach 100%, but values in excess of 50% should be
achievable. Lower values indicate that the question is not nearly as effective at discriminating between
students of different ability as it might be and therefore is not a particularly good question.

--[[User:Phil Butcher|Phil Butcher]] 15:53, 27 September 2010 (UTC)

Moodle 2.0 release notes

2010-09-27T15:09:28Z

Pgb: /* Quiz module and question bank */

{{Work_in_progress}}

Expected final release date: '''Early November, 2010''' (if testing goes well)

Interim releases for testing:

* [[Moodle 2.0 Preview 1 release notes]] - May 4, 2010
* [[Moodle 2.0 Preview 2 release notes]] - May 17, 2010
* [[Moodle 2.0 Preview 3 release notes]] - May 31, 2010
* [[Moodle 2.0 Preview 4 release notes]] - June 30, 2010
* [[Moodle 2.0 Release Candidate 1 release notes]] - September 21, 2010

Moodle 2.0 contains a lot of large new features, some completely rewritten features, and hundreds of bug fixes. For full details (more than you probably want!), see [http://tracker.moodle.org/browse/MDL/fixforversion/10122 the full list of fixed issues in 2.0].

This page is a summary of the major things to look for (links and screenshots will be added over time).

==Major new features==

===[[Community hub|Community hubs]]===

* Anybody can set up a Community hub, which is a directory of courses for public use or for private communities. The code is implemented as separate GPL plugin for Moodle.
* Sites can register to any Community hub (instead of just moodle.org)
* Teachers on registered sites can publish their full courses to Community hubs, for download
* Teachers on registered sites can also advertise their courses on Community hubs, for people to join
* Teachers on any site can search all public Community hubs and download courses as templates for their own courses
* Users on any Moodle site can also search Community hubs for courses (and communities of practice) to participate in. Initially we are encouraging ''''communities of teaching practice'''' but any sort of course can be listed.

===[[Repositories|Repository support]]===

* Moodle now supports integration with external repositories of content, making it really simple to bring documents and media into Moodle via an AJAX interface that looks like a standard '''Open''' dialogue in desktop applications.
* Initial plugins in 2.0 include: Alfresco, Amazon S3, Box.net, File system on Server, Flickr, Google Docs, Mahara, MERLOT, Picasa, Recent Files, Remote Moodle sites, WebDAV servers, Wikimedia, Youtube. These are simple to develop, so many more are expected.
* You can also import files from your desktop or by specifying a URL.

===[[Portfolios|Portfolio support]]===

* Modules can now export their data to external systems, particularly useful for portfolios where snapshots of forums, assignments and other things in Moodle are useful to record in a journal or a portfolio of evidence
* Different formats are supported (currently LEAP2A, HTML, Images and Text, but others like PDF can be added)
* Initial plugins in 2.0 include: Box.net, Flickr, Google Docs, '''Mahara''' and Picasa.

===[[Completion]]===

* Teachers can now specify conditions that define when any '''activity''' is seen as completed by a student. For example, when a certain number of posts have been made, or a grade has been reached, or a choice has been made.
* Teachers can now specify conditions that define with any '''course''' is seen as completed by a student. Conditions include activity completion, but could also be by grade, date or a number of other criteria.
* Teachers and students can see reports that show the progress of any user within a course, or through a series of courses.

===[[Conditional activities]]===

* Access to activities can be restricted based on certain criteria, such as dates, grade obtained, or the completion of another activity.
* These can be chained together to enable progressive disclosure of the course content, if that is desired.

===[[Cohorts]]===
* Also known as "Site-wide groups", these are site-wide collections of users that can be enrolled into courses in one action, either manually or synchronised automatically

===[[Web Services|Web services support]]===
* Support for standards-based web services across the entire Moodle code base, allowing the admin to expose particular functions of Moodle for use by:
** Administrative systems such as HR or SIS applications
** Mobile clients
* Framework contains a very high-level of security with a detailed token system and complete control over the range of functions exposed
* All defined functions are automatically available via:
** XML-RPC
** AMF (Flash)
** REST
** SOAP (PHP)

===[[IMS Common Cartridge import|IMS Common Cartridge]]===
* Moodle can now import courses in IMS Common Cartridge format (commonly used by publishers)

===New blocks===
* [[Comments block]] - like a shoutbox, allows comments to be added to any page. Great for student feedback.
* [[My private files block]] - allows easy access to one's private file repository in Moodle (with quota support)
* [[Community block]] - keeps track of external courses one is interested in
* [[Course completion status block]] - reports on the completion status of your courses

===[[Plagiarism Prevention|Plagiarism prevention]]===

* Moodle supports integration with plagiarism prevention tools such as Turnitin

==Major improvements to existing core features==

===[[Backup 2.0|Backup and restore]]===

* Completely rewritten Backup/Restore framework, no longer bound by memory (can work with '''any size course''').
* Completely new backup format.
* Improved interface.
* Backup can be made of whole courses, but also specific sections or activities.

===[[Blocks 2.0|Blocks]]===
* Blocks are now consistently implemented on every page in Moodle
* No longer any limit to the block regions (in addition to left and right, put them at the top, center or bottom of pages)
* Any block can be made sticky (appears in all the contexts below, eg throughout a course).
* Blocks can be "docked" on the side of the screen (if the theme supports it)

===[[Blogs 2.0|Blogs]]===
* Support for comments on each blog entry
* Removal of group-level and course-level blogs (these are converted into forums on upgrade)
* Support for external blog feeds (synchronised to Moodle blog)

===[[Comments 2.0|Comments]]===
* User comments (Glossaries, Databases, Blogs, etc) are now all consistently handled and displayed throughout Moodle, using AJAX if available
* User activity reports will include all the comments made by that user

===[[Enrolments 2.0|Enrolment plugins]]===
* Major improvements in the handling of guests and guest accounts
* Support for multiple forms of enrolment at the same time
* More detailed control over enrolment in courses

===[[File handling 2.0|File handling]]===

* Full support for Unicode file names on all operating systems.
* Metadata about each file (author, date, license, etc) and what the file is used for are stored in the database.
* Duplicate files (for example, a large video file use in two different courses) are only stored once, saving disk space.
* Files are no longer just "uploaded to the course". Files are connected to the particular bit of Moodle content that uses them. (For example, a file may belong to a file resource, a forum post or a wiki page). Access to these files is then controlled by the same rules as as that bit of Moodle, increasing security.

===[[HTML editor 2.0|HTML editor]]===
* New editor based on TinyMCE
* Works on more browsers
* Resizable editing area
* Cleaner XHTML output
* Full integration with configured external repositories to import and embed media into text

===[[Messaging 2.0|Messaging]]===
* All email sent by Moodle is now treated as a message
* A message overview panel allows users to control how messages are sent to them
* Initial message output plugins in Moodle 2.0 include: Email, Jabber and Popups

===[[My Moodle 2.0|My Moodle page]]===
* More customisable My Moodle page with new blocks for showing relevant information
* Admin can design (and optionally force) site-wide layouts for My Moodle
* My Moodle page given more prominence as the main "home page" for users

===[[Navigation 2.0|Navigation]]===
* Standard "Navigation" block on every page showing contextual links, while allowing you to jump elsewhere quickly
* Standard "Settings" blocks on every page shows contextual settings as well as settings for anything else you have permissions for

===[[Ratings 2.0|Ratings]]===
* User ratings (Glossaries, Databases, Forums, etc) are now all consistently handled and displayed throughout Moodle, using AJAX if available
* Aggregation of using ratings into activity grades is now standardised in all activities

===[[Roles 2.0|Roles and permissions]]===
* Improved and simplified AJAX interfaces for defining and assigning roles
* Improved and simplified interfaces for tweaking permissions in any given context
* New "Archetypes" concept replacing the "Legacy roles" concept.
* New archetype "manager" to define the role of most people with system-wide editing rights, separate from "admin" role.

===[[RSS feeds 2.0|RSS feeds]]===
* All RSS feeds are now secured using a random per-user token in the URL
* Tokens can be updated by the user at any time (if they suspect a feed URL has been compromised)
* RSS feeds are now more accurate (eg they support forums with separate groups), and are generated efficiently whenever required

===[[Development:Themes 2.0|Themes]]===
* Many new themes in the core distribution
* All HTML and JS ouput is now far more efficient (server-side caching) and consistent (tableless layout, new CSS, YUI Framework)
* Themes can change the HTML of the page if they wish
* Core support for custom menus in all themes (for example at the top of the page)

===[[Translation 2.0|Translation system]]===
* [http://lang.moodle.org/ New web portal] to make it easer for groups to collaborate on translating Moodle, and to keep their translations up-to-date.
* More efficient [[Development:Languages/AMOS|storage format for language strings]] should slightly improve performance.

===User profile pages===
* Site-wide user profile page can be customised by users with blocks, news, feeds and so on
* Course-specific user profile pages show course blocks and standard profile information, plus information for teachers of that course

==Major improvements to activity modules==

===Lesson===
* Refactored internal code
* Forms are now standard Moodle forms

===Quiz module and question bank===

* [[Development:quiz_navigation|Quiz navigation improvements for students]]
* [[Development:Flagging_questions_during_a_quiz_attempt|Flagging questions during a quiz attempt]]
* [[Development:Quiz_report_enhancements|Quiz report enhancements]] - Major improvements to the quiz reports, especially regrading and item analysis
* [[Development:Quiz_report_statistics|Quiz report statistics]] - A brief guide
* [[Development:Quiz_UI_redesign|Quiz editing interface improvements]]
* Different settings (open/close date, number of attempts, password, time limit) for each group or student (MDL-16478)
* [[Development:Administration page for question types|Administration page for question types]]
* [[Development:Moodle 2.0 question bank improvements|Question tagging and improved searching in the question bank]]
* MDL-8648 Essay questions can now be randomised by random questions

===Resource===
* All the resource types have been refactored into real modules, and cleaned up
** File - for displaying a file, possibly with supporting files (like a HTML mini-site)
** Folder - for displaying a collection of documents
** URL - for displaying a page with a given URL
** Page - for a single page, edited online using the HTML editor
** IMS - for showing a regular IMS content package
* Better XHTML-compliant support for frames, iframes and embedding in all these modules

===SCORM===

* New [[SCORM module]] settings - display attempt status, display course structure, force completed, force new attempt, lock after final attempt - allowing the behaviour dictated to the SCORM object by the authoring package to be changed MDL-11501
* New reporting interface including sortable/collapsible table with group select box and ability to download in Excel, ODS and text format MDL-21555
* New SCORM player UI with better navigation, improved performance and better handling of stage size MDL-22951

===[[Wiki module 2.0|Wiki]]===
* Completely re-written from scratch, based on NWIki from UPC
* Support for Mediawiki-style syntax, as well as Creole
* Interface improvements

===[[Workshop module 2.0|Workshop]]===

* Completely rewritten from scratch
* Vastly improved interface for managing stages and users

==System requirements==

Since Moodle 2.0 is such a major release, we are allowing ourselves some increases in the requirements.

* PHP must be 5.2.8 or later (it was released 08-Dec-2008)
* Databases should be one of the following:
** MySQL 5.0.25 or later (InnoDB storage engine highly recommended)
** PostgreSQL 8.3 or later
** Oracle 10.2 or later
** MS SQL 2005 or later
* Any standards-supporting browser from the past few years, for example:
** Firefox 3 or later
** Safari 3 or later
** Google Chrome 4 or later
** Opera 9 or later
** MS Internet Explorer 7 or later (Even [http://googleenterprise.blogspot.com/2010/01/modern-browsers-for-modern-applications.html Google don't support IE6 any more])
** etc

==Upgrading==

When upgrading to Moodle 2.0, you must have Moodle 1.9 or later. if you are using an earlier version of Moodle (eg 1.8.x) then you need to upgrade to Moodle 1.9.x first. We advise that you test the upgrade first on a COPY of your production site, to make sure it works as you expect.

For further information, see [[Upgrading to Moodle 2.0]].

==For developers: API changes==

See [[Development:Migrating_contrib_code_to_2.0]]

* [[Development:Plugin system changes in Moodle 2.0]] - all the different types of plugin are now handles more consistently when it comes to installation and upgrading, capabilities, events, and so on.
* [[Development:DB_layer_2.0_migration_docs|Database layer changes]] - you will need to update your code.
* [[Development:Using_the_file_API|File handling changes]] - you will need to update your code.
* [[Development:Migrating your code code to the 2.0 rendering API|Rendering layer changes]] - should be mostly backwards compatible, but you are advised to upgrade your code.
* Require capability used to do an automatic require_login. It no longer does so. All pages must explicitly call require_login if they need it. MDL-19882
* [[Development:Moodle_2.0_question_type_API_changes|Changes to the question type API]]
* MNet has been refactored and tidied up - related third party code needs to be checked
* Changes and improvements to the [[Development:Local_customisation|Local customisation system]].
* Javascript
* YUI
* custom profile fields values are loaded into $USER->profile array instead of directly into $USER object

==See also==

*[[Moodle 1.9 release notes]]
*[[QA testing]]

[[Category:Release notes]]
[[Category:Moodle 2.0]]

[[fr:Notes de mise à jour de Moodle 2.0]]
[[es:Notas de Moodle 2.0]]