Difference between revisions of "Quiz reports"

Jump to: navigation, search

Note: You are currently viewing documentation for Moodle 3.3. Up-to-date documentation for the latest stable version of Moodle is probably available here: Quiz reports.

(Tidy up a bit)
Line 2: Line 2:
 
<p class="note">'''Please refer to [[TOC_with_notes#Quiz|these notes]] before editing this page.'''</p>
 
<p class="note">'''Please refer to [[TOC_with_notes#Quiz|these notes]] before editing this page.'''</p>
  
Depending upon how the site is setup, you will find all of the reports on '''the course navigation menu'''.   By clicking on the quiz name and then "Attempts" you will go to the default Results report called Grades.
+
You will find the quiz reports in '''the navigation menu''' under ''Quiz name'' -> Results. You can also get to the quiz reports where you see a summary of the number of attempts that have been made like "Attempts: 123". That link also goes to the reports.
  
:The standard reports can be difficult to find for a 1.9 user of Moodle in a teacher's role. They are NOT in the Quiz administration menu. However once the navigation menu is displayed on the left, you should always be able to see the links as displayed below.  
+
[[File:Quiz results menu in a course navigation.png|center|2.1 navigation menu showing a Quiz's Result menu links]]
 +
 
 +
==Grades report==
 +
 
 +
This shows all the students' quiz attempts, with the overall grade, and the grade for each question. There are links to review all the details of a student's attempt, just as the student would see it.
 +
 
 +
The results can be downloaded in a variety of formats.
 +
 
 +
There is a setting form at the top of the page that can be used to control what is displayed.
 +
 
 +
At the bottom of the page is a graph showing the distribution of scores.
 +
 
 +
===Deleting attempts===
 +
 
 +
This report can be used to delete selected quiz attempts, which may sometimes be necessary.
 +
 
 +
Note that you cannot edit the quiz after it has been attempted, so that is one reason you may wish to delete all attempts.
  
[[File:Quiz results menu in a course navigation.png|center|2.1 navigation menu showing a Quiz's Result menu links]]
+
===Re-grading attempts===
  
==Quiz Results > Grades page==
+
The user-interface for re-grading all, or selected, attempts are also on this page.
When the teacher clicks on the link to the quiz, they will see an "Attempts" link just above the "Preview quiz now" button.  This will bring them to the Results > Grades page.
 
  
===Overview===
+
You can 'dry-run' a regrade. That is, see which scores would change if you did a real re-grade, but without actually updating the grades yet.
On this page the teacher may:
 
*Set preferences for page
 
**Set show/download filters for attempts
 
**Set show/download filters for grading
 
*Set preferences for the report
 
**Page size
 
**Include question marks
 
There are also buttons for:
 
*Regrading the quiz
 
**Regrade all
 
**Dry run a full grade
 
**Regrade attempts marked as needing regrading
 
**Regrade selected attempts
 
**Delete selected attempts
 
*Downloading table data in several formats
 
*A chart showing # of students grades in a range
 
  
Depending upon the filter, the teacher may review each students quiz and see how they answered a specific question from this page.
+
===Report options===
  
 +
* Which students/attempts
 +
* Whether to show the individual question grades
 +
* How many attempts per page
  
  
===Quiz result report===
+
===Report columns===
  
#First name / Surname -
+
# First name / Surname -
 
# Started on - the date and time quiz was started
 
# Started on - the date and time quiz was started
 
# Completed - the date and time completed
 
# Completed - the date and time completed
Line 44: Line 46:
 
[[File:Quiz_result_report_min_6_col_report.png|thumb|center|Report showing only basic information]]
 
[[File:Quiz_result_report_min_6_col_report.png|thumb|center|Report showing only basic information]]
  
===Preference for Results > Grades page===
+
===Display options===
  
 
Show / download options :
 
Show / download options :
Line 57: Line 59:
  
 
Report preferences
 
Report preferences
*Page size
+
* Page size
 
* Show/download marks for each question
 
* Show/download marks for each question
  
===Show results report===
+
Tip: To sort the results by two columns, first click on the column heading you want to be the '''second''' key, and then click on the column heading you want to be the '''primary''' key.
Once you set the above preferences, click on "Show report" button. This produces a list of "Participants".
+
 
 +
 
 +
==Responses report==
 +
 
 +
This is very similar to the Grades report, except that it shows the responses the students gave, rather than the marks they earned.
 +
 
 +
It is also possible to show the question text or the right answer, to compare with the student's response. This is helpful when the question is randomised.
 +
 
 +
 
 +
==Statistics report==
 +
 
 +
This report gives a statistical (psychometric) analysis of the quiz, and the questions within it. The front page of this report gives a summary of the whole test. There are then links to drill down into a detailed analysis of a particular question.
 +
 
 +
The full report (overview, and detailed analysis of all questions) can be downloaded in a variety of formats, as can the quiz structure analysis table.
 +
 
 +
For a general introduction to psychometrics, this [http://www.youtube.com/watch?v=F5Lu3EdsXNo&NR=1 45-minute seminar] gives a very good an approachable introduction.
 +
 
 +
===Overall quiz statistics===
 +
 
 +
This is the first page of the report.
 +
 
 +
====Quiz information====
 +
 
 +
This table gives some basic information about the test as a whole, including some statistics for the total score:
 +
* Quiz name
 +
* Name of the course it is part of
 +
* Open and close dates (if applicable)
 +
* Total number of first/add attempts
 +
* Average grade for first/all attempts
 +
* Median grade
 +
* Standard deviation of grades - how wide the spread of grades is
 +
* [http://en.wikipedia.org/wiki/Skewness Skewness] and [http://en.wikipedia.org/wiki/Kurtosis Kurtosis] of the grade distribution
 +
* Coefficient of internal consistency (sometimes called [http://en.wikipedia.org/wiki/Cronbach%27s_alpha Cronbach Alpha]) - This is a measure of whether all the items in the quiz are testing basically the same thing. Thus it measures the consistency of the text, which is a lower bound for the validity. Higher numbers here are better.
 +
* Error ratio - the variation in the grades comes from two sources. First some students are better than others at what is being tested, and second there is some random variation. We hope that the quiz grades will largely be determined by the student's ability, and that random variation will be minimised. The error ratio estimates how much of the variation is random, and so lower is better.
 +
* Standard error - this is derived from the error ratio, and is a measure of how much random variation there is in each test grade. So, if the Standard error is 10%, and a student scored 60%, then their real ability probably lies somewhere between 50% and 70%.
 +
 
  
With the Select all / Deselect all options you can check / un-check all the names in the list. This will allow you to regrade or deleted the selected attempts.
+
====Quiz structure analysis====
  
:''Tip:'' To sort the results by two columns, first click on the column heading you want to be the '''''second''''' key, and then click on the column heading you want to be the '''''primary''''' key.
+
This lists all the questions in the quiz with various statistics:
  
== Quiz Results > Responses page==
+
* '''Q#''' - shows the question number, question type icon, and preview and edit icons
Same filters for show /download
+
* '''Question name''' - the name is a link to the detailed analysis of this question.
*Only show /down load that are graded for each user
+
* '''Attempts''' - how many attempts included this question
Include check boxes for
+
* '''Facility Index''' - a fancy name for the average score as a percentage. Higher score means easier question.
*Summary of the question
+
* '''Standard Deviation''' - how much variation there was in the scores for this question
*Summary of the response given
+
* '''Random guess score''' - the score the student would get by guessing randomly
*Summary of the right answer
+
* '''Intended/Effective weight''' - Intended weight is simply what you set up when editing the quiz. If question 1 is worth 3 marks out of a total of 10 for the quiz, the the intended weight is 30%. The effective weight is an attempt to estimate, from the results, how much of the actual variation was due to this question. So, ideally the effective weights should be close to the intended weights.
 +
* '''Discrimination index''' - this is the correlation between the score for this question and the score for the whole quiz. That is, for a good question, you hope that the students who score highly on this question are the same students who score highly on the whole quiz. Higher numbers are better.
 +
* '''Discriminative efficiency''' - another measure that is similar to Discrimination index
  
== Quiz Results > Statistics page==
+
Where random questions are used, there is one row in the table for the random question, followed by further rows, one for each real question that was selected in place of this random question.
This page produces a table of statistics and an analysis of each question and a graph of the Facility and discriminative efficiency for each question.
 
  
Preference are to calculate statistics from first attempt or all attempts
+
[[:dev/Quiz_statistics_calculations|Quiz statistics calculations]] gives further details on all these quantities.
  
There is a lot of information on this page concerning the over all quiz.
 
  
===Item analysis===
+
===Quiz question statistics===
This section was written for 1.9, where the statistical report was called "Item analysis".  It needs updating for 2.0 for the specifics.  The concepts are still used in 2.x.
 
  
An item analysis report presents data about each quiz's question in a table. The report gives measures that aid in analyzing and judging the performance of each question. This is done for all students or a group, who took the quiz at the time of the report.  
+
You get to this page by clicking on the name of a question in the Quiz structure analysis.
  
In short, this report will tell the teacher what percentage of the students selected each answer, how the highest scoring quiz takers answered the question compared to lowest scoring quiz takers and other statistical assessment information. It is possible to sort the table by each of its data columns.
+
====Question information====
  
From this report it is also possible to find a link that will allow the teacher to edit any question in the quiz.
+
Some basic information about the question.
  
The statistical parameters used are defined in the help link next "Item Analysis Table" header, or in the help file \moodle\lang\en_utf8\help\quiz\itemanalysis.html
+
====Question statistics====
  
[[Image:Quiz Results ItemAnalysis example.JPG]]
+
This repeats the information from the table row from the Quiz structure analysis that relates to this question.
===Analysis columns===
 
* '''Q#''' - shows the question id number, icon type and a preview popup window link that has an edit link embedded in it.
 
* '''Question Text''' - the question
 
* '''Answer text''' - each answer
 
* '''Partial credit''' - how much credit was given by teacher for each answer
 
* '''R counts''' - how many selected the answer and the total attempts
 
* '''% R''' - the percentage that selected that answer
 
* '''Facility Index (% Correct)''' - the overall difficulty of the question
 
* '''Standard Deviation (SD)''' - measure of variation of selected answers
 
* '''DI & DC columns''' - Question effectiveness measures
 
  
===Descriptions defined===
+
====Analysis of responses====
* '''Facility Index (% Correct)''' - The overall difficulty of the questions.
 
FI = (Xaverage) / Xmax
 
where Xaverage is the mean credit obtained by all users attempting the item, and Xmax is the maximum credit achievable for that item. If questions can be distributed dicotomically into correct / incorrect categories, this parameter coincides with the percentage of users that answer the question correctly.
 
  
* '''Standard Deviation (SD)''' - The range of responses
+
This gives a frequency analysis of the different responses that were given to each part of the question. The details of the analysis depends on the question type, and not all question types support this. For example, essay question responses cannot be analysed.
  
This parameter measures the spread of answers in the response population. If all users answers the same, then SD=0. SD is calculated as the statistical standard deviation for the sample of fractional scores (achieved/maximum) at each particular question.
 
  
* '''DI & DC columns''' - Effectiveness measures
+
===Report options===
Both DC and DI can be used as powerful methods of evaluating the effectiveness of the quiz when assessing differentiation of learners. The advantage of using Discrimination Coefficient as opposed to Discrimination Index is that the former uses information from the whole population of learners, not just the extreme upper and lower thirds. Thus, this parameter may be more sensitive to detect item performance.
 
  
** '''Discrimination Index (DI)'''
+
You can choose whether to run the report on all attempts, or just the first attempt by each student. Some of the calculations used in the report are based on assumptions that may not apply to quizzes that allow more than one attempt.
This provides a rough indicator of the performance of each item to separate high scores vs. scorers. This parameter is calculated by first dividing learners into thirds based on the overall score in the quiz. Then the average score at the analysed item is calculated for the groups of top and bottom performers, and the average scored subtracted. The mathematical expression is:
 
DI = (Xtop - Xbottom)/ N
 
where Xtop is the sum of the fractional credit (achieved/maximum) obtained at this item by the 1/3 of users having the highest grades in the whole quiz (i.e. number of correct responses in this group), and Xbottom is the analog sum for users with the lower 1/3 grades for the whole quiz.
 
  
This parameter can take values between +1 and -1. If the index goes below 0.0 it means that more of the weaker learners got the item right than the stronger learners. Such items should be discarded as worthless. In fact, they reduce the accuracy of the overall score for the quiz.  
+
Also, because computing the statistics takes some time, the report will store the computed values and re-use them for up to 15 minutes. Therefore, there is a display of how recently the statistics were calculated, with a button to recalculate them immediately.
  
** '''Discrimination Coefficient (DC)''' -
 
  
This is another measure of the separating power of the item to distinguish proficient from weak learners.
+
==Manual grading report==
The discrimination coefficient is a correlation coefficient between scores at the item and at the whole quiz. Here it is calculated as:
 
DC = Sum(xy)/ (N * sx * sy)
 
where Sum(xy) is the sum of the products of deviations for item scores and overall quiz scores,
 
N is the number of responses given to this question,
 
sx is the standard deviation of fractional scores for this question and,
 
sy is the standard deviation of scores at the quiz as a whole.
 
  
Again, this parameter can take values between +1 and -1. Positive values indicate items that do discriminate proficient learners, whereas negative indices mark items that are answered best by those with lowest grades. Items with negative DC are answered incorrectly by the seasoned learners and thus they are actually a penalty against the most proficient learners. Those items should be avoided.
+
This page makes it easy to manually grade questions in the quiz.
  
 +
The first screen of the report lists all the questions in the quiz that need to be, or have been, manually graded (for example essay questions) with the number of attempts. There is an option to also show the questions that have been automatically graded, which if useful if you ever need to edit the grades by hand.
  
  
== Quiz Results > Manual grading page==
+
===Grading a question===
This page shows the questions in the quiz that require a teacher to manually grade.  For example, an essay question. This report helps you grade questions of these types by listing just the questions that still need to be graded.
 
:''Tip:'' The grade of any question in a quiz can be manually overridden, and a comment added. A teacher can do this from the review page.
 
  
==Delete selected attempts==
+
In the summary table, there are links you can use to grade all attempts at a question that need grading. There is also the option to edit the grades that have already been awarded.
This is one of the buttons below the grades. This allows you to delete an attempt by a student who has mistakenly used up an attempt when the quiz is set to allow only 1 attempt or so.
+
 
 +
You can choose how many question attempts are shown on each page, and what order they are shown in.
 +
 
 +
If you need to grade anonymously, the go to Settings -> Quiz administration -> Permissions, and remove the permissions 'See student idnumbers while grading' and/or 'See student names while grading'.
 +
 
 +
Note that you can also manually grade student's attempts on the attempt review page.
  
A related discussion is here:
 
http://moodle.org/mod/forum/discuss.php?d=62972
 
  
 
==See also==
 
==See also==
*[http://moodle.org/mod/data/view.php?d=13&perpage=10&search=Quiz+report&sort=46&order=ASC Additional contributed code report plugins] your Moodle administrator has installed.
 
  
*[[Detailed responses report plugin]]
+
* Details of the [[:dev/Quiz_statistics_calculations|Quiz statistics calculations]]
*[[Item analysis theoretical background]]
+
* [http://www.youtube.com/watch?v=F5Lu3EdsXNo&NR=1 presentation about test psychometrics (statistics)]
 +
* [http://moodle.org/mod/data/view.php?d=13&perpage=10&search=Quiz+report&sort=46&order=ASC Additional contributed code report plugins] your Moodle administrator has installed.
 +
 
  
 
[[de:Test-Berichte]]
 
[[de:Test-Berichte]]

Revision as of 13:40, 1 August 2011

Please refer to these notes before editing this page.

You will find the quiz reports in the navigation menu under Quiz name -> Results. You can also get to the quiz reports where you see a summary of the number of attempts that have been made like "Attempts: 123". That link also goes to the reports.

2.1 navigation menu showing a Quiz's Result menu links

Grades report

This shows all the students' quiz attempts, with the overall grade, and the grade for each question. There are links to review all the details of a student's attempt, just as the student would see it.

The results can be downloaded in a variety of formats.

There is a setting form at the top of the page that can be used to control what is displayed.

At the bottom of the page is a graph showing the distribution of scores.

Deleting attempts

This report can be used to delete selected quiz attempts, which may sometimes be necessary.

Note that you cannot edit the quiz after it has been attempted, so that is one reason you may wish to delete all attempts.

Re-grading attempts

The user-interface for re-grading all, or selected, attempts are also on this page.

You can 'dry-run' a regrade. That is, see which scores would change if you did a real re-grade, but without actually updating the grades yet.

Report options

  • Which students/attempts
  • Whether to show the individual question grades
  • How many attempts per page


Report columns

  1. First name / Surname -
  2. Started on - the date and time quiz was started
  3. Completed - the date and time completed
  4. Time taken - the amount of time it took the student to start and submit the quiz
  5. Grade/x - the number of points students scored; 'x' is the maximum number of points
  6. Feedback - any feed back remarks
Report showing only basic information

Display options

Show / download options :

  • Participants who have attempted the quiz
  • Participants who have not attempted the quiz
  • all attempts - list all the students on the course no matter if they did the test or not
  • all Participants who have or have not attempted the quiz

Only show / download attempts check boxes for:

  • that are graded for each user (Highest grade)
  • that have been regraded / are marked as needing regrading

Report preferences

  • Page size
  • Show/download marks for each question

Tip: To sort the results by two columns, first click on the column heading you want to be the second key, and then click on the column heading you want to be the primary key.


Responses report

This is very similar to the Grades report, except that it shows the responses the students gave, rather than the marks they earned.

It is also possible to show the question text or the right answer, to compare with the student's response. This is helpful when the question is randomised.


Statistics report

This report gives a statistical (psychometric) analysis of the quiz, and the questions within it. The front page of this report gives a summary of the whole test. There are then links to drill down into a detailed analysis of a particular question.

The full report (overview, and detailed analysis of all questions) can be downloaded in a variety of formats, as can the quiz structure analysis table.

For a general introduction to psychometrics, this 45-minute seminar gives a very good an approachable introduction.

Overall quiz statistics

This is the first page of the report.

Quiz information

This table gives some basic information about the test as a whole, including some statistics for the total score:

  • Quiz name
  • Name of the course it is part of
  • Open and close dates (if applicable)
  • Total number of first/add attempts
  • Average grade for first/all attempts
  • Median grade
  • Standard deviation of grades - how wide the spread of grades is
  • Skewness and Kurtosis of the grade distribution
  • Coefficient of internal consistency (sometimes called Cronbach Alpha) - This is a measure of whether all the items in the quiz are testing basically the same thing. Thus it measures the consistency of the text, which is a lower bound for the validity. Higher numbers here are better.
  • Error ratio - the variation in the grades comes from two sources. First some students are better than others at what is being tested, and second there is some random variation. We hope that the quiz grades will largely be determined by the student's ability, and that random variation will be minimised. The error ratio estimates how much of the variation is random, and so lower is better.
  • Standard error - this is derived from the error ratio, and is a measure of how much random variation there is in each test grade. So, if the Standard error is 10%, and a student scored 60%, then their real ability probably lies somewhere between 50% and 70%.


Quiz structure analysis

This lists all the questions in the quiz with various statistics:

  • Q# - shows the question number, question type icon, and preview and edit icons
  • Question name - the name is a link to the detailed analysis of this question.
  • Attempts - how many attempts included this question
  • Facility Index - a fancy name for the average score as a percentage. Higher score means easier question.
  • Standard Deviation - how much variation there was in the scores for this question
  • Random guess score - the score the student would get by guessing randomly
  • Intended/Effective weight - Intended weight is simply what you set up when editing the quiz. If question 1 is worth 3 marks out of a total of 10 for the quiz, the the intended weight is 30%. The effective weight is an attempt to estimate, from the results, how much of the actual variation was due to this question. So, ideally the effective weights should be close to the intended weights.
  • Discrimination index - this is the correlation between the score for this question and the score for the whole quiz. That is, for a good question, you hope that the students who score highly on this question are the same students who score highly on the whole quiz. Higher numbers are better.
  • Discriminative efficiency - another measure that is similar to Discrimination index

Where random questions are used, there is one row in the table for the random question, followed by further rows, one for each real question that was selected in place of this random question.

Quiz statistics calculations gives further details on all these quantities.


Quiz question statistics

You get to this page by clicking on the name of a question in the Quiz structure analysis.

Question information

Some basic information about the question.

Question statistics

This repeats the information from the table row from the Quiz structure analysis that relates to this question.

Analysis of responses

This gives a frequency analysis of the different responses that were given to each part of the question. The details of the analysis depends on the question type, and not all question types support this. For example, essay question responses cannot be analysed.


Report options

You can choose whether to run the report on all attempts, or just the first attempt by each student. Some of the calculations used in the report are based on assumptions that may not apply to quizzes that allow more than one attempt.

Also, because computing the statistics takes some time, the report will store the computed values and re-use them for up to 15 minutes. Therefore, there is a display of how recently the statistics were calculated, with a button to recalculate them immediately.


Manual grading report

This page makes it easy to manually grade questions in the quiz.

The first screen of the report lists all the questions in the quiz that need to be, or have been, manually graded (for example essay questions) with the number of attempts. There is an option to also show the questions that have been automatically graded, which if useful if you ever need to edit the grades by hand.


Grading a question

In the summary table, there are links you can use to grade all attempts at a question that need grading. There is also the option to edit the grades that have already been awarded.

You can choose how many question attempts are shown on each page, and what order they are shown in.

If you need to grade anonymously, the go to Settings -> Quiz administration -> Permissions, and remove the permissions 'See student idnumbers while grading' and/or 'See student names while grading'.

Note that you can also manually grade student's attempts on the attempt review page.


See also