Note: You are currently viewing documentation for Moodle 3.1. Up-to-date documentation for the latest stable version of Moodle is probably available here: Using certainty-based marking.

Using certainty-based marking: Difference between revisions

From MoodleDocs
No edit summary
Line 3: Line 3:
'''Note:''' This documentation is based on the more comprehensive [http://www.ucl.ac.uk/lapt/moodle/cbm/cbm.htm  Certainty-Based Marking] from Tony Gardner-Medwin.
'''Note:''' This documentation is based on the more comprehensive [http://www.ucl.ac.uk/lapt/moodle/cbm/cbm.htm  Certainty-Based Marking] from Tony Gardner-Medwin.


==Certainty-Based marking (CBM)==
=What is Certainty-Based marking (CBM)?=
*After each answer, you say how sure you are that your answer is correct.
*After each answer, you say how sure you are that your answer is correct.
*This is on a 3-point scale: C=1 (low), C=2 (mid) or C=3 (high)
*This is on a 3-point scale: C=1 (low), C=2 (mid) or C=3 (high)
*We don't rely on words like 'sure' or 'very sure' because these mean different things to different people
*We don't rely on words like 'sure' or 'very sure' because these mean different things to different people
*The mark scheme and the risk of a penalty determine when you should use each C level:
*The mark scheme and the risk of a penalty determine when you should use each C level:
==How CBM works==


{| class="wikitable"
{| class="wikitable"
Line 58: Line 55:
*If unsure, you should avoid any risk of penalty by choosing C=1
*If unsure, you should avoid any risk of penalty by choosing C=1
*In between, you are best to use C=2: you gain 2 or lose 2 depending on whether you are right.
*In between, you are best to use C=2: you gain 2 or lose 2 depending on whether you are right.
*The graph below shows how the average mark at each C level depends on the probability that your answer will be right.
**Suppose you think you only have a 50% chance of being right: The highest graph for 50% on the bottom scale is black, for C=1. So you will expect to boost your marks on average most by acknowledging your low certainty (C=1).
**If you think you can justify your answer well, with more than an 80% chance of being correct, then the red graph is highest, for C=3. Opt for this.


[[File:cfgraphs3.gif|link=|]]
[[File:cfgraphs3.gif|link=|]]
*The graph shows how the average mark at each C level depends on the probability that your answer will be right.
**Suppose you think you only have a 50% chance of being right: The highest graph for 50% on the bottom scale is black (for C=1). So you will expect to boost your marks on average most by acknowledging your low certainty (C=1).
**If you think you can justify your answer well, with more than an 80% chance of being correct, then the red graph is highest, for C=3. Opt for this.
*Note that you can't ever expect to gain by misrepresenting your certainty. If you click C=3 (the red line) when you aren't sure, you will expect to do badly - with very likely a negative mark. You might be lucky; but on average you will lose! If you understand the topic well, and think your answer is reliable, then you will lose if you opt for C=1 or C=2 rather than C=3.  You will do best if you can distinguish which answers are reliable and which uncertain.


*Note that you can't ever expect to gain by misrepresenting your certainty. If you click C=3 (the red line) when you aren't sure, you will expect to do badly - with very likely a negative mark on average. If you understand the topic well, and think your answer is very probably right, then you will lose if you opt for C=1 or C=2 rather than C=3. You do best if you can distinguish which answers are reliable and which uncertain.
==Feedback about your CBM performance==
*If you are using CBM with feedback given after each answer, then the mark tells you a lot: -6 will make you ask how you justified confidence in a wrong answer. Think (along with any available explanations and your books or notes) how to tie together different bits of your knowledge.
*If you repeatedly get answers right with C=1 then ask yourself if perhaps you know the subject better than you think. Ask how you might relate your answers to things that you definitely do know and are more confident about.
*When you finally submit, you will see your percentages correct for each C level you used. Ideally these might be around 50%,70%,90% for C=1,2,3 whether you know the subject well (lots of answers at C=3) or poorly (lots at C=1). The feedback will warn if you are tending to overestimate or underestimate how reliable your answers are. If overconfident, think about why! Perhaps you were careless; perhaps you have misconceptions, or are giving confident answers to questions you don't really understand or that are phrased ambiguously. Use this to guide your study and to improve your interaction with teachers.
*Note that some questions in a quiz may carry more weight than others, so for example you might get -12 if the question carries double weight and you make a confident error. Since this makes feedback harder to interpret, it probably isn't a good idea for quiz-setters to vary weights in this way when using CBM.
*If you've only answered a few questions, CBM feedback may not mean very much: 2 or 3 unconfident correct answers may just be luck! What is important is that you think about why you were confident or unconfident, and learn from that.


==Scores==
=CBM Grades and Scores=
==The CBM Grade in Moodle==
==Accuracy==
==CBM Bonus==
==Enhanced Accuracy with CBM: Accuracy + Bonus==
For more information on scores (relating to the UCL version of CBM but also relevant to Moodle) please see [http://www.ucl.ac.uk/lapt/moodle/cbm/cbmscores.htm Self test scores with and without CBM] by Tony Gardner-Medwin.
For more information on scores (relating to the UCL version of CBM but also relevant to Moodle) please see [http://www.ucl.ac.uk/lapt/moodle/cbm/cbmscores.htm Self test scores with and without CBM] by Tony Gardner-Medwin.


[[es:Puntuación Basada en Certeza]]
[[es:Puntuación Basada en Certeza]]

Revision as of 13:14, 12 December 2013


Note: This documentation is based on the more comprehensive Certainty-Based Marking from Tony Gardner-Medwin.

What is Certainty-Based marking (CBM)?

  • After each answer, you say how sure you are that your answer is correct.
  • This is on a 3-point scale: C=1 (low), C=2 (mid) or C=3 (high)
  • We don't rely on words like 'sure' or 'very sure' because these mean different things to different people
  • The mark scheme and the risk of a penalty determine when you should use each C level:
Certainty level C=1 C=2 C=2
Mark if correct 1 2 3
Mark if wrong 0 -2 -6
Probability correct < 67% 67–80% >80%
Description Unsure Mid Quite sure
  • Certainty levels 1, 2, 3 always give you marks 1, 2, or 3 when you are correct
  • If you are wrong, then unless you opted for C=1 you will lose marks: -2 at C=2 and -6 at C=3

Why use CBM?

  • To make students think about how reliable their answer is.
  • To encourage students to try to understand the issues, not just react immediately to a question.
  • To challenge: if a student won't risk losing marks if wrong then they don't really know the answer.
  • If a student is a careful thinker but not very confident. they will gain in confidence.
  • It is more fair - a thoughtful and confident correct answer deserves more marks than a lucky hunch.
  • Students need to pay attention if they make confident wrong answers: think, reflect, learn!
  • Efficient study requires constantly questioning how our ideas arise and how reliable they are.

lplconf.gif

How to decide on the best certainty level

  • If you're sure, obviously you do best with C=3. But you will lose twice over (-6) if you are actually wrong!
  • If unsure, you should avoid any risk of penalty by choosing C=1
  • In between, you are best to use C=2: you gain 2 or lose 2 depending on whether you are right.

cfgraphs3.gif

  • The graph shows how the average mark at each C level depends on the probability that your answer will be right.
    • Suppose you think you only have a 50% chance of being right: The highest graph for 50% on the bottom scale is black (for C=1). So you will expect to boost your marks on average most by acknowledging your low certainty (C=1).
    • If you think you can justify your answer well, with more than an 80% chance of being correct, then the red graph is highest, for C=3. Opt for this.
  • Note that you can't ever expect to gain by misrepresenting your certainty. If you click C=3 (the red line) when you aren't sure, you will expect to do badly - with very likely a negative mark. You might be lucky; but on average you will lose! If you understand the topic well, and think your answer is reliable, then you will lose if you opt for C=1 or C=2 rather than C=3. You will do best if you can distinguish which answers are reliable and which uncertain.

Feedback about your CBM performance

  • If you are using CBM with feedback given after each answer, then the mark tells you a lot: -6 will make you ask how you justified confidence in a wrong answer. Think (along with any available explanations and your books or notes) how to tie together different bits of your knowledge.
  • If you repeatedly get answers right with C=1 then ask yourself if perhaps you know the subject better than you think. Ask how you might relate your answers to things that you definitely do know and are more confident about.
  • When you finally submit, you will see your percentages correct for each C level you used. Ideally these might be around 50%,70%,90% for C=1,2,3 whether you know the subject well (lots of answers at C=3) or poorly (lots at C=1). The feedback will warn if you are tending to overestimate or underestimate how reliable your answers are. If overconfident, think about why! Perhaps you were careless; perhaps you have misconceptions, or are giving confident answers to questions you don't really understand or that are phrased ambiguously. Use this to guide your study and to improve your interaction with teachers.
  • Note that some questions in a quiz may carry more weight than others, so for example you might get -12 if the question carries double weight and you make a confident error. Since this makes feedback harder to interpret, it probably isn't a good idea for quiz-setters to vary weights in this way when using CBM.
  • If you've only answered a few questions, CBM feedback may not mean very much: 2 or 3 unconfident correct answers may just be luck! What is important is that you think about why you were confident or unconfident, and learn from that.

CBM Grades and Scores

The CBM Grade in Moodle

Accuracy

CBM Bonus

Enhanced Accuracy with CBM: Accuracy + Bonus

For more information on scores (relating to the UCL version of CBM but also relevant to Moodle) please see Self test scores with and without CBM by Tony Gardner-Medwin.