Blog / Confidence in Self-Testing: Check Calibration, Not Just Correct Answers
Confidence in Self-Testing: Check Calibration, Not Just Correct Answers
Confidence becomes useful learning data when it is recorded before feedback and compared with later correctness.
October 12, 2026 · 6 min read · By Cepingku Editorial Team
Ready to start?
Start the Adult Cognitive Tasks Profile →Finishing a practice question with a strong feeling of certainty is not the same observation as answering it correctly. Performance is the outcome: correct or incorrect. Confidence is a probability judgment about that same answer before feedback. Keeping both records turns a vague impression of knowing into something that can be checked. Koriat’s original research on judgments of learning examines the cues people use when they judge later recall; those cues and subsequent memory performance are related but are not the same variable.
Calibration asks a checkable question
Calibration asks whether answers given, for example, 80% confidence are actually correct about 80% of the time across a relevant set of attempts. It does not require a person to be certain or uncertain on every item, and it is not a personality label. It simply compares stated probability with an observable outcome. A small one-off set cannot establish a stable personal calibration profile, but it can reveal where a study routine should collect better evidence.
A fictional four-item calculation
Use this deliberately fictional example only. Before checking answers, a learner records confidence for four practice answers; after checking, the learner records each same-answer outcome, forming (0.90, 1), (0.80, 0), (0.60, 1), and (0.50, 0). The first number is the probability that that answer is correct; 1 means correct and 0 means incorrect. Brier introduced verification of probability forecasts with squared differences. The common binary-event single-outcome convention averages (p−o)²; scikit-learn documents that form. The original 1950 Brier formulation sums errors for both outcome categories, so for binary outcomes it is twice the single-outcome value. Thus the fictional mean below is 0.265 under the single-outcome convention and 0.530 under Brier’s two-category sum. Keep the convention fixed when comparing records.
| Item | Confidence, outcome, and squared error |
|---|---|
| 1 | (0.90−1)² = 0.01 |
| 2 | (0.80−0)² = 0.64 |
| 3 | (0.60−1)² = 0.16 |
| 4 | (0.50−0)² = 0.25 |
| Mean for this fictional set | (0.01+0.64+0.16+0.25) ÷ 4 = 0.265 |
The value 0.265 is not a diagnosis, a norm, a measure of calibration by itself, or a score for this site. It is only an arithmetic summary of four invented observations. Squared-error summaries also reflect how well questions separate different confidence levels and how stated probabilities are distributed. The second item is useful precisely because it was high-confidence and wrong: it identifies a place to revisit the rule, condition, or wording that produced the error. Erasing such answers because they feel embarrassing would remove the evidence needed to improve a study method.
Inspect groups, not only one total
The same record can be grouped by confidence. The 90% and 80% answers fall in an 80–100% bin, yet only one of the two is correct: 50% observed accuracy in that tiny bin. The 60% bin has one correct answer and the 50% bin has one incorrect answer. Four observations are far too few for a general conclusion, but the logic is clear. Over many comparable questions, a consistently high-confidence bin with much lower accuracy signals that ‘I know this’ needs a more demanding check.
Bins also discard detail and their boundaries are choices. Keep the original probability, question version, timing, and correctness alongside any grouped chart. Two learners can have the same 75% correct rate while differing sharply in calibration: one may reserve 90% confidence for answers that usually hold up, while the other may be highly certain on several errors.
Make the check delayed and observable
Immediate familiarity after reading can be informative for finding an error, but it does not guarantee later retrieval. In two experiments, Roediger and Karpicke compared repeated studying with immediate free-recall tests after prose study. Repeated study was better on a final test after five minutes, whereas prior testing produced greater retention on tests after two days and one week, even though repeated study increased confidence. That result does not prescribe one delay for every learner or material. It does give a practical reason to include an unaided later check instead of treating a just-read feeling of fluency as proof.
A reproducible study log
Choose one small, defined target: ten terms or five already studied calculation problems. Each time you answer without notes, record 50%, 60%, 70%, 80%, or 90% confidence before feedback, then pair that probability with the correctness of that same answer. At a prewritten later time, answer new or rewritten questions on the same learning target; treat these as a separate round by recording a fresh probability before feedback and pairing it with each new item’s own outcome. Do not score the first round’s probabilities against later new items. Calculate squared errors within each round, list the answers at 80% or above that were wrong, and inspect the rule, boundary condition, or misread wording behind each one. Repeat only after correcting the reason for the error. The Adult Cognitive Tasks Profile is an assessment entry point for reflection on task responses. This article does not claim that it uses these questions, has norms or psychometric properties from these studies, or can estimate an individual’s memory or calibration.
Want to see where you land?
Take the Adult Cognitive Tasks Profile and get your results in minutes.
Take the Adult Cognitive Tasks Profile →