Looking at the worst offenders described in Half of all long-term Quantum Country lapses come from just 12% of its questions—can we predict these from initial reader accuracies, i.e. Quantum Country in-essay question accuracy measures?
Not exactly. The 5 questions responsible for the most lapses are 13th, 15th, 29th, 25th, and 7th worst initial accuracy, respectively. So they’re not even really ordered the same way. Meanwhile, the 5 questions with the worst accuracy are 12th, 7th, 9th, 38th, and 20th in terms of responsibility for the most lapses. So these lists are related—it’s not like there’s a total reversal—but the degree doesn’t match.
Are the other essays consistent? Worst 5 offenders in terms of lapses map to (x)th worst initial accuracy:
QCVC: 13th, 15th, 29th, 25th, 7th
Worst 5 initial accuracies in terms of xth worst lapses (after 5 days):
Rough conclusion: initial accuracies don’t reliably predict the questions which account for most of the lapses. If you tried to predict worst-lapses from the 5 worst-initial accuracy, you’d end up in the wrong half of the distribution half the time!
So. What about Quantum Country first repetition session accuracy measures? Do those more reliably predict subsequent behavior?
Looking at QCVC, of the 10 questions with the worst in-essay accuracy, 5 are among the 10 questions with the worst first-review accuracy. Looking only at the bottom 5, 3 are among the bottom 10 in terms of worst first-review accuracies. So these lists are certainly related, but not as strongly as one might imagine.
Worst 5 first-review accuracies in terms of xth worst lapses:
Rough conclusion: If you tried to predict worst-lapses from the 5 worst first-review accuracies, you’d do pretty well. You’d get the right half of the distribution in all these cases, and you’d predict the worst cards correctly much of the time.
QCVC’s worst-lapses expressed in terms of first-review accuracies: 2nd, 7th, 3rd, 9th, 10th
Looking now at Quantum Country second repetition session accuracy measures:
Worst 5 second-review accuracies in terms of xth worst lapses:
QCVC’s worst-lapses expressed in terms of first-review accuracies: 2nd, 7th, 6th, 5th, 8th
Rough conclusion: the second repetition distribution of question accuracies isn’t a meaningfully different predictor from the first repetition.