Half of all long-term Quantum Country lapses come from just 12% of its questions

This figure is drawn from student-question histories which span at least half a year in total length 20210318092419, counting all forgotten questions after the first successful attempt.

This result isn’t very sensitive to the length of the review history. If you look only at histories of at least 1 year in length, the figure drops to 10%. For 90 days, the figure is 13%.

You might wonder: are these lapses all “up front”? Are lapses more evenly distributed among the questions after a student has demonstrated, say, two weeks of retention? No: if you consider only lapses which occur after that point, half of lapses still come from just 13% of questions.

These figures don’t vary much when slicing by essay 20210318095850:

• QCVC: 13% (15 questions) account for half of all long-term lapses among QCVC questions
• Search: 22% (8 questions) …
• Teleportation: 13% (3 questions) …
• QM: 14% (7 questions) …

Looks roughly power-law distributed; here’s QCVC:

The flip side: 95% of Quantum Country questions produce no lapses for most readers’ first half year

In absolute terms

These “hard” questions may produce the most lapses, but Quantum Country users rarely forget after demonstrating five-day retention, so in absolute terms, it’s not very many.

Over the first half year, the hardest decile of questions produce no lapses for roughly half of users, one lapse for another ~30%, rising to 2-4 for the bottom 20%. 20210325113106

Extending that timeframe to a year, the numbers grow but don’t double: the hardest decile still produces no lapses for roughly half of users, one lapse for another ~20%, and around 2-6 for the last 30%.

So what?

If we want to cut down further on long-term lapses, we should focus on this small set of “problem” questions. Simple approaches to try include more conservative scheduling and question refactoring.