Quantum Country users seem to forget most prompts quite slowly

Users of Quantum Country have a much shallower overall Forgetting curve than I would have initially expected.

But now (2022-01-17) I think this is mostly because QCVC questions are initially forgotten at very different rates: looking at all questions flattens the effect of forgetting because many questions are easy. And there’s plenty of forgetting among “lower-ability” readers.

Data from 2021 between-users schedule RCT

Observations I’ve made so far, using data from the 2021-04 Quantum Country schedule experiment (last updated 2021-11-16):

• For QCVC prompts which were initially remembered, there’s very little forgetting (20211011120323):
  • 1 day (using users from 2019’s original schedule): 89%
  • 5 days (using aggressiveStart users): 86%
  • 1 week (137 readers, 6174 reviews): 86%
  • 2 weeks (140 readers, 5814 reviews): 85%
  • 1 month (89 readers, 4108 reviews): 84%
  • 2 months (49 readers, 1599 reviews): 83%
  • 3 months (15 readers, 295 reviews): 78%
• For QCVC prompts which were initially forgotten, there’s a sharper drop; 84% to 56%; 89% at one day; see longer discussion: Quantum Country users who forget in-essay exhibit sharp forgetting curves
• For the ten QCVC questions with the lowest in-essay accuracy (20211116152859):
  • 1 day (using users from 2019’s original schedule): 78%
  • 1 week (76 readers, 420 reviews): 77%
  • 2 weeks (61 readers, 338 reviews): 69%
  • 1 month (51 readers, 291 reviews): 64%
  • 2 months (28 readers, 118 reviews): 57%
  • This isn’t just due to Quantum Country users who forget in-essay exhibit sharp forgetting curves; here are the recall rates for people who remembered the question initially 20211130185543:
  • 1 week: 74% (N=74 readers, 273 reviews)
  • 2 weeks: 71% (N=67 readers, 277 reviews)
  • 1 month: 65% (N=47 readers, 210 reviews)
  • 2 months: 65% (N=25 readers, 85 reviews)
• For the ten QCVC questions with the highest in-essay accuracy (20211116163825):
  • 1 week (115 readers, 681 reviews): 95%
  • 2 weeks (120 readers, 649 reviews): 95%
  • 1 month (77 readers, 443 reviews): 95%
  • 2 months (42 readers, 181 reviews): 93%

So the “hardest” prompts should be studied on a dramatically different schedule from the “easiest” prompts. Even 1 day isn’t a short enough interval to achieve 90%. Prompts which are initially forgotten should be restudied rapidly.

This conclusion seems to extend to longer delays after the first delayed session. For readers who remembered the answer in-essay and then again in their first review session:
20211123171811:

• 1 week -> 3 weeks (37 readers, 1420 reviews): 96%
• 1 week -> 6 weeks (10 readers, 447 reviews: 93%
• 2 weeks -> 6 weeks (38 readers, 1205 reviews): 94%
• 1 month -> 3 months (16 readers, 326 reviews): 96%

For readers who forgot in-essay and recovered immediately thereafter, subsequent longer delays don’t seem to pose too much of a penalty 20211123104202:

• 1 week -> 3 weeks (22 readers, 127 reviews): 89%
• 1 week -> 6 weeks (7 readers, 56 reviews): 71%
• 2 weeks -> 6 weeks (18 readers, 82 reviews): 98%

Theories about why curves might be flatter than expected

Now (2022-01-17) I think the curves are pretty plausible, given QCVC questions are initially forgotten at very different rates. But below are some more explanations.

One possible explanation: To what extent do people “lie” when self-grading spaced repetition prompts?. But 2021-04 Quantum Country schedule experiment lends additional evidence that this doesn’t explain what’s going on. The median user in each condition is admitting that they’ve forgotten roughly 15% of prompts. Sure, maybe they’re lying to some degree, but they’re admitting to forgetting plenty. Why doesn’t this figure vary?

There’s also a selection effect: the population of people who comply with reviewing when their first session is two months later is probably skewed more conscientious than those whose first session is one week later. I’d expect this to inflate the accuracies of the later samples, artificially flattening the curve.

Data from earlier experiments

Data from 2020-01 Quantum Country efficacy experiment: see Accuracy rates for withheld Quantum Country questions were roughly equal at 5 days and 1 month (2020 efficacy experiment)

2021 forgetting curve experiments

See Log: Quantum Country analysis, 2021-03-03 and 2021-03-18 for more; excerpts:
(conditioned on a correct first answer)

(missed first attempt)