Starting late 2019, Quantum Country reports progress to users in terms of Demonstrated retention. This value increases when the user successfully remembers an item across a longer interval. Readers are encouraged to view these values as “levels”: 5 days, 2 weeks, 2 months, etc. You’re given a “next goal” in terms of this measure: “you’ve successfully demonstrated 2 weeks of retention across this essay’s questions; your next goal is 1 month.”
But… is this a coherent measure of progress? Once you’ve reached “1 month of demonstrated retention,” does that mean you’re very unlikely to forget across a period of less than a month in the future (assuming you continue to review according to our schedule)? To put it another way, does this measure reliably establish a lower bound?
Looks like: yes. 20210316150459
After demonstrating 2 weeks of retention, about 1.2% of traces include backsliding (~0.8% once, 0.3% twice, 0.1% three times, much less above that)
After demonstrating 1 month of retention, about 1% of traces include backsliding. (0.6% once, 0.2% twice, <= 0.006% above that).
After demonstrating 2 months of retention, about 0.8% of traces include backsliding below that level (0.5% once, 0.2% twice, much less above that).
This is lower than I would have expected. I imagine it’s skewed by the large number of traces which don’t extend very far past that point. Let’s look at the (certainly much smaller) pool of data where the total length of the trace is a year or more. (20210316151815).
Worth noting that only about 200 users have traces longer than a year. But given these numbers’ proximity to the first batch of numbers, I’m inclined to say the true values lie in this ballpark, at least for that first year. If you lower the threshold to half a year, you get about 500 users, and similar numbers: 2.3% / 2.2% / 1.5% / 0.8% / 3.4% for backsliding after demonstrating 5 days / 2 weeks / 1 month / 2 months / 4 months.
Now, in these numbers, I’ve combined instances in which multiple backslides occurred. These are worth special attention, but it’s worth noting that the vast majority of backslides in our data set happen only once.
Looking at this gives us a powerful way to interpret the goal of “optimizing” a Spaced repetition memory system: after the first few repetitions, once an item has been answered correctly a couple times, you’re just trying to maintain that memory as cheaply as possible.
Most traces don’t include lapses, though. So in terms of creating interventions, it’s perhaps more helpful to consider: if a lapse just happened, how likely is it to cascade into a backslide? The conditional probabilities by demonstrated retention level, over 1-year+ traces 20210325163220:
(updated 2021-11-18):
So the chances of a cascade still aren’t very high.
Unsurprisingly, given Half of all long-term Quantum Country lapses come from just 12% of its questions, a small number of questions are responsible for most of the backslides. 11 questions (5% of all questions on the site) are responsible for half the backslides at the 5-day level. In fact, the three “what’s the matrix representation of…?” questions from QCVC are by themselves responsible for 21% of the backslides at the 5-day level. 20210325164658
This finding has a few implications:
Follow-up questions: