Accuracy rates for withheld Quantum Country questions were roughly equal at 5 days and 1 month (2020 efficacy experiment)

2020-01 Quantum Country efficacy experiment seems not to have measured any actual forgetting. While groups with prompts withheld performed much worse than those who had been practicing, we didn’t measure any decay in performance between the condition which had prompts withheld for five days and the condition which had prompts withheld for a month: the median user remembered 5/9 questions in the first case and 6/9 in the second.

In fact, there might have been a small increase in performance over time between these two groups. I’ve controlled for obvious survivalship effects, so it’s possible that these results are due to the effects of practicing all the other questions, some of which indirectly depend on the same knowledge.

• Distribution of correct answer counts at first review session (i.e. 5 days): 20210322154652
  • 25th / 50th / 75th %ile: 4, 5, 7
  • Average: 5.3
  • N = 72
  • 
  • n.b. this includes only members of this condition who eventually reviewed all these prompts after 30 days, to control for compliance
• Distribution of correct answer counts at 30 days: 20210322155242
  • 25th / 50th / 75th %ile: 4, 6, 8 20210322155308

• Average: 6.1
• N = 131

Per-question analysis

On a per-question basis, the story looks basically the same 20210322163553. One outlier is that the number of users who could correctly answer “What’s the length of |ᴪ> in Dirac notation?” more than doubled. Maybe this is the kind of thing which would be especially reinforced by practicing other questions in the essay, but it’s not clear to me that the other questions should be all that different.