Spaced repetition memory system

A spaced-repetition memory system combines the Testing effect and the Spacing effect to enable efficient memorization of many thousands of facts (Spaced repetition memory systems are extremely efficient). Some people also use them for a broader set of tasks (see below). Spaced repetition memory systems make memory a choice, but they’re not just for rote facts: Spaced repetition memory systems can be used to develop conceptual understanding.

The first consumer system of this kind was Supermemo, created by Piotr Wozniak. It adopted and popularized the term “spaced repetition”; prior literature had used a variety of terms (often referring to more specific facets of the underlying phenomenon).

• Related systems and possibilities:
  • Differences in my approach to spaced repetition systems
  • Implementations:
    • Traditional: Supermemo, Mnemosyne, Anki
    • Unusual variations:
      • Mnemonic medium
      • Execute Program
      • RemNote
      • Mochi
      • Readwise
      • Chessable MoveTrainer
      • Revuu
  • Unusual applications of spaced repetition memory systems
  • Spaced repetition systems can be used to program attention
    • Spaced repetition systems as catechism
  • Spaced repetition memory systems can be used to prompt application, synthesis, and creation
• On properties of memory systems:
  • What’s the maximum intake rate of an efficient spaced repetition memory system?
    • Optimizing spaced repetition system schedules
  • Self-grading vs. machine-grading in spaced repetition memory systems
  • To what extent does spaced repetition memory system practice constitute deliberate practice?
  • SRS retry mechanism
• Barriers to adoption
  • Many people view memory as unimportant to deep creative work
  • Writing good spaced repetition memory prompts is hard
    • Important attributes of good spaced repetition memory prompts
    • Studying another person’s spaced repetition memory prompts is usually ineffective
    • The limiting factor in spaced repetition system capacity is writing enough good prompts
    • Using machine learning models to generate good spaced repetition prompts
  • The critical thing to optimize in spaced repetition memory systems is emotional connection to the review session and its contents
    • Regular spaced repetition memory practice is an onerous habit to adopt
      • Spaced repetition memory systems don’t rapidly demonstrate their benefits
      • Spaced repetition memory systems make you feel like your memory is worse than it really is
    • Spaced repetition review sessions often become boring and detached without a steady stream of new prompts
    • The dominant culture around spaced repetition memory systems is fixated on meaningless goals
• Responses to common objections
  • SRS are only relevant for rote learning of facts: “school learning”
    • People underrate the importance of memory because of bad experiences with rote memorization in school
    • Existing SRS are almost exclusively used for simple declarative knowledge, contra Spaced repetition memory systems can be used to develop conceptual understanding
    • Memory systems help you focus on deeper engagement by automating away rote memorization
    • Memory augmentation can accelerate the unpleasant early stages of learning a subject
    • (caveat: it’s hard to write prompts for conceptual understanding!)
  • SRS are only for material you don’t care about enough to remember “naturally.” Better to “learn by doing.”
  • External memory aids are sufficient
  • Mnemonic techniques obviate SRS
  • Knowledge learned through SRS doesn’t transfer
  • The real value in SRS comes from making one’s own prompts
    • Maybe! How important is it to write your own spaced repetition memory prompts?
• On algorithms
  • Simple spaced repetition algorithms mistreat expected failures
  • SRS retry mechanism
  • Optimizing spaced repetition system schedules

References

Branwen, G. (2009). Spaced Repetition for Efficient Learning. Retrieved December 16, 2019, from https://www.gwern.net/Spaced-repetition

Who invented the name: spaced repetition? - supermemo.guru

Backlinks

• Mnemonic medium
  • The mnemonic medium embeds a Spaced repetition memory system within narrative prose. Spaced repetition memory systems make memory a choice, but they’ve suffered significant barriers to adoption. The mnemonic medium is designed to solve many of these problems.
• Spaced repetition systems can be used to program attention
  • Spaced repetition memory systems make memory a choice, but the computerized component’s value lies specifically in dynamically scheduling and selecting questions to be reviewed. In some sense, the efficacy of a Spaced repetition memory system comes from its power to program your attention (Programmable attention). Think: “{cron} for your mind.”
• Spacing effect
  • Spaced repetition memory system algorithms taken advantage of this to implement efficient learning systems.
• Quizlet
  • Quizlet is a commercial Spaced repetition memory system.
• Execute Program
  • Execute Program offers interactive lessons on programming tools by Gary Bernhardt. The lessons interleave prose with live programming problems. Akin to the Mnemonic medium, the site integrates a Spaced repetition memory system to repeat lessons’ tasks over time.
• Knowledge work should accrete
  • A Spaced repetition memory system achieves this for memory: when you find information useful, you can invest a little effort to make sure you always have it available. Over time, one’s spaced repetition library accumulates thousands of questions, and (I strongly suspect) that knowledge makes it easier to be an effective knowledge worker later.
• Spaced repetition memory systems are extremely efficient
  • Using a Spaced repetition memory system, you can memorize the answers thousands of questions in just a few minutes a day. This is the unintuitive consequence of an exponential at work in the Spacing effect: Spaced repetition yields exponential returns for small increases in effort.
• Spaced repetition memory systems make memory a choice
  • One fascinating consequence of using a Spaced repetition memory system is that they make memory a choice. Once you’ve adopted a memory practice, if you want to remember something, you can simply cause it to happen: just take a few moments to write a question about it. Over the next few weeks, you’ll encode it durably into long-term memory. Moreover, it’ll only take you a few minutes cumulatively: Spaced repetition memory systems are extremely efficient. So not only do these systems make memory a choice, but they make the choice very low-stakes: Deciding to remember something with a spaced repetition system is a lightweight gesture.
• Elaborative encoding
  • Some Spaced repetition memory system users explicitly write prompts to promote elaborative encoding. Experiments by Karpicke and Smith (2012) suggest this may not be adding much, vs. retrieval practice alone. Related: Retrieval practice appears to be a more effective learning activity than elaborative encoding.
• Writing good spaced repetition memory prompts is hard
  • People regard flashcards as something trivial from their school days, so they don’t take writing them very seriously. But it’s awfully hard to write good prompts for a Spaced repetition memory system. For example, good prompts:
• Piotr Wozniak
  • Creator of Supermemo and popularizer of Spaced repetition memory system and Incremental reading.
• Supermemo
  • The original digital Spaced repetition memory system, created by Piotr Wozniak. Includes support for various related personal information management systems, such as Incremental reading.
• Spaced repetition memory systems can be used to prompt application, synthesis, and creation
  • A Spaced repetition memory system like Anki is primarily designed to help people memorize a lot of declarative knowledge, like vocabulary. But the same mechanisms can be used to create relatively unorthodox cards which prompt application, synthesis, and creation.
• The mnemonic medium can be extended to one’s personal notes
  • In using a Spaced repetition memory system, you’ll fill it with notes on what you’re learning, observing, and thinking. Unfortunately, Existing spaced repetition systems discourage evergreen notes. A memory system will help you retain and continuously engage with what you write, but it won’t much help you build on those ideas over time. An Evergreen notes system will help you build on your ideas over time, but it won’t help you retain and continuously engage with those notes (outside of Evergreen note maintenance approximates spaced repetition). So you’re stuck either duplicating your efforts messily in two separate systems, or giving up one system’s benefits.
• Important attributes of good spaced repetition memory prompts
  • This note collects ideas about how to encode knowledge into Spaced repetition memory system prompts, both to support memory of facts but also to foster richer understanding (Spaced repetition memory systems can be used to develop conceptual understanding).
• Similarities and differences between evergreen note-writing and Zettelkasten
  • I’ve embedded a Spaced repetition memory system into my notes: The mnemonic medium can be extended to one’s personal notes. This has substantially changed the medium, but in ways I don’t yet understand very well.
• Execute Program’s lessons don’t unlock until you’ve successfully reviewed their prerequisites
  • Students of Execute Program can’t immediately access all the lessons in a course. Most of a course’s lessons begin “locked.” Those lessons depend on other lessons in the course: the only initially-unlocked lessons are {those with no dependencies in the course}. Lessons only unlock once {all their dependencies have not only been read, but also recently reviewed (via embedded Spaced repetition memory system mechanisms)}.
• Programmable attention
  • Michael Nielsen has suggested that the review sessions of a Spaced repetition memory system don’t just help you remember things: it orchestrates your repeated attention over time across hundreds of tiny tasks, too many to manage by hand. Systems like these are a form of programmable attention.
• Unusual applications of spaced repetition memory systems
  • While a Spaced repetition memory system is primarily designed to help people remember facts, their flashcard mechanism can be used for a variety of other purposes.
• Studying another person’s spaced repetition memory prompts is usually ineffective
  • Once one understands that Spaced repetition memory systems make memory a choice and that Learning increasingly complex ideas may amount to forming larger effective chunk sizes, it’s awfully tempting to gobble up new knowledge by downloading others’ decks of Spaced repetition memory system prompts. But assuming you’ve never studied it before, you’re much better off reading a textbook chapter on cell biology than you would be memorizing someone else’s deck: you must Learn before you memorize. Separately, Spaced repetition memory prompts alone are a poor communications medium; you still need the textbook’s narrative to ground the material.
• Spaced repetition memory systems can be used to develop conceptual understanding
  • The most obvious way to use a Spaced repetition memory system is to memorize simple facts: names, definitions, numerical constants, etc. The vast majority of memory system users appear to only ever use them for this purpose. But spaced repetition isn’t just useful for memorizing simple facts. The same mechanism can be used to deeply understand complex material, particularly if you write our own prompts (How important is it to write your own spaced repetition memory prompts?). Such conceptually-oriented prompts might test you on connections, implications, causes, consequences. These broader ideas don’t naively feel like facts which can be memorized—but they often are. Besides the benefit of memorizing these conceptual ideas, engaging with higher-level relationships in this way over time can keep you in contact with the topic and help you internalize it more deeply. (related: The mnemonic medium keeps readers in contact with material over time)
• Two-component model of memory
  • In practical use as part of a Spaced repetition memory system, a third parameter for item complexity must generally be introduced, since SRS items are generally not actually atomic, as far as our brains are concerned.
• Memory augmentation can accelerate the unpleasant early stages of learning a subject
  • One common negative reaction to working with a Spaced repetition memory system is: “I’d much rather get my hands dirty: then I’ll naturally end up remembering whatever’s important.” (Enabling environments focus on creating opportunities for growth and action, not on skill-building) But when you’re first interacting with a new subject, it’s hard to take any meaningful steps at all: at least for a while, you often can’t hold enough of the new terms and ideas in your head simultaneously long enough to do anything meaningful (related: Novices in enabling environments often can’t do what’s enabled). In this context, a Spaced repetition memory system can help by accelerating you through this awkward, unpleasant stage to the point where you can actually have a meaningful experience with the material (related: Enabling environments’ activities directly serve an intrinsically meaningful purpose).
  • One common negative reaction to working with a Spaced repetition memory system is: “I’d much rather get my hands dirty: then I’ll naturally end up remembering whatever’s important.” (Enabling environments focus on creating opportunities for growth and action, not on skill-building) But when you’re first interacting with a new subject, it’s hard to take any meaningful steps at all: at least for a while, you often can’t hold enough of the new terms and ideas in your head simultaneously long enough to do anything meaningful (related: Novices in enabling environments often can’t do what’s enabled). In this context, a Spaced repetition memory system can help by accelerating you through this awkward, unpleasant stage to the point where you can actually have a meaningful experience with the material (related: Enabling environments’ activities directly serve an intrinsically meaningful purpose).
• Readwise
  • They’ve gradually introduced some features of a Spaced repetition memory system. In the course of reading through your daily highlights, you can smoothly “upgrade” the interesting ones into cloze deletions or even custom question/answer prompts. This is a clever strategy for onboarding people new to spaced repetition: the product delivers (slight) value if you’re totally passive, but as users become invested, they can apply incremental effort over time and get more out of the product.
• The critical thing to optimize in spaced repetition memory systems is emotional connection to the review session and its contents
  • Many researchers are trying to make a more efficient Spaced repetition memory system, but I believe that the critical thing to optimize is emotional connection to the review session and its contents—and conversely, to ruthlessly minimize elements which provoke a sigh. These systems, left to their natural inclinations, naturally decay to produce dutiful sessions which feel disconnected from anything that matters to you.
• OS-level spaced repetition system
  • What if there were an “OS-level” Spaced repetition memory system? What if, rather than living “inside an app’s shoebox”, as in Anki and other existing tools, prompts were framed more like files in folders—readable and writable throughout the system?
• Spaced repetition may be a helpful tool to develop or change habits
  • You could memorize pieces of this with a Spaced repetition memory system, but you can also use the same system to help “install” this habit in your mind. Here are some example questions you might try:
• Hickory Training
  • Defunct startup which attempted to sell a Spaced repetition memory system as a service to companies, for training purposes. The CEO was {Brian Tobal}.
• Sana Labs
  • an integrated Spaced repetition memory system suggests re-review at a later date
  • I imagine that there will be some Spaced repetition memory system component, given the big retention circle.
• RemNote
  • RemNote is a web-based Note-writing system with a structured knowledge model and first-class operations for adapting notes into a Spaced repetition memory system, a la The mnemonic medium can be extended to one’s personal notes.
• Spaced repetition systems as catechism
  • Likewise, a Spaced repetition memory system’s review sessions help readers remember the material of a book, but they also trigger readers to re-engage with material they’ve read over time. That’s practically useful because many readers will see new connections after they’ve sat with the content for a few weeks. But maybe it also fosters a change in identity: you’re not just a person who read that essay one time, you’re a “student of that topic” in some more continuous sense.
• Spaced repetition review sessions often become boring and detached without a steady stream of new prompts
  • Daily practice with a Spaced repetition memory system should feel vital, relevant. When one hasn’t added many new prompts in a while, the daily review sessions begin to feel stuck in the past, disconnected from what you’re thinking about right now. This appears to be a common failure mode for new users, who add many new prompts at once in a fit of initial enthusiasm, then feel bored when they’re still reviewing those same prompts (and no others) months later.
• The mnemonic medium supplies expert-authored prompts to remove the burden of prompt-writing
  • Writing good spaced repetition memory prompts is hard, but the Mnemonic medium removes that barrier to adopting a Spaced repetition memory system by supplying author-written prompts, which allow the readers to retain the material without writing their own.
• Spaced repetition memory prompts should be written to discourage shallow “pattern matching”
  • One challenge for the efficacy of a Spaced repetition memory system is that sometimes users memorize answers to questions shallowly, through “pattern matching,” rather than by integrating the knowledge more deeply or actually thinking about the question. But Spaced repetition memory prompts should ensure reviewers must retrieve answers from memory.
• Mochi
  • Mochi is a web-based Markdown-centric Note-writing system with an integrated Spaced repetition memory system.
• Spaced repetition memory systems make you feel like your memory is worse than it really is
  • One challenge for the adoption and day-to-day experience of a Spaced repetition memory system is that you will spend almost all of your review time on material which you find difficult to remember. Easily-remembered material will rapidly accelerate to intervals of many months, so you’ll spend little time on that, compared to a prompt which is stuck at days-or-weeks-length intervals. Your expected accuracy rate across your entire collection may be 95+%, but your accuracy rate in a given session is likely to be much lower.
• Using spaced repetition systems to see through a piece of mathematics - Michael Nielsen
  • Michael Nielsen explores a method for deepening one’s understanding of a mathematical result, to the point that one becomes so familiar with the pieces that you can almost see the result directly. The method is expressed in terms of writing Spaced repetition memory system prompts, but as he notes near the end of the piece, the SRS format isn’t really necessary to the process. It just provides some pressure to piece apart the ideas into fine-grained atoms. (Note-writing does something similar for me, at a coarser grain: Evergreen notes should be atomic) But the advantage of writing these pieces as SRS prompts, of course, is that one will remember the results.
• SRS retry mechanism
  • “Retry” mechanisms of various kinds are commonly found in a Spaced repetition memory system. Usually the way these work is that if you fail to answer a question, it will be presented again a few minutes later, or at the end of the session. In other words, the item is temporarily assigned a short interval, and after it’s successfully remembered, it gets a new “post-lapse” interval, typically longer.
• Spaced repetition memory systems don’t rapidly demonstrate their benefits
  • Spaced repetition memory systems are extremely efficient, but the curves start shallow. So a new user’s experience with a Spaced repetition memory system is front-loaded in terms of effort… and delayed in terms of powerful results. Initially, it seems no better than a trivial flashcard system which they might have used in school. And the long-term benefit is very non-obvious: Spaced repetition yields exponential returns for small increases in effort, but almost nothing else does, and People have poor intuitions for exponentials.
• Simple spaced repetition algorithms mistreat expected failures
  • Most Spaced repetition memory system algorithms have a target recall rate in mind. Typical values are something like 85%-90%. But this means that in a typical session, if everything’s going exactly right, we should expect to miss 10-15% of questions. And this isn’t (necessarily) because our encoding of those items is poorer than the others: memory is, at least in part, inherently stochastic.
• Spaced repetition memory prompts should encode ideas from multiple angles
  • Flashcards, like those of a Spaced repetition memory system, get a deservedly bad reputation in part because they remind people of rote school learning. People remember being forced to memorize isolated facts like the names of different categories of cloud—and they remember parroting back those answers without building any enduring understanding. Part of the problem is that Educational objectives often subvert themselves: you probably didn’t really care about learning that material. But that issue aside, these isolated exercises are particularly brittle because they develop knowledge which can only be used in one context, disconnected from broader conceptual frameworks and understandings: Rich understanding is about connection.
• Regular spaced repetition memory practice is an onerous habit to adopt
  • So a successful Spaced repetition memory system must deliver significant personal value immediately to overcome this barrier to adoption. Most systems don’t: they’re an empty box meant for the user to fill. But Writing good spaced repetition memory prompts is hard, so most users’ initial review sessions will comprise a small number of mediocre prompts. And Spaced repetition memory systems don’t rapidly demonstrate their benefits. So people churn.
• Memory systems help you focus on deeper engagement by automating away rote memorization
  • Deep understanding requires detailed knowledge of fundamentals. One way to view the role of a Spaced repetition memory system, then, is that they efficiently automate away the essential but rote elements of learning so that you can focus on more interesting, meaningful, or conceptual elements.
• Salience of improvement drives skill development
  • I suspect that this is one key reason why People generally develop skills to a plateau and then stop, and in particular why Athletes and musicians pursue virtuosity in fundamental skills much more rigorously than knowledge workers do. If you’ve never heard of a Spaced repetition memory system, it’s probably not obvious that memory can be made a solved problem. If you’ve never heard of Evergreen notes, it’s probably not obvious that Knowledge workers usually have no specific methods for developing ideas over time.
• Deciding to remember something with a spaced repetition system is a lightweight gesture
  • People who haven’t actually used a Spaced repetition memory system often think of it as a tool you might apply “when you want to memorize something.” But this is an awful way to understand how to use an efficient memory system. Without augmentation, explicitly memorizing information is quite onerous, so it’s not something people often do. It’s reserved for extremely important details, details which are worthy of memorization’s high costs.
• Ye, J., Su, J., & Cao, Y. (2022). A Stochastic Shortest Path Algorithm for Optimizing Spaced Repetition Scheduling. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 4381–4390
  • For me, the interesting central idea in this paper is that they use half-life as a key representation of the state of a Spaced repetition memory system prompt. The trouble, of course, is that you can’t directly measure a prompt’s half-life. You get one binary observation, and you don’t get to repeat it, since the practice event changes the future half-life. The sequence of half-lives for an item are correlated, of course, but—still, tricky.
• Self-grading vs. machine-grading in spaced repetition memory systems
  • In a Spaced repetition memory system, the key input to the control system is: did the user remember the prompt correctly? Some systems are self-graded: the user indicates whether they remembered or not. Others ask the user to input the answer; then the system computes whether the user was correct.
• Spaced repetition memory prompts should connect and relate ideas
  • When using a Spaced repetition memory system to learn facts, the naive approach leads to learning a lot of facts in isolation: the name of a protein, the date of an event, the size of a physical constant. But because Rich understanding is about connection, it’s better to relate these facts to other ideas. This will both help you remember more reliably (due to Elaborative encoding) but also help you develop a richer understanding that goes beyond simple facts.
• Revuu
  • An interactive command-line-based Spaced repetition memory system focused on programming exercises, created by Larry Sanger. Tasks are somewhat larger than in traditional contexts like Supermemo or Anki; you’re asked to write a program which solves a particular problem.
• Execute Program doesn’t have non-executable prompts
  • That lesson couldn’t also have this more typical Spaced repetition memory system prompt:
• Thought Saver
  • Spaced repetition memory system created by Spencer Greenburg and associates. Includes support for widgets so that it functions as a host for the Mnemonic medium.
• Demonstrated retention reliably bounds future recall attempts on Quantum Country
  • Looking at this gives us a powerful way to interpret the goal of “optimizing” a Spaced repetition memory system: after the first few repetitions, once an item has been answered correctly a couple times, you’re just trying to maintain that memory as cheaply as possible.
• Spaced repetition memory prompts should usually focus on one atomic unit
  • If you’ve just learned a new way to cook peas, you might write a prompt for a Spaced repetition memory system like: “Q. How do I cook peas sous vide?” “A. 18m @ 70°C”. But I find that questions like this are usually a struggle—I’ll frequently forget the answer. Questions which draw on multiple independent ideas seem to trip over this problem.
• Mnemosyne
  • A popular open-source Spaced repetition memory system. The primary developer is {Peter Bienstman}, a Belgian photonics scientist.
• How might we situate tools for thought within intrinsically meaningful contexts?
  • What if “practice sessions” for a Spaced repetition memory system could be excised completely, instead structuring one’s authentic environment so that “practice” naturally occurs through daily actions? e.g. Twitter is a kind of programmable attention; Evergreen note maintenance approximates spaced repetition. Related: The dominant culture around spaced repetition memory systems is fixated on meaningless goals.
• Would spaced repetition memory systems perform better with varied question texts?
  • One problem often encountered with a Spaced repetition memory system is that one learns a question’s text too literally and responds to its surface features, rather than really thinking about its content: Spaced repetition memory prompts should be written to discourage shallow “pattern matching”. Maybe it’d be better to arrange for varied forms of the question text to be displayed during review sessions, either created by the author or machine generated (Language models may generate useful question variations for spaced repetition prompts).
• Spaced repetition memory prompts should be concise
  • Related to but distinct from Spaced repetition memory prompts should usually focus on one atomic unit, it’s generally best to keep Spaced repetition memory system prompts concise.
• Anki
  • A popular open-source Spaced repetition memory system.
• Duolingo
  • Duolingo is a language learning app with Spaced repetition memory system components.
• Execute Program’s prompts act both as application prompts and recall prompts
  • The Mnemonic medium’s basic prompt type is an atomic recall prompt in the typical Spaced repetition memory system style: highly atomic, strictly testing your memory. By contrast, Execute Program’s basic prompt type is—or at least looks like—an application prompt (see The mnemonic medium can help readers apply what they’ve learned through simple application prompts).
• Spaced repetition memory prompts should unambiguously produce a specific answer
  • When writing prompts for a Spaced repetition memory system, it’s easy to accidentally write a question which has correct answers besides the one you intend. You must include enough context that reasonable alternative answers are clearly wrong, while not including so much context that you violate concision (Spaced repetition memory prompts should be concise) or encourage pattern matching (Spaced repetition memory prompts should be written to discourage shallow “pattern matching”).
• Chessable MoveTrainer
  • A Spaced repetition memory system designed specifically for learning chess moves. It’s integrated with instructional material, alternating prose + visual explanation with quizzes, akin to how Execute Program’s lessons unfurl their prose in response to reader interaction.
• Simple application prompts can be presented the same way as recall prompts in the mnemonic medium
  • Because the experiences of these question types have so much overlap, solutions to the design problems of memory prompts should mostly apply to the design problems of simple application prompts. This suggests that simple application prompts can be presented with little adaptation in the existing review environment of a Spaced repetition memory system.
• How might spaced repetition memory systems intervene when the student struggles with material?
  • The typical Spaced repetition memory system does two things: it shortens the interval until the next presentation, and (usually) it asks the student to retry the prompt in the same session (SRS retry mechanism).
• Expert response heuristic, after Issa Rice
  • Issa Rice suggests that one useful heuristic for writing prompts for a Spaced repetition memory system (and in particular for the Mnemonic medium) is that someone who’s already an expert in that topic should be able to answer the prompt without referring to a specific mnemonic essay or source.
• Differences in my approach to spaced repetition systems
  • There are already many implementations of a Spaced repetition memory system. Why invest more effort here? Collecting notes on some of the key differences in my approach here to help key myself oriented toward high-order bits.
• To what extent does spaced repetition memory system practice constitute deliberate practice?
  • Do the review sessions of a Spaced repetition memory system constitute Deliberate practice, after Ericsson?
• Spaced repetition mechanics create a sense of effortlessness
  • The core mechanics of a Spaced repetition memory system remove decision-making and gumption from the critical path. Say that you’d like to study the molecular pathway of cell metabolism. Without an SRS, you’d need to make a plan like “I study cell biology on Tuesday evenings,” remember that plan, and summon the will to execute the plan repeatedly. But if you have an active SRS practice, you can throw some prompts into your library and be confident that you’ll see them again over time.
• Avoid orphan spaced repetition memory prompts
  • When adding prompts to a Spaced repetition memory system, it’s usually a mistake to write prompts about a detail that seems interesting, but which is disconnected from everything else you’re thinking about. Spaced repetition memory prompts should connect and relate ideas—and it’s even better to make them connect and relate to ideas that you’re regularly thinking about.
• Spaced repetition memory prompts should ensure reviewers must retrieve answers from memory
  • A Spaced repetition memory system depends on the Testing effect: you’ll remember something longer if you have to actually retrieve it from memory than if you’re simply re-exposed to it. So when writing prompts, make sure that the prompt requires the reader to actually retrieve the memory you want to reinforce.
• Vision statement
  • My current research explores memory systems—environments which build on powerful ideas from cognitive psychology to make durably remembering something into a primitive action. Prior work on memory systems (e.g. Spaced repetition memory system) has focused mostly on explicit memorization of simple facts, like country capitals and key dates. But memory-focused primitives can augment more complex and more meaningful tasks. I’m interested in how they might helpfully suffuse and augment the activities of everyday life—reading, writing, conversation, art. For example, I believe we can create memory systems which make it much easier to learn abstract, conceptual knowledge (Spaced repetition memory systems can be used to develop conceptual understanding). And I believe these systems can accelerate creative work by helping people internalize key insights and influences.
• Existing spaced repetition systems discourage evergreen notes
  • Though notes in a Spaced repetition memory system are atomic in the same way as Evergreen notes (Evergreen notes should be atomic), they’re in many ways too atomized (Traditional spaced repetition memory prompts are atomized). The form discourages incremental synthesis and distillation.
• Traditional spaced repetition memory prompts are atomized
  • The prompts in a Spaced repetition memory system are an unordered, unstructured set. Each prompt is intentionally quite fine-grained and atomic, since that’s what seems to work best for effective memorization. But this lack of structure creates a feeling of wandering through a forest, able to see only one leaf at a time.
• Spaced repetition prompt design is about designing tasks for your future self
  • Writing good spaced repetition memory prompts is hard, but here’s one useful mental model. When you make a prompt for a Spaced repetition memory system, you are giving your future self a recurring task. Prompt design is task design.
• The mnemonic medium’s design relies on light-weight prompts
  • Tiny tasks: Good recall prompts in a Spaced repetition memory system test just one thing, so they can be answered quite quickly. Quantum Country’s median prompt review time is 6 seconds (as of 2019/12).
• Recall rates are a misleading proxy for more meaningful goals of the mnemonic medium
  • The existing Spaced repetition memory system culture’s fixation on often-meaningless memorization makes this failure mode particularly common: The dominant culture around spaced repetition memory systems is fixated on meaningless goals.
• Conceptual information may have much slower optimal spaced repetition schedules
  • Almost all the prior literature on the Spacing effect and on naturalistic applications of a Spaced repetition memory system concern declarative information like word pairs, term definitions, or simple property values. I suspect that conceptual information, densely linked to prior and related knowledge, will exhibit very different memory dynamics.
• Self-graded spaced repetition memory systems are more efficient than machine-graded systems
  • Interaction is a cost center in interface design, so a machine-graded Spaced repetition memory system impose greater costs on their users. If the user is meant to type in the answer, that can easily 2-4x the amount of time spent on each task.
• Forgotten spaced repetition prompts should probably have smaller future spacing intervals
  • My naive implementation of a Spaced repetition memory system scheduling algorithm in Quantum Country and Orbit has a constant geometric expansion factor for intervals. I thought: well… maybe that’s fine for most conceptual material.
• Salience prompts
  • You can sort of implement these in a Spaced repetition memory system, but the scheduling is probably all wrong.
• Cloze deletion
  • I typically run into this concept in the context of Spaced repetition memory system, which have somewhat distorted the term.
• Missed answers in Execute Program’s reviews don’t penalize the student unless they “give up”
  • Execute Program’s Spaced repetition memory system prompts are machine-graded, so if readers make semantically-unimportant typos in their responses, the system marks the prompt wrong. These false negatives can be quite annoying, particularly since Execute Program’s lessons don’t unlock until you’ve successfully reviewed their prerequisites. In the SQL course, the false negative rate is high enough that in early 2020, the system was changed so that incorrect responses no longer “count” as failing, so long as the student eventually produces a correct answer. Readers can try as many times as they want, and the next review’s interval will still increase according to the SRS curve. Subsequent lessons unlock irrespective of user performance.
• Avoid yes-no spaced repetition memory prompts
  • Yes/no Spaced repetition memory system prompts are an attractive nuisance. Many simple statements you might read are basically saying “X is true.” An apparently easy way to turn that into a question is to ask “Is X true?” But these questions aren’t very useful: Spaced repetition memory prompts should ensure reviewers must retrieve answers from memory
• Retry outcomes may give extra retrievability signal
  • A big problem for models and evaluation in a Spaced repetition memory system is that you’re trying to estimate a continuous value (probability of recall, or retrievability per Two-component model of memory) over time, but all you get are these sparse one-bit binomial samples: remembered, or didn’t remember. If a reader failed to remember, their retrievability might be 10%, or it might be 90%.
• Kinnu
  • the queue consists of small chunks of content interspersed with two-choice Spaced repetition memory system prompts
• Cloze deletions can be used to create typical N-sided flashcards
  • A typical Spaced repetition memory system represents a Cloze deletion card differently from front/back flashcards, but it’s interesting to observe that you can think of typical flashcards as a special case of cloze deletions.
• How does the mnemonic medium’s efficacy vary across platform-knowledge-style topics?
  • There’s ample experimental data that the Spacing effect applies across many topics, and there’s anecdotal data that a Spaced repetition memory system can effectively encode material across many topics. We may need domain-specific problem-solving to establish how to encode each topic’s knowledge into prompts.
• Self-graded spaced repetition memory systems avoid false negatives
  • In a Spaced repetition memory system review session, if I make a typo in my SQL expression because I’ve misremembered the name of a function, that’s bad and I should probably review the task again. But if I make a typo because my finger slipped, I’m still “right” as frat I’m concerned. False negatives are extremely irritating in machine-graded systems. They’re also inefficient, of course, since the prompt will re-appear sooner. But enraging the user is the much more serious issue.
• IA vs AI
  • Spaced repetition memory system
• Eglington, L. G., & Pavlik Jr, P. I. (2020). Optimizing practice scheduling requires quantitative tracking of individual item performance. Npj Science of Learning, 5(1), 15
  • The authors point out in this paper that the literature on Spaced repetition memory system has mostly focused on finding the schedule regime which produces the highest recall rates, irrespective of time costs. The Testing effect is powerful; more practice will produce more stability. So how should one think about these trade-offs?
• Commercially-successful spaced repetition memory systems are all machine-graded
  • Anki, Mnemosyne, and SuperMemo are self-graded. Duolingo, Memrise, and Quizlet (for the most part) are machine-graded. The Spaced repetition memory system products with widespread adoption and significant commercial success are all machine-graded.
• Machine-graded spaced repetition memory systems can perform more advanced knowledge modeling
  • If Duolingo asks me to translate a Spanish phrase into English, and I mistranslate a single word, it can notice which word I mistranslated, and adjust the schedule for related tasks accordingly. A self-graded Spaced repetition memory system can only know that the entire response was wrong. (Note that I don’t think Duolingo actually does this!)
• Self-graded spaced repetition memory systems accommodate a wider variety of prompts
  • “Improvise a melody in pentatonic minor.” Many valuable Spaced repetition memory system prompts are, to various degrees, fundamentally non-machine-gradable—particularly those described in Spaced repetition memory systems can be used to prompt application, synthesis, and creation. Self-graded systems will accommodate these more experimental questions more readily.
• How to make memory systems widespread? - Michael Nielsen
  • Article by Michael Nielsen on how to make Spaced repetition memory system and similar systems an important practice for many people.
• Funnels, tubes, and fields, after Joe Edelman
  • It’s interesting to notice that my ideas around Spaced repetition memory system function in all three of these modes.
• How to reinforce insights from problem solving?
  • If you’re studying a textbook like University Physics - Young and Freedman, we have some idea of how to encode the conceptual material’s explication into Spaced repetition memory system prompts. So you’ll know what flux is, conceptually, and you won’t forget Gauss’s law. But it’s not enough to merely read the explication and practicing questions about its contents. You really do need to work through the examples and the problems. Lots of insights and understandings emerge in that context which don’t emerge when you’re just reading.
• 2022-01-27
  • I had a lovely chat with Giacomo Randazzo yesterday about modeling and scheduling in Spaced repetition memory system. A few takeaways for me:
• 2020-08-25
  • 8:18: starting to document barriers to adoption in Spaced repetition memory system, screwing around with violin plots
• Augmented reality and spaced repetition
  • How might Augmented reality be combined with a Spaced repetition memory system, either for ingestion or for review? Or perhaps it implies some other phase / modality?
• Brainscape
  • A commercial, web-based Spaced repetition memory system along the lines of Quizlet. No significant new ideas as far as I can tell. Heavily focused on shared decks. Seems pretty unserious about the science, or maybe that’s just a distortion created by their marketing team.
• [[Kang, S. H. K., McDermott, K. B., & Roediger, H. L. (2007). Test format and corrective feedback modify the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19(4–5), 528–558.

https://doi.org/10.1080/09541440601056620]]
* I’ll summarize just the second experiment, since it’s more relevant to the kind of Spaced repetition memory system design I’m doing. Students who practiced via short-answer enhanced their recall the most relative to a control condition and a condition in which students simply read the correct answers. Short-answer-practicing students performed better than those who practiced via multiple-choice on a multiple choice post-test (d=0.41) but not significantly better than MC-practicers on a short-answer post-test.

• [[Karpicke, J. D., & Smith, M. A. (2012). Separate mnemonic effects of retrieval practice and elaborative encoding. Journal of Memory and Language, 67(1), 17–29.

https://doi.org/10.1016/j.jml.2012.02.004]]
* Is the Testing effect produced by Elaborative encoding? When you review prompts in a Spaced repetition memory system, you give yourself an opportunity to form new connections based on your current context and the different thoughts you have during the review session.

• Librarium
  • VR-based study environment with flashcard-based Spaced repetition memory system; environmental Memory palace scaffold; crowdsourced decks; Kaplan-provided learning content. Focus on SAT, LSAT, MCAT. Product workflow seems a bit diffuse, but haven’t tried it myself yet. Seems focused on method of loci rather than spaced repetition. Web site is very… non-product-company-y.
• Marr’s levels of analysis
  • It’s interesting to analogize these as lenses for thinking about Tools for thought. In my study of a Spaced repetition memory system, for example:
• Mozer, M. C., Pashler, H., Cepeda, N., Lindsey, R., & Vul, E. (2009). Predicting the Optimal Spacing of Study: A Multiscale Context Model of Memory. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, & A. Culotta (Eds.), Advances in Neural Information Processing Systems 22 (pp. 1321–1329).
  • This paper proposes a model which can fit the spacing function (i.e. of the Spacing effect) and the Forgetting curve of empirical learning data (e.g. from a Spaced repetition memory system).
• [[Nioche, A., Murena, P.-A., de la Torre-Ortiz, C., & Oulasvirta, A. (2021). Improving Artificial Teachers by Considering How People Learn and Forget. 26th International Conference on Intelligent User Interfaces, 445–453.

https://doi.org/10.1145/3397481.3450696]]
* This paper attempts to produce a better scheduler for a Spaced repetition memory system by combining a model-based (per-student, per-item) estimator of student memory state, with a more holistic planning model which attempts to take students’ time constraints and test date into account.

• Podsie
  • A Spaced repetition memory system intended for integration into classroom contexts. Its items are drawn from assignments and quizzes created by the teacher. In private beta as of 2021-03-17.
• Questions by Julia Evans
  • Even though the form factor is roughly flashcards, it’s not trying to be a Spaced repetition memory system. It’s more an unusual interactive article. I do find that it runs very much afoul of Interaction is a cost center in interface design: I wonder if it would be possible to create a more fluid interaction, perhaps inspired in part by the rapid interactions in Execute Program (see Execute Program’s lessons unfurl their prose in response to reader interaction).
• Readlang
  • Language-learning-focused Spaced repetition memory system meant for ingesting vocabulary naturalistically as you browse. Its core mechanism is that you can highlight some foreign language text, see a translation, then add that text and its translation as an SRS prompt.
• Taylor, W. L. (1953). “Cloze Procedure”: A New Tool for Measuring Readability. Journalism Quarterly, 30(4), 415–433
  • Having read this paper, I see that the term “cloze deletion” is abused as used in Spaced repetition memory system: as described in this paper, it’s meant to be a random deletion from a passage.