Private copy; not to be shared publicly; part of Patron letters on memory system experiments
Early, rough thinking, but perhaps useful for others interested in the design of learning environments. Assumes detailed familiarity with memory systems.
When everything goes right, my memory practice totally transforms the experience of diving into a new topic. I’ll feel like I can keep turning a crank, and my understanding will just ratchet up and up, durably and inevitably. But quite a lot of the time, everything doesn’t go right. I’ll find that I’ve ended up with brittle, parochial understanding of a new topic. Review questions will strike me as boring or alienating; I’ll feel like I’m parroting, rather than really understanding.
One way to address this problem is to improve as a practitioner of memory systems. I grow my armory of prompt-writing strategies; I monitor my emotional connection more attentively; I intervene more aggressively when something’s not working. Over-simplifying, the quality of my memory practice depends on the quality of my prompts, and on the processes I’ve put into place for monitoring and revising them over time. If I’ve produced a brittle understanding, that means I need better prompts and prompt-writing processes.
Prompt-writing ability is a useful framing. I’ve improved enormously at writing and maintaining prompts over the years; my memory practice is much more versatile and reliable as a result. Yet it’s also worth thinking about the limitations of this framing. Aspirationally, I’d like a memory practice which not only helps me build and maintain a flexible, reliable understanding, but which actually deepens my understanding over time.
I’ve been wondering: to what extent is achieving that goal about learning to write better prompts? Are individual prompts the best primitives for that task? Generalized flashcards are a convenient representation for declarative knowledge, but I’d like to help people learn complex conceptual topics. If we step back and look at what’s actually necessary to produce that sort of understanding, and at the mechanisms of practice, could we find a better-matched primitive?
This exploration is motivated in part by my experiences with Alex, a motivated adult student whom I helped study physics earlier this year1. Among other experiments, I wrote hundreds of memory prompts for Alex to help him internalize his readings and tutoring sessions. I learned a great deal by watching him grapple with these concepts both inside and outside his memory system. Being an observer, rather than a learner, really helped: when I’m writing and reviewing my own prompts, I often find it tough to create reflective distance in the moment, to see what’s going on and what’s breaking.
One key issue I saw in Alex’s experience was trouble with transfer. Often, he learned to answer the prompts I’d written, but his understanding was brittle: he struggled to draw on what he’d learned in slightly different circumstances. In my experiments, it seemed that what Alex really needed was much more variation in his practice with each idea.
For some concepts, I tried the standard memory system practice, writing a dozen (or more!) prompts about this one idea, accessing it from many different angles and with many different surface features. This may have produced more flexible understanding, but the impact on review sessions was quite negative: even when prompts are shuffled, it’s obnoxious to answer many variations of the same question in a single review session. Because these variations are so burdensome, I didn’t write them for as many concepts as I probably should have. And from a cognitive perspective, retrieval will be much less effortful after the first question, so the subsequent questions will probably not produce as much reinforcement as they could.
Emotionally and cognitively, I believe it would be much better to “smear” these variations out across many review sessions. Conceptually, I thought of myself as writing many variations on the same Platonic idea, using various strategies to solicit that idea in different ways. These prompts were a single thing in my mind—a “prompt cluster”—but represented as an unrelated set in the memory system, despite the high cognitive overlap.
A related phenomenon: we saw a lot of pattern matching in Alex’s review experience. His comments would often include the telling phrase “this one”—i.e. “Oh, I know this one has a trick where…”. This is probably a sign of trouble for transfer, as we’ve discussed, but it’s also a marker of emotional friction. It feels bad to parrot. It makes you (rightly!) question whether your practice is a good use of time. It can create a feeling of rote drudgery. It seems to make Alex (and me) less likely to pay close attention, not only to that question but sometimes to the ones which follow. I believe that pattern matching creates emotional distance, a dulling of involved participation.
So it’s interesting to consider the design goal: how might we make pattern matching never occur? What if an idea is never accessed in exactly the same way twice? In fact, I have a strong instinct that this is a terrible idea as written. There’s something very powerful about the reflexive stimulus–response pattern that highly entrained memory practice establishes. And the feeling of flow in practice sessions is delicate: you don’t want rote boredom, but you also don’t want meaningless make-work. Still, I’d like to experience these tradeoffs more viscerally.
Alex also needed variation in scope of the prompts over time. Immediately after reading an explanation, he most needed prompts which acted like a simple reading comprehension check—in many cases, he hadn’t actually understood or absorbed some of the important statements on the page. Then he needed help building a durable memory for key details: terms, notation, equations, relationships, etc. But that’s not enough to actually understand the topic. We’d meet to discuss the concepts and to solve problems together, and I’d write more prompts for him each time, often about progressively higher-level details.
Once the fundamentals were in place, what he really needed was problem-solving practice. The problems were tough for him, so they needed to start quite simple and become more complex. Intermittently we’d retrace some worked examples in great detail. After days of problem-solving practice in one chapter, he’d want to move onto the next chapter to keep motivated, but we needed to ensure that he’d swing back to earlier chapters and continue to solve problems about their ideas over time. At this point, the “reading comprehension” prompts and some of the basic prompts on notation felt burdensome to review, but more complex declarative prompts, like those on equations, still seemed necessary to keep those details fresh.
Eventually, Alex and I got to the point where we were connecting the material we were learning with new research papers he found interesting. It’s conceivable to me that review sessions could be systematically orchestrated to include that sort of activity: he showed me the papers he wanted to understand, and I said things like “oh, once you’ve got some experience with electric flux, we’ll be able to make sense of this figure together.”
Alex’s study sessions needed to consist of different activities at different stages of his learning process. He needed more complex practice activities to really understand the material, but those couldn’t be introduced immediately. Conversely, trivial “bootstrapping” tasks could feel obnoxious when he’s just finished an activity which uses those same ideas in much more complex ways. If we want a memory practice to deepen our understanding over time, that will likely mean shifting the distribution of tasks over time to create progressively more complexity, elaboration, and creative opportunity.
In summary, my experience with Alex suggests that to produce fluid understanding, one would benefit from much more fluidity in review. A straightforward flashcard works if you want to remember the value of the electric constant. But to internalize complex conceptual matter, you want to be pushed to draw on each idea from different angles, with different wording, in different modalities, alongside different combinations of other ideas, at different levels of complexity over time.
Part of my hazy thesis here is that much of the potential of a memory practice is not about memory, per-se: it’s about the daily sessions. A steady Anki user has designated a ~5-15 minute slice in their day for systematically improving their knowledge and understanding. That’s a powerful vessel. It can be filled with rote recall flashcards, but more experienced memory system users discover that prompts can be used to stimulate a wider range of thought.
My contention is that if you’re trying to learn a complex conceptual topic, we should fill that daily vessel with whatever activities would most effectively and enjoyably produce the understanding we desire. If the material is new, it might be best to reinforce your memory of key details, and ask you to predict the next steps of some partially-worked examples. In the next session, you might be asked to solve some simple problems; in the next session, some different, more complex ones. One week later, you might be offered an alternative derivation of a theorem you’d studied, and some questions relating the two approaches. Perhaps it would be helpful to ask you to synthesize an explanation, or to brainstorm a list of questions you have about the concept. Each session’s contents are completely different, unless some activity seems particularly desirable to repeat verbatim.
In some sense, I’m arguing that “memory systems” can be thought of as “practice systems”. You can’t quite orchestrate what I’ve described for yourself with today’s tools. But experienced memory system users have internalized many strategies for filling their practice vessels—that is, for digesting ideas into interesting and effective practice tasks. These strategies are a reflection of several more fundamental (generally implicit) understandings:
As Michael Nielsen has pointed out, it’s tough to directly communicate these strategies or these understandings. But I suspect there is a kind of “pattern language” hiding within these strategies we develop. I've now written memory system prompts for quite a lot of technical material; it would be interesting to work through those prompts and their associated context, looking for "vocabulary" and "grammar".
The flashcard primitive captures declarative knowledge quite directly. If you’re studying Italian and want to remember the word “carciofo”, you can write it on a flashcard and write “artichoke” on the back. If you want to learn anatomy, your flashcard can be an arrow pointing to a part of an illustration. You can learn lead’s atomic number by writing “lead’s atomic number” on a flashcard and “82” on the back. Context makes the task obvious; in each case, the implied task is to answer: “What is this?”
But when you’re learning about the relationship between electric potential and electric potential energy, you can’t put that concept directly into a memory system. It’s multifaceted. There are lots of tasks you should practice to understand that relationship. You should make sure you know the symbolic relationship, sure. But you should also be able to apply that relationship in various scenarios; you should be able to see when it makes sense to use one of these concepts rather than the other; you should see the parallel between this relationship and the relationship between electric field strength and electric force. You’ll want to form new connections to this concept over time, as you learn more. For example, once you’ve been introduced to capacitance, you’ll be able to work with a new facet of this relationship.
One way to model all this is as a large collection of primitive prompts, each reinforcing a different “atom” of knowledge related to that concept. But as we’ve discussed, they’re all so highly related that it would be better to smear them over many sessions. Feedback on their scheduling should be at least somewhat linked. You’d want many of the task details to vary with each presentation. And some of these tasks should be added later, once you’ve digested the earlier ones or learned more.
Practically speaking, when I write prompts for a concept like this, the way I’m thinking about it is that there’s a single thing—the concept—and I’m turning it in the light to see it from many angles, to see how it interacts with other objects in the space. My implicit prompt-writing “pattern language” suggests a family of tasks, and as I start pulling on those threads, more patterns suggest themselves. Sometimes it’ll feel like I’m working with the focal concept and another adjacent concept at once; or I’ll feel like I’m zooming in on some facet of the concept so far that the facet itself becomes the object of my attention. But the concept itself retains a sort of integrity in my mind. For the most part, mentally, I’m “putting that concept” into my memory system, in richer and richer ways. The concept is the primitive noun, and the verb (“putting it into my memory system”) is a messy, complex thing which depends on my pattern language and my prior knowledge.
I can put the word “carciofo” directly into my memory system. There’s almost no distance between the object itself (“carciofo = artichoke”) and its representation on a flashcard (“Q: carciofo? A: artichoke”). That flashcard basically is the declarative knowledge atom. That’s its strength as a primitive. Memory systems were developed for learning declarative knowledge, so maybe we shouldn’t be surprised that their core primitive has a natural representational unity with that kind of information. But now we’re trying to stretch memory systems to work well for internalizing complex concepts, too. Reframing “memory systems” as “practice systems” for a moment, is there some other kind of primitive waiting to be created—one which would let me “add the concept” as an elementary operation, in the same way I can “add a vocabulary word” to one of today’s memory systems?
In the past few months, lots of people have been trying to use large language models to automate the creation of memory system prompts from explanatory text. This might be close to tractable now (1, 2), for simple declarative knowledge. But I’m more interested in memory systems for their potential to help me learn complex conceptual material, and in deeply internalizing ideas that are relevant to my creative work. This is a very different task, not obviously amenable to the same kind of direct automation. The language model doesn’t seem to have the pattern language we’ve been discussing—in part because it’s still something being actively discovered at the frontier of the memory system practitioner community.2
But if I could externalize that pattern language into something the model understands, maybe we could produce a memory system in which ideas are a primitive, alongside concrete tasks. Here, I’m using the word “idea” very broadly to cover a multifaceted-but-distinctly-coherent element—like the relationship between electric potential energy and electric potential. In such a system, perhaps you could “add a concept” directly, optionally with a comment about the nature of your interest, and the system would use the pattern language to create novel practice activities on the fly in each session.
What does it mean to “add a concept”? What specifically would you be adding? One natural route would be to add a passage from a book, perhaps with some markup designating the central idea within its context. In this world, the memory system library would consist not (or not just) of static flashcards, but rather of a set of marked-up references to texts or personal notes. This would also mean that if a task gave you trouble, you could easily navigate to the writings which inspired it.
This conceptual design isn’t “use a language model to generate prompts, then add them to a library”; it’s “add ideas to a library, then use a model to generate activities dynamically.” Those activities would vary in each session to help you encode a fluid grasp of the concept. They can become more complex to help deepen your understanding over time—for instance, using the pattern language to combine multiple ideas. If the pattern language is concrete enough that it can be reified in the interface, we can give different kinds of activities different visual identities (e.g. problem-solving practice, visualization, generating examples, explanation synthesis) and allow users to quickly flip between alternative activities for a given idea.
My claim here is that much more fluid practice will produce much more fluid understanding. But idea-as-primitive is appealing from an interaction design perspective, too3. My concrete emotional experience as a memory system user is: I hear or read or think something that excites me. I think, “ah! I want to bring this into my practice!” But I can’t quite bring “this” into my memory practice; I have to find a way to transform it into some other object which I can bring into my practice. Sometimes, that transformation task is a welcome exercise. It brings me closer to the idea that stimulated me, helps me pick it apart and understand it better. But much of the time, it just feels like a burden. I want to emphasize that this impulse is not about efficiency or “lowering the floor.” It’s about making the core verb feel better—making it more naturally aligned with my internal intent and my emotional interest. I think this explains the attraction of cloze deletions: they’re closer to “adding the thing itself.” The trouble is that they don’t really work for complex material, at least not when used so directly.
Language models are shiny, but I don’t think that the best way to explore the ideas I’ve articulated here is through a leap to systematization and automation. I can experiment with “practice systems”, variation, problem-solving, escalation, and “pattern languages” manually, working with another student or perhaps with my own studies. Intimacy and rapid experimentation still feel like the right properties to prioritize for now.
One serious reservation I have about these ideas is: by expanding the scope of memory systems to include such a wide range of practice and learning activities, I dilute the specific powerful idea that they represent (efficient, systematic retrieval practice using a simple primitive). It might be better to let memory systems remain tightly scoped, better to play to their natural strengths. What I’ve been describing is instead closer to an intelligent tutoring system. I have a lot of reservations about those systems, but the last time I grappled with that body of research was years ago when I was at Khan Academy. If I continue down this path I expect I’ll need to return to that literature with fresh eyes, and to draw some clearer delineation between my ideas and those of that field.
1 Alex’s professional life has been undergoing some significant upheaval, so his studies have been on pause recently. We may resume in time, and/or I may start working with another student.
2 This relates to a gripe I have about the recent gush of “AI tutors”: they lack a clear theory of instruction. Some of them are told to quiz you with boring problems; others seem simply told to “be a world-class tutor and answer the student’s questions.” What does it mean to be a world-class tutor? That’s hard enough to answer; it seems clear the model doesn’t have a strong opinion. What does it mean in this person’s context, specifically? A better system needs to describe a theory of instruction—what is supposed to happen in the course of a session, what does that mean cognitively, and how should the tutor bring that about, in interaction with the student’s behavior? If there were millions of tokens in the training set with good answers to those questions, the model might not need more steering. But like the memory system pattern language, this sort of knowledge is mostly tacit; where it is written out, accounts differ so widely as to offer little guidance.
3 There’s an extended discussion of “ideas as primitives” in last October’s letter (“Lessons from summer 2022’s mnemonic medium prototype”), mostly from an interaction design perspective.