Private copy; not to be shared publicly; part of Patron letters on memory system experiments
There’s a funny downside to doing all this research on memory and comprehension—a sort of loss of innocence. I’ve viscerally internalized just how poorly I’ll understand and retain complex ideas when I read in my default “casual” gear. And I’ve learned something about methods which produce deeper and sturdier understandings: memory systems, active reading practices, elaborative notes, and so on.
But now I have two new problems. The first is that part of me feels I “should” be reading in those slower gears all the time, or at least much more of the time. Casual non-fiction reading now carries a tinge of guilt. I’m keenly aware of how little I’m really internalizing.
The second problem is a new sense of finitude and scarcity. It takes a long time to study a text properly! In this mode, a page I might have casually read in a minute now consumes ten minutes—or twenty, or an hour. I’ll spend tens of hours carefully studying one monograph, but my reading list is miles long. At this rate, I can only read a tiny fraction of it. What if a critical insight is hiding halfway down the stack?
How should I contend with these problems? How should I think about the tradeoffs between breadth and depth—or quantity and quality—in my reading? My observations will be necessarily personal, focused on my own proclivities and priorities. But people often write me with similar questions, and I hope this will help.
I want to begin by observing that there’s something very wrong with the way these discussions often go: they frame the issue as a simple optimization problem. “How can I read as many books as possible?” or “I have ten hours per week of study time; how can I learn as much as possible to a satisficing level of depth?”
If taken too seriously, this stance is corrosive to the kinds of authentic curiosity and creative engagement that produce good work, good insight, and—for me—a good life.
If I’m reading about ideas I found personally meaningful, and the explanation is captivating, I want to wallow in the text. I want to figure out how to go deeper, not how to “get through it” faster.
The real trouble is that much of my non-fiction reading is in a category I’d label “merely useful”. A given paper might not be worthy of devotional attention, but it does help, in some way, to accomplish my goals. We might improve upon the optimization framing by asking something like: “How to find the most beautiful sources of insight for a given aim? Is there a path which only traverses texts worth wallowing in?”
One problem with this new framing is that such a path often doesn’t exist. If I’m trying to reach the edge of human knowledge about a given question, I’ll find that many key ideas are buried haphazardly within piles of otherwise prosaic sources. Of course, there are enough beautiful texts about beautiful ideas to occupy a lifetime, but if I were to read exclusively from that list, learning a little about a thousand topics, I would be sacrificing my personal creative interests.
Another problem is that I’m often not quite sure what my aim is—I’m exploring, following my nose, looking for ideas that resonate with an inchoate interest. I’m not ready to wallow in anything yet. Sometimes that’s because I’m reading in off hours, when I have the energy to sniff around but not for careful study.
Both of these problems require making trade-offs in favor of breadth. Now that I’ve grounded the concerns in meaning, I’m more comfortable exploring how we might “optimize”. If I’ve exhausted the most compelling sources for my present aims, or if I’m not sure what my aim is, or if I’m reading with low energy, how should I orchestrate my reading to support my creative practice and my search for beauty?
When I dove into the reading comprehension intervention literature, I didn’t know enough about the space to know what I wanted to read carefully. For the first week or two, I was just sniffing around, my questions too vague to articulate well. In this time, I certainly wasn’t studying deeply. I was sifting through dozens of papers, making (mostly mental) notes of key words, people, and concepts. And I was paying attention to my internal reactions (“Boring!”; “Ooh…”; “Dubious.”; etc) to help me get a clearer sense of what I really wanted to know.
I mostly prioritized breadth during this time. I was honest with myself: I knew I wasn’t absorbing much. I wrote some notes and memory prompts here and there, almost as signposts, but my goal was to begin to roughly chart the space. I wanted to know what the main “territories” are, how they relate to each other, and where the main ideas might be found within those regions. Eventually, some questions cohered, though they were still quite broad:
I used these questions to structure a somewhat slower read of a smaller pile of papers and books which had emerged as “canonical” during my initial sweep. This pass helped me refine my questions and my reading list. For example, I found myself asking much more specific questions about methods for distinguishing reading comprehension gaps from issues of memory encoding and retrieval.
As my questions got sharper, and as I triangulated a better bibliography, I started reading much more carefully. I ensured that I could fully reconstruct the procedures of key experiments. I took care to understand the details of theoretical frameworks well enough to apply them to my own questions.
The full “dynamic range” of my speed across this exercise was quite wide. Initially, I’d just read abstracts—maybe a minute per paper. After a few weeks, I’d spend an hour or two on a paper. And after a month, I’d spend a day or two deeply studying a particularly good paper.
In my introduction, I mentioned two problems: I often feel like I “should” be reading in slower gears all the time; when reading slowly, I feel anxious about how little ground I can cover. My literature review example illustrates one way I think about resolving these problems. When I’m not sure what my aim is, I don’t need to feel like I should be reading in a slower gear. In fact, in this situation, I should mostly be reading quite shallowly, trying to build an index and to inform my sense of what matters. I can cover lots of ground in this mode, and I can cheaply triangulate a smaller pile of higher-quality works for more careful study. It’s also a mode I can use constructively when I’m low on energy. Critically, though, I don’t delude myself into thinking that I’m learning much at the object level. Once I have a clearer sense of my aims, my heightened emotional connection to those sharpened questions makes it easy to set aside quality time for slow, careful reading. Then, when I’ve exhausted the most compelling sources, I’m best served by speeding up again, grazing over wider swaths until I find ideas worth lingering over again.
When I started using memory systems, I felt a chronic optimizer’s anxieties: which material should I add to my memory system, given the cost of review? In how much detail? Happily, I don’t worry much about this anymore. Now I view review time as more or less “free”. I’ll illustrate with a few examples.
The first essay of Quantum Country is about 25,000 words long and contains 112 prompts. Let’s call it one for every 225 words. A typical reader will take about four hours to read that essay. With the current schedule, they’ll spend about an hour reviewing those prompts in the first year, and a small fraction of that time thereafter.
I recently wrote prompts covering the first section of Jim Hefferon’s Linear Algebra textbook. This section is about 2,500 words long, and I wrote 28 prompts, roughly one every 90 words. My test readers take about 45 minutes to read this section. I don’t have long-term data on these prompts’ review time, but my own data from similar texts would suggest they’ll spend about 15 minutes reviewing in the first year.
Taken together, these examples suggest that memory practice adds an overhead of a quarter to a third of the original reading time. That’s really not enough for me to be worried about. My original optimizer anxieties were thinking about these review costs in terms of doubling—or quadrupling!—the effective time. That’s simply not the case. If I find some material striking or possibly useful, that extra quarter or third is well worth paying in almost all cases.
By comparison, the problem set associated with that linear algebra section took me about 2.5 hours to complete: ten times the “overhead” of the memory practice. How many problems should I do, if they’re provided? If the problems are about ideas I really want to understand, they’re precious gifts; my rough policy is to continue as long as I find myself engaged or surprised in the course of answering them.
If I feel that I’d rather cover 25% more ground than reliably internalize whatever details I find important, then one of these things is usually true:
In fact, review costs are effectively much lower than we’ve discussed, because review time is rarely rival with deep reading time. I review in fragmentary time: waiting in line, on a bus, between appointments, and so on. I wouldn’t be reading seriously in that time, anyway. For me, review time is more directly rival with, say, Twitter time.
So I don’t worry about the time costs of review anymore. But two variants of this concern do remain relevant. The first is that while reviewing is cheap, writing good prompts can easily double or triple my reading time. Now, that’s a misleading observation—much of what’s happening is that the prompt-writing process forces me to a deeper level of comprehension, and to make connections and elaborations which wouldn’t have occurred without the prompt-writing. Assuming the material is meaningful, that’s all time very well spent. But a big chunk of the time is fairly mechanical, and this cost is much more impactful than the cost of review. (More on this later.)
Another much more salient version of the “time cost of review” concern is the emotional cost of review. For example, when I’m first engaging with a topic, I often have a poor sense of what’s important, and of how much reinforcement will be necessary. A few months later, I’ll often run into a glut of prompts written during that early naïveté, and I’ll find myself bored and emotionally disconnected, wanting to do something else. That’s certainly not how I want review sessions to feel. I’m not worried about the cost of reviewing a given prompt in terms of time; I worry much more in terms of the damage which unsuitable prompts can do to my relationship with review.
Educational technology designers often talk in terms of shifting a Pareto frontier in learning. That is, if you have some fixed amount of time, and you use the best methods available, you can learn a given topic to some level of depth. But if we develop new methods, we can expand that limit, so that you’ll be able to learn to a deeper level in the same amount of time. Conversely, if you want to learn some material to a fixed level, new technology may lower the time required for the best-available route.
I think this framing can often be improved by replacing the time axis with “gumption” or “energy” or “enthusiasm.”
Relative to studying with traditional flashcards, an algorithmically scheduled spaced repetition system lets me learn more material more deeply in a given amount of time. But I don’t think time is the most important factor limiting my use of traditional flashcards. The more important issue is that traditional flashcards rapidly drain my enthusiasm. After a few sessions, I’ll know most of the cards quite well, and I won’t need to review them. But the format still requires me to flip through the entire deck, answering every card, in order to ensure that I reinforce the few which give me trouble. The exponentially expanding schedule of spaced repetition systems reduces (but does not eliminate) this gumption tax. And because it focuses review on the prompts I’m more likely to miss, I feel the value of review more keenly, which generates gumption. These dynamics make me willing to use spaced repetition systems in more situations than I’d use traditional flashcards.
I listen to most podcasts and talks at 2-3x speed. This isn’t primarily because I want to “get through them” faster. It’s because at slower speeds, these media usually aren’t insight-dense enough to hold my attention, energetically. At 1x, my attention will often wander off, or I’ll find that I want to listen to something else. The speed control is a technology which shifts my gumption frontier: it lets me get more out of these media at a given level of enthusiasm.
Personalized learning systems like Khan Academy aim to optimize learning by algorithmically ranking exercises so that each student works on the best-available task at a given time. This is usually framed in terms of learning more per unit time. But again, I think the more important issue is gumption. Nothing saps enthusiasm like a long string of problems you can already solve fluently. You may also quickly lose gumption if you’re asked to tackle problems significantly beyond your level of development. Now, when we reframe the issue in terms of gumption, rather than time, we can also see that a good solution will require more than estimating the probability of a successful answer. It makes us ask how we might orchestrate activities which are meaningful and rewarding for the student.
My enthusiasm for the mnemonic medium and for my recent highlighter-based memory system proposal draws on a similar logic. Yes, it takes time and expertise to write good memory prompts. But in many cases, the more decisive bottleneck is gumption. Comprehensive prompt-writing often feels disruptive and draining; in many situations, I don’t write nearly as many prompts as I would care to review. I’d love to shift that Pareto frontier—to reach a given level of depth with a lower tax on my gumption, or to increase my depth of study for a given level of enthusiasm.
I grew up as a programmer. And, like most programmers, I absolutely loved “learning by doing”. Rather than reading a reference manual on a new language, I’d just dive in, try to implement something in it, and learn just-in-time when I encountered issues. This method was great for me in a number of ways. It gave me rapid feedback loops and rapid rewards. It kept my learning activities more closely connected to my actual goal (writing programs). When I did engage in “book learning”, I’d be especially motivated because I’d be doing so in response to some specific issue I’d encountered. And all the hands-on effort helped cognitively, too: to build fluency in a practical skill like programming, we must induce and learn patterns across many experiences.
What kinds of material can be learned in this way? What kinds of problems arise in this style of learning? I don’t yet have good answers to these questions. For now, I’ll share a few sketchy observations.
When I was a teenager, I’d been programming in C for years, but I often encountered inscrutable crashes due to “accessing bad memory”. I spent hours staring at my code, changing things semi-randomly, trying to track down the bugs. None of it made sense. It turned out that this was because I simply didn’t understand pointers and memory allocation on a conceptual level, at all. I was cargo-culting pointers, copying the relevant lines from other programs without any idea of what they were doing. No amount of “just trying to build programs” fixed this problem. I needed to sit down and actually learn the relevant conceptual material, which was covered in multiple books which I owned but hadn’t bother to read. Unfortunately, at this point, I’d built bad habits around reading: I’d developed a dependence on learning by doing, and I didn’t have the attention span to study difficult conceptual material. It took me years to fully fill these holes. And then the same problems repeated soon thereafter when I “learned” OpenGL by copying and building upon code from tutorials. I made some awfully elaborate games despite my enormous conceptual gaps. But more and more often, I hit impenetrable walls. Then in university, I implemented various machine learning systems from scratch but didn’t understand how they worked. Again, I soon found myself stuck, and couldn’t exactly diagnose why.
The moral of this story is not that I should have read a textbook front-to-back before doing any serious projects myself. All the hands-on work made me incredibly enthusiastic about the domain, and did give me real skills. I would have benefited from a mentor who could tell me very directly: “Oh, you’re stuck because you don’t understand pointers. Read these couple of pages, then let’s chat about it, and I’ll suggest what you might look at next.” That would have saved me a few years of frustration.
My programming story was possible only because programming is a field where you can start engaging in meaningful activity with very little understanding. Cooking was this way, too. Topics with that property are particularly amenable to learning-by-doing approaches. The learning problem mostly becomes one of orchestrating secondary activities to fill conceptual gaps as they become salient. In my experience, the likeliest failure mode is that I spend too little time on conceptual learning, because I can’t discern the impacts of my gaps, or the possible value of acquiring certain understandings.
On the other hand, most topics do require a fair amount of “book learning” before I can dive into a meaningful project of my own. I have a few ways of dealing with these situations.
One approach is to treat the first phase of “book learning” like map-making, reading shallowly to understand what kind of initial project I might like to tackle, and what I should learn to make that possible. Then I can read deeply in those slices, confident that I’ll be rewarded with some meaningful activity.
Another approach is to find a way to fall in love with the ideas themselves. I grew up uninterested in math largely because I only had bad math books and bad math teachers. I saw math as instrumental, as broccoli to be swallowed to help me with my programming. Later in life, I was introduced to discussions of math filled with beauty and awe, and I found it easy to immerse myself in these without needing to have any particular project in mind. I dearly wish I’d found that kind of math earlier.
I’m often asked: Are memory systems really necessary? If the ideas are important, won’t they come up naturally in your work? And if they’re not important, isn’t it good that they be forgotten? A few of my standard answers echo the discussion above:
But I think these objections do point to important problems with memory systems. Review often feels boring and detached from things I actually care about. Too often, I’m captivated by a discussion of an idea, but when the associated prompts arise in my memory system, I’m left cold. And skill-related practice often feels like it’s much less effective than it could be because I’m not doing those tasks in the real environment where that knowledge will be used. I believe it’s possible to make progress on all these problems, both by becoming more skilled with the systems we already have, and by creating system-level improvements.
Thanks to Michael Nielsen, Alec Resnick, and Gary Bernhardt for many past discussions which inform my views on these points. Thanks also to José Luis Ricón, whose article “Massive input and/or spaced repetition” nudged me to write on this topic. And finally, thanks to Nick Barr for the term “devotional learning”, which I’m roughly appropriating here.