Hi! I’m Andy Matuschak. You’ve stumbled upon my working notes. They’re kind of strange, so some context might help.
These notes are mostly written for myself: they’re roughly my thinking environment (Evergreen note-writing as fundamental unit of knowledge work; My morning writing practice). But I’m sharing them publicly as an experiment (Work with the garage door up). If a note seems confusing or under-explained, it’s probably because I didn’t write it for you! Sorry—that’s sort of an essential tension of this experiment.
For now, there’s no index or navigational aids: you’ll need to follow a link to some starting point. You might be interested in §What’s top of mind.
PS: My work is made possible by a crowd-funded research grant. If you find these ideas interesting and want to see them developed further, please consider becoming a micro-grantmaker yourself on Patreon.
PS: Many people ask, so I’ll just note here: no, I haven’t made this system available for others to use. It’s still an early research environment, and Premature scaling can stunt system iteration.
Particularly in Silicon Valley, when one has a prototype or an inkling that works well, the temptation is to scale it out. Make it work for more people and more use cases, turn it into a platform, make the graphs go up and to the right, etc. This is obviously a powerful playbook, but it should be deployed with careful timing because it tends to freeze the conceptual architecture of the system.
General infrastructure simply takes time to build. You have to carefully design interfaces, write documentation and tests, and make sure that your systems will handle load. All of that is rival with experimentation, and not just because it takes time to build: it also makes the system much more rigid.
Once you have lots of users with lots of use cases, it’s more difficult to change anything or to pursue radical experiments. You’ve got to make sure you don’t break things for people or else carefully communicate and manage change.
Those same varied users simply consume a great deal of time day-to-day: a fault which occurs for 1% of people will present no real problem in a small prototype, but it’ll be high-priority when you have 100k users.
Once this playbook becomes the primary goal, your incentives change: your goal will naturally become making the graphs go up, rather than answering fundamental questions about your system.
One huge advantage to scaling up is that you’ll get far more feedback for your Insight through making process. It’s true that Effective system design requires insights drawn from serious contexts of use, but it’s possible to create small-scale serious contexts of use which will allow you to answer many core questions about your system. Indeed: technologists often instinctively scale their systems to increase the chances that they’ll get powerful feedback from serious users, but that’s quite a stochastic approach. You can accomplish that goal by carefully structuring your prototyping process. This may be better in the end because Insight through making prefers bricolage to big design up front
Eventually, of course, you’ll need to generalize the system to answer certain questions, but at least in terms of research outcomes, it’s best to make scaling follow the need expressed by those questions. In that sense, it’s an instrumental end, not an ultimate end.
Because Effects of the mnemonic medium on reader memory, it’s extremely tempting to scale it up. Yet the arguments described in Premature scaling can stunt system iteration suggest that platformization and generalization must be carefully timed.
In early 2020, there’s plenty of demand for a more general Mnemonic medium, but there are many dangers to attempting to rapidly scale it up.
We feel confident that we don’t know what the medium wants to be. For example, we only just recently discovered that The mnemonic medium can help readers apply what they’ve learned through simple application prompts. Those experiments represent potentially enormous changes to the medium. Further, The mnemonic medium can be adapted to author an experience which unfolds over time, and we’ve barely scratched that surface.
Separately, the medium is not yet good enough. For example, it’s true that Mnemonic essays may offer detailed retention of their contents in exchange for 35-50% reading time overhead, but our small experiments have suggested that there’s a huge amount of low-hanging fruit there.
Instead, we plan to lay the foundations for future scaling in 2020, emphasizing abstractions which won’t impede experimentation. It’s important that we scale to a small number of additional authors: their experiences will teach us much about the writer’s side of the medium.
The story I’d like to be able to tell:
Within Quantum Country, we have strong evidence of 1, moderate evidence of 3, and emerging evidence of 2. We have no evidence of 4 yet.
All this data is among readers who answered 80%+ of QCVC questions in the essay before their first review session.
After 6 repetitions (using an earlier, less aggressive SRS schedule), most 2019H1 users average about 54 days of retention per question.
The median 2019H2 reader had demonstrated 2-week retention on 95%+ of QCVC prompts by session 9. Source These users average about 24 days of retention per question after 3 repetitions of every prompt. We don’t have data on later repetition numbers yet.
But is this just a survivorship effect? Would these readers have developed this retention in any case? Or is it a selection effect—did these readers already have detailed retention of this material? What is the causal impact of the mnemonic medium’s review sessions on reader retention?