Hi! I’m Andy Matuschak. You’ve stumbled upon my working notes. They’re kind of strange, so some context might help.
These notes are mostly written for myself: they’re roughly my thinking environment (Evergreen notes; My morning writing practice). But I’m sharing them publicly as an experiment (Work with the garage door up). If a note seems confusing or under-explained, it’s probably because I didn’t write it for you! Sorry—that’s sort of an essential tension of this experiment (Write notes for yourself by default, disregarding audience).
For now, there’s no index or navigational aids: you’ll need to follow a link to some starting point. You might be interested in §What’s top of mind.
👋 Andy (email, Twitter, main personal site)
PS: My work is made possible by a crowd-funded research grant from my Patreon community. You can become a member to support future work, and to read patron-only updates and previews of upcoming projects.
PS: Many people ask, so I’ll just note here: no, I haven’t made this system available for others to use. It’s still an early research environment, and Premature scaling can stunt system iteration.
Particularly in Silicon Valley, when one has a prototype or an inkling that works well, the temptation is to scale it out. Make it work for more people and more use cases, turn it into a platform, make the graphs go up and to the right, etc. This is obviously a powerful playbook, but it should be deployed with careful timing because it tends to freeze the conceptual architecture of the system.
General infrastructure simply takes time to build. You have to carefully design interfaces, write documentation and tests, and make sure that your systems will handle load. All of that is rival with experimentation, and not just because it takes time to build: it also makes the system much more rigid.
Once you have lots of users with lots of use cases, it’s more difficult to change anything or to pursue radical experiments. You’ve got to make sure you don’t break things for people or else carefully communicate and manage change.
Those same varied users simply consume a great deal of time day-to-day: a fault which occurs for 1% of people will present no real problem in a small prototype, but it’ll be high-priority when you have 100k users.
Once this playbook becomes the primary goal, your incentives change: your goal will naturally become making the graphs go up, rather than answering fundamental questions about your system (contra Focus on power over scale for transformative system design).
One huge advantage to scaling up is that you’ll get far more feedback for your Insight through making process. It’s true that Effective system design requires insights drawn from serious contexts of use, but it’s possible to create small-scale serious contexts of use which will allow you to answer many core questions about your system. Indeed: technologists often instinctively scale their systems to increase the chances that they’ll get powerful feedback from serious users, but that’s quite a stochastic approach. You can accomplish that goal by carefully structuring your prototyping process. This may be better in the end because Insight through making prefers bricolage to big design up front
Eventually, of course, you’ll need to generalize the system to answer certain questions, but at least in terms of research outcomes, it’s best to make scaling follow the need expressed by those questions. In that sense, it’s an instrumental end, not an ultimate end.
(See central discussion in How can we develop transformative tools for thought; subsection “How to invent Hindu-Arabic numerals?”)
Certain theoretical insights aren’t apparent when staring at the theories in isolation. But when those theories are instantiated in systems, observations of the system’s behavior can reveal the theory from new angles, expanding the space of possible insight. In the best cases, theoretical insights expand the systems you can construct, and insights from those systems expand your theory, in a virtuous cycle.
But to pursue both “halves” of this cycle requires expertise in both system-creation and in the subject matter; i.e. Inventing the Hindu-Arabic numerals requires both design genius and mathematical genius.
Most practitioners focus either on developing theoretical insight or on making systems, paying only lip service to the other half of this cycle. Academics often build toy systems which don’t create meaningful contexts for their ideas (Academic software interfaces generally aren’t suitable for serious use). Designers and engineers often focus on building systems without deep understanding of the theory which underlies that system, or they focus on scale before power (contra Focus on power over scale for transformative system design).
Both types of practitioners often develop systems for the sake of developing a system—not to address some personally meaningful problem they’re very motivated to solve: Tool-makers usually lack connection to a serious context of use. The insights drawn from such systems are generally quite weak: Effective system design requires insights drawn from serious contexts of use.
This insight is particularly important in the development of Tools for thought.
More on how to achieve insight through making:
Matuschak, A., & Nielsen, M. (2019 0). How can we develop transformative tools for thought? https://numinous.productions/ttft
When developing systems which push the frontiers of their own foundational theory, Insight through making suggests that you’ll need to make simultaneous progress in theory-space and system-space to spot the new implications in their conjoined space. Effective system design requires insights drawn from serious contexts of use: you must constantly instantiate new theoretical ideas in new systems, then observe their impact in some serious context of use.
These small feedback loops are better suited to bricolage than to big design up front. Critically, though, you must try to make the bricolage accrete: the temptation is often to move in more of a drunkard’s walk.
I suspect that this is part of why Ted Nelson’s systems weren’t as successful as they could have been: he and his colleagues engaged in endless up-front design without seriously using any of their ideas to author text or do their work.
I probably err on the side of just making things happen… I get so much of a thrill bringing things to life… as soon as it comes to life it starts telling you what it is.
—Dan Ingalls, as quoted in Nielsen, M. (2018). Augmenting Long-term Memory. http://augmentingcognition.com/ltm.html
Scrappy prototypes are great: they allow scrappy iteration and quick evaluation. But many critical insights will only emerge in the context of a serious creative problem that’s not about the system itself. This is a key claim of Insight through making.
That sounds like standard practice: of course systems have to be evaluated! But most system designers don’t take “serious” seriously: Tool-makers usually lack connection to a serious context of use.
Observing how your theories (represented in systems) interact with reality can yield insights which help improve your theories. The character of those insights will depend on the context in which the system was used. If the system isn’t used seriously, the insights will be more like those which a pure theorist could have seen. Those were possible without actually building a system.
Pixar’s a good example of an organization which creates serious contexts of use, which in turn drive system design: Pixar’s movies and technology development act as coupled flywheels.
Common challenges:
Related theory:
Matuschak, A., & Nielsen, M. (2019). How can we develop transformative tools for thought? Retrieved December 2, 2019, from https://numinous.productions/ttft
Concretely: suppose you want to build tools for subject X (say X = differential geometry). Unless you are deeply involved in practicing that subject, it’s going to be extremely difficult to build good tools. It’ll be much like trying to build new tools for carpentry without actually doing any carpentry yourself. This is perhaps part of why tools like Mathematica work quite well – the principal designer, Stephen Wolfram, has genuine research interests in mathematics and physics. Of course, not all parts of Mathematica work equally well; some parts feel like toys, and it seems likely those are the ones not being used seriously internal to the company.
Brooks, F. P., Jr. (1994). The Computer Scientist as Toolsmith II ACM Allen Newell Award Lecture. SIGGRAPH.
It aims us at relevant problems, not just exercises or toy-scale problems.
It keeps us honest about success and failure, so that we don’t fool ourselves so easily.
It makes us face the wholeproblem, not just the easy or mathematical parts. In computational geometry, for example, we can’t avoid the cases of collinear point triples or coplanar point quadru- ples. We can’t assume away ill-conditioned cases.
Facing the whole problem in turn forces us to learn or develop new computer science, often in areas we otherwise never would have addressed.
Besides all of that, it is just plain fun to look over the shoulders of those discovering how proteins work, or designing submarines, or fabricating on the nanometer scale.
Software-based researchers often strive to build systems containing high-level ideas that are likely to generalize, since those make for more compelling academic papers. However, we believe that trying to be too general actually hinders scale and sustainability. To build long-lasting software that can organically grow a large userbase, one must instead start specific.
In 2009 we created Python Tutor with a very specific goal in mind: to provide a convenient way for students and instructors (such as ourselves) to walk through Python code step-by-step and see the values of variables.