Scrappy prototypes are great: they allow scrappy iteration and quick evaluation. But many critical insights will only emerge in the context of a serious creative problem that’s not about the system itself. This is a key claim of Insight through making.
That sounds like standard practice: of course systems have to be evaluated! But most system designers don’t take “serious” seriously: Tool-makers usually lack connection to a serious context of use.
Observing how your theories (represented in systems) interact with reality can yield insights which help improve your theories. The character of those insights will depend on the context in which the system was used. If the system isn’t used seriously, the insights will be more like those which a pure theorist could have seen. Those were possible without actually building a system.
Pixar’s a good example of an organization which creates serious contexts of use, which in turn drive system design: Pixar’s movies and technology development act as coupled flywheels.
Common challenges:
Related theory:
Matuschak, A., & Nielsen, M. (2019). How can we develop transformative tools for thought? Retrieved December 2, 2019, from https://numinous.productions/ttft
Concretely: suppose you want to build tools for subject X (say X = differential geometry). Unless you are deeply involved in practicing that subject, it’s going to be extremely difficult to build good tools. It’ll be much like trying to build new tools for carpentry without actually doing any carpentry yourself. This is perhaps part of why tools like Mathematica work quite well – the principal designer, Stephen Wolfram, has genuine research interests in mathematics and physics. Of course, not all parts of Mathematica work equally well; some parts feel like toys, and it seems likely those are the ones not being used seriously internal to the company.
Brooks, F. P., Jr. (1994). The Computer Scientist as Toolsmith II ACM Allen Newell Award Lecture. SIGGRAPH.
It aims us at relevant problems, not just exercises or toy-scale problems.
It keeps us honest about success and failure, so that we don’t fool ourselves so easily.
It makes us face the wholeproblem, not just the easy or mathematical parts. In computational geometry, for example, we can’t avoid the cases of collinear point triples or coplanar point quadru- ples. We can’t assume away ill-conditioned cases.
Facing the whole problem in turn forces us to learn or develop new computer science, often in areas we otherwise never would have addressed.
Besides all of that, it is just plain fun to look over the shoulders of those discovering how proteins work, or designing submarines, or fabricating on the nanometer scale.
Software-based researchers often strive to build systems containing high-level ideas that are likely to generalize, since those make for more compelling academic papers. However, we believe that trying to be too general actually hinders scale and sustainability. To build long-lasting software that can organically grow a large userbase, one must instead start specific.
In 2009 we created Python Tutor with a very specific goal in mind: to provide a convenient way for students and instructors (such as ourselves) to walk through Python code step-by-step and see the values of variables.