Dynamic mediums usually lack an authored time dimension

==TODO this note should be refactored into several other notes==

One thing that distinguishes a Dynamic medium from other forms is that it behaves and responds (Victor 2012). Unlike other media forms, almost all dynamic mediums behave and respond the same way all the time. Users’ experiences change over time as they grow and interact with the medium differently, but that’s rarely a designed property. Dynamic mediums lack practices for communicating with an authored time dimension.

In film and books, by contrast, directors and authors carefully model and shape the participant’s experience over the course of the work. Often that takes the form of narrative, which can create emotional connection or help people gradually understand complex topics (Narrative as cognitive scaffolding). There’s no narrative form native to dynamic mediums.

Prose interleaved with dynamic environments creates a disjointed experience

One common approach is to interleave brief interactions with dynamic media into prose (e.g. Hart and Case, Patel. Victor (2010) is particularly fine-grained, though it’s not really trying to be narrative). This approach to narrative feels quite disjoint: the two forms have been glued together, not evolved into something new.

The shallow version of this issue is that the text and dynamic elements are almost always physically separated, not visually integrated. The reader’s eye bounces back and forth between the text and the interactive elements, attaching referents to objects and churning working memory. This is a problem with static figures, too; the solutions described by e.g. Tufte mostly haven’t been applied in the dynamic domain.

The felt experience has subtler bumps. Reading prose feels very different from engaging with interactive elements. I think it’s mostly due to abrupt changes in participatory stance: it feels a bit like someone asking you to switch between the passenger’s seat (prose) and the driver’s seat (dynamic elements) every minute or two.

One reason this swapping feels bad is that you know you’re only meant to drive for a few moments at a time. When prose and interaction are interleaved in this way, the interactions with the dynamic media are generally brief by design. The granularity must be fine enough that readers can comfortably remember the guidance provided in the preceding text. This makes the interactive moments feel shallow and lowers commitment. You know you’re not going to be there long; it’s not an environment to do serious creative work of your own. It’s more like an illustration—which is fine—but dynamic medium authors often aspire to make a more powerful Enabling environment. Some authors add a “sandbox mode” at the end of their articles, but this doesn’t mitigate the shallowness felt in-narrative.

Another challenge with prose interleavings is that the textual narrative completely ignores your interaction with the dynamic elements. The author suggests you try something; you do it; then the author’s text proceeds along its course, irrespective of whatever you just did. It’s emotionally alienating, much like talking to an automated phone tree. Sometimes the robot voice says things that sort of follow what you asked for, and just as often it serenely recites non sequitur. No one would expect prose narrative to naturally follow their thoughts and actions, but when the dynamic environment behaves and responds—when the author is trying to create a Participatory environment—the adjoining text’s interpersonal distance is magnified. Writers try to be incredibly empathic with their readers’ thoughts and feelings. These dynamic environments inadvertently make that job much harder—too hard, perhaps, for the medium.

These challenges are quite similar to the difficulties faced by board game instruction manuals.

Board games suffer from the same problems; video games don’t

Board games and video games both need to communicate explanation and narrative, but the way these two mediums communicate produces very different felt experiences.

When playing a new board game, you might take the instructions out of the box and read them for a few minutes, then set up the game, then try to remember the instructions while playing. Inevitably, you’ll return to the instructions every couple minutes during the first play session. This experience is awfully bumpy, with many of the same challenges as prose interleaved with dynamic interactions. You might feel like you’re constantly trying to remember what you’re “supposed to do.” You might “lose your place” in both the text and the game when you switch focus. You might have trouble emotionally committing to the game when you know you’ll only be able to play a few moments before returning to the manual. The manual’s voice feels quite distant from what’s actually happening on the game board.

Good video games mostly avoid these problems while accomplishing the same goals.

Video games often communicate with voice rather than text. Sometimes voice acting appears in non-interactive cutscenes, sharply delimited from ordinary play, but the best examples (e.g. Portal) present narrative as a seamless element of the interactive environment, never “stealing” the camera or the controls away from the player. Players’ eyes and attention don’t bounce back and forth between instructions and interaction in these games. To support players’ memory, such games might also add short textual phrases—again integrated into the game environment—but these generally act as summaries. They’re pointers to the prior “live” experience, not primary narrative elements of their own.

Because the narrative communication is integrated into the environment, it can behave and respond just like the rest of the environment’s elements. In good games, the authored narrative feels—continuously—like a response to players’ actions (i.e. an Enacted experience; see also Enacted experiences amplify the power of narrative). This feeling contrasts sharply with the unpleasant sense of distance felt in prose narrative presented alongside interactive environments. Of course, the dynamic environment means that the narrative could vary in response to players’ actions, but that’s challenging and poorly-understood. Good video game narratives feel naturally integrated into the interactive environment despite the fact that they’re often completely static.

Centering on the dynamic environment, not on the narrative

The examples I’ve given so far of narrative dynamic media are mostly focused on narrative. They mostly look like an article; text dominates the experience. The dynamic representations exist to support the narrative; they’re rarely meant to be powerful environments to think in apart from the text.

Even in more ambitious Narrated explorable attempts like Sanderson and Eater’s Visualizing quaternions, the “text” is primary. The dynamic representation isn’t meant to be a persistent environment supporting the viewer’s later creative work (see Matuschak (2018) for more on this piece).

It’s interesting to consider flipping that dynamic, just as video games center on the environment and present narrative as a transitory overlay. Instead of an article on programming with small embedded playground environments floating alongside the text, a programming environment with natural affordances for narrative. (e.g. What might it mean to situate games like Shenzhen I\/O inside pro environments like an IDE?) This might both resolve some of the narrative issues described above. More importantly, it makes serious use the primary consideration when designing the dynamic representation (Powerful enabling environments focus on expert use).

Twitch demonstrates the possibilities here in an interesting way. It’s not a narrative that you yourself experience, but it’s quite compelling to see an expert talk while centered on their own (dynamic) expert environment. (See also Could streaming help convey tacit knowledge?)

Twitch also demonstrates the potential value of separating the dynamic medium’s author (e.g. Adobe) from the narrative’s author (e.g. the streamer). Right now, narrative dynamic mediums almost always combine these roles. There are advantages to that, but executable notebooks begin to demonstrate the surprises which can arise from expanding authorship.
ev
More on a possible mechanism: Enacted experiences have incredible potential as a mass medium

==TODO==

• Precedent in the The Young Lady’s Illustrated Primer
• Executable books have many of the same issues, plus a few others, but with a few benefits

References

Hart, V., & Case, N. (2014, December 8). Parable of the Polygons. Retrieved from https://ncase.me/polygons/

Matuschak, A. (2018, October 26). Narrated explorables: Three mental models. Retrieved from https://medium.com/khan-academy-early-product-development/narrated-explorables-three-mental-models-e16e0d80e4c1

Patel, A. (2014, May 26). Introduction to the A* Algorithm. Retrieved from Red Blob Games website: https://www.redblobgames.com/pathfinding/a-star/introduction.html

Sanderson, G., & Eater, B. (2018). Visualizing quaternions. Retrieved from https://eater.net/quaternions

Tufte, E. R. (1997). Visual explanations: Images and quantities, evidence and narrative. Cheshire, Conn: Graphics Press.

Valve Corporation. (2007). Portal.

Victor, B. (2012, May). Stop Drawing Dead Fish. Presented at the SF SIGGRAPH. Retrieved from https://vimeo.com/64895205