2022-11-30 Patreon update - Cultivating depth and stillness in research

A consistent challenge in my development as a researcher has been: how to cultivate deep, stable concentration in the face of complex, ill-structured creative problems?

In roles oriented around operation and execution, I benefited enormously from standard “productivity” advice. Task managers and time-planning tools were essential. But now, task managers and calendars only help with the least important pieces of my work.

Bill Thurston writes:

Mathematics is a process of staring hard enough with enough perseverance at the fog of muddle and confusion to eventually break through to improved clarity.

This description resonates more with my experience of design research than anything Getting Things Done has to say, valuable though it was in my past life. To make progress in my present work, I need to “stare hard enough and with enough perseverance at the fog of muddle and confusion.” But if I’d read that last sentence five years ago, I don’t think I’d have really understood what it meant. I wouldn’t have grasped how difficult it is to stare this way, or how impossible progress is without this state of mind. Here’s what I might tell my past self:

“Why is this so hard? Because you’re utterly habituated to steady progress—to completing things, to producing, to solving. When progress is subtle or slow, when there’s no clear way to proceed, you flinch away. You redirect your attention to something safer, to something you can do. You jump to implementation prematurely; you feel a compulsion to do more background reading; you obsess over tractable but peripheral details. These are all displacement behaviors, ways of not sitting with the problem. Though each instance seems insignificant, the cumulative effect is that your stare rarely rests on the fog long enough to penetrate it. Weeks pass, with apparent motion, yet you’re just spinning in place. You return to the surface with each glance away. You must learn to remain in the depths.”

I’ve gotten much better at this. I need to get much better still! I’d like to share a few notes about my progress with this problem—mostly just reflecting “aloud”. My strategies aren’t at all intended to generalize, but I hope that my experiences might offer some hints to others seeking more depth.

Why flinch away? Some personal psychology

First: why do I flinch away when progress is slow and next steps are unclear? Why do some people seem not to flinch away? I’ve noticed a few patterns at play.

The first seems to be faulty expectations. I spent years in the tech industry; I internalized the pace of progress appropriate to industry problems. Some part of me expects that same pace on a totally different class of problems. The slow pace of research problems feels viscerally much less stimulating than I’m used to. Unbidden, my attention seeks out other more immediately rewarding targets. Sometimes this is obvious (
“answer email”, “browse Twitter”); but behaviors like “read some papers” or “hack together a prototype” are often subtle grasps for more immediate stimulation.

It’s possible to get a feel for this effect on very short time scales. If I find myself sucked into an hour-long Twitter binge, I’ll become noticeably more habituated to ultra-fast reward cycles. For hours afterwards, everything else will feel much slower, way less stimulating. I’ll suddenly need real willpower to read a book for a solid hour. The acute effect wears off after a few hours, but some fraction of it persists into the next day.

What to do about faulty expectations? An illustrative list:

• Collect vivid stories which reinforce a more realistic pace of progress for this type of work. Memoirs of scientists and artists are great for this. Mason Curry’s Daily Rituals is a nice anthology in this vein.
• Practice mentally noting the impulses as they arise; make it a game to catch them as “early” as possible, listening for ever-quieter cravings.
• Savor the subtle insights which really do occur regularly in research. Think of it like cultivating a much more sensitive palate.
• Consciously fortify myself when interacting with industry people; don’t compare my pace to theirs; don’t accidentally internalize their values. This is quite tough for me living in San Francisco. I’m immersed in industry culture here, and most of my friends are founders.

Another important pattern for me is fear. When progress isn’t evident, I quietly wonder: “Is this a good idea after all? Is progress even possible? Is the problem here that I’m not good enough to make progress?” When I exert willpower to press on, I inflame those automatic fears. “Wait, I’m continuing anyway?! That’ll only make my incompetence more obvious! Others will lose respect for me; I’ll get cut out of things; I’ll end up alone and miserable.” As a deeply lonely teenager, I learned that I could earn others’ regard and become valued in a community by “doing cool stuff on the internet.” So, even today, my automatic response to these fears is to switch to an activity which produces some kind of visible output. Make a prototype, write up some notes, sketch a concept. These are appropriate behaviors at times, of course, but not when pursued as fearful substitutes for what I’m actually trying to do.

What to do about fearful reactions? An illustrative list:

• These fears are modulated by my faulty expectations; see previous list.
• Visualize close friends; notice that our relationships don’t seem even slightly contingent on the status of my work. Notice that less-close relationships which do seem contingent don’t feel terribly precious; notice that I don’t fear losing them.
• Feel into the past work that’s most gratifying in hindsight; notice that it is never the flailing result of an impulse to produce “output”. Notice that this work is exclusively the product of perseverance and unchartable paths.
• Mentally note these fearful sensations, as early as possible.
• Obvious tactics everyone recommends for this sort of thing: therapy, meditation, psychedelics.

Another pattern has to do with my stance towards the work. I’m much more likely to flinch away from difficulty when I view my research problem as a task, as something to be accomplished. I’m much less likely to flinch away when I’m feeling intensely curious, when I truly want to understand something, when it’s a landscape to explore rather than a destination to reach. Happily, curiosity can be cultivated. And curiosity is much more likely than task-orientation to lead me to interesting ideas.

I make a practice of regularly checking in about whether I have a dutiful stance towards some aspect of my research. Once I notice, I can usually summon curiosity by asking lots of questions, imagining potential implications, and so on1. Often I need to improve the framing, to find one which better expresses what I’m deeply excited about. If I can’t find a problem statement which captures my curiosity, it’s best to drop the project for now.

Curiosity can also totally change my relationship to setbacks. Say I’ve run an experiment, collected the data, done the analysis, and now I’m writing an essay about what I’ve found. Except, halfway through, I notice that one column of the data really doesn’t support the conclusion I’d drawn. Oops. It’s tempting to treat this development as a frustrating impediment—something to be overcome expediently. Of course, that’s exactly the wrong approach, both emotionally and epistemically. Everything becomes much better when I react from curiosity instead: “Oh, wait, wow! Fascinating! What is happening here? What can this teach me? How might this change what I try next?” The same applies to writing. For example, when one topic doesn’t seem to fit a narrative structure, it often feels like a problem I need to “get out of the way”. It’s much better to wonder: “Hm, why do I have this strong instinct that this point’s related? Is there some more powerful unifying theme waiting to be identified here?”2

Big morning block

I’m wary of metrics, and I don’t think concentration lends itself to sensible KPIs. But in my experience, most of my progress has come from careful monitoring and reflection over time—a mixture of qualitative and quantitative data, all seasoned with much salt.

The most important practical thing I’ve done for my depth of concentration is this: I do my primary creative work in one giant unbroken block, starting 7-8AM and ending 1-2PM, with no meetings or extended breaks. I think the important thing here is not the solution, which is probably specific to me, but the process I used to reach it.

My initial prompt was mundane: if I want to take a meeting, when should I schedule it to cause minimal disruption? I tried coupling meetings with lunch breaks; I tried end-of-day; I tried mid-afternoon coffee dates. Reviewing my journals, I noticed that no matter what I did, I almost never got much depth in the afternoon. Evenings usually felt even worse.

I’ve long been fascinated by what William James has called “the energies of men”:

I wish to spend this hour on one conception of functional psychology, a conception never once mentioned or heard of in laboratory circles, but used perhaps more than any other by common, practical men — I mean the conception of the amount of energy available for running one’s mental and moral operations by. Practically every one knows in his own person the difference between the days when the tide of this energy is high in him and those when it is low, though no one knows exactly what reality the term energy covers when used here, or what its tides, tensions, and levels are in themselves.

Years ago, I got an Apple Watch app called Tracker. It’s very simple: it taps me on the wrist at a few random times each day, and asks me to indicate my mental energy level on a 1-to-5 scale3. What’s important about this app is that it samples randomly, rather than relying on me to choose when I record samples; and also that it’s an unobtrusive one-tap interaction. The notification appears on my wrist with five buttons; I tap one; it disappears. At no point am I looking at my phone.

I’ve used this app for years to run experiments involving my energy levels. Turning to this data again in the context of my meeting question, a clear story quickly emerged. Here’s a year of data (~50-75 samples per hour):

Conditioned by years in a typical office, I’d been working from roughly 8 AM through noon, then taking an extended break for lunch and to recharge, then returning to my desk around 1:30 or 2 PM for a second round… which never seemed to go all that well. That was a huge mistake! I’ve still got lots of mental energy around noon, and even around 1 PM. I was taking a break during that period, then returning just in time for my 2 PM energy nose-dive, which doesn’t recover until a much shorter second wind at around 6-7 PM4.

So this year I’ve worked in one big, 6-7 hour block, starting between 7-8AM and wrapping up between 1-2PM. I prepare lunch in advance5 and eat it at my desk. Depth of concentration is cumulative, and precious. An extra hour or two of depth is enormously valuable. I reliably get more done—and with more depth—in that 6-7 hour morning block than I’d previously done in 9-10 hours throughout the day.

This feels wonderful. By 2PM, I’ve done my important work for the day. I know that no more depth-y work is likely, and that I’ll only frustrate myself if I try—so I free myself from that pressure6. I take meetings; I exercise; I meditate; I go on long walks. I’ll often do shallower initial reads of papers and books in the afternoon, or handle administrative tasks. Sometimes I’ll do easy programming work. It’s all “bonus time”, nothing obligatory. My life got several hours more slack when I adopted this schedule, and yet my output improved. Wonderful!

Tuning breaks

I simplified a bit in this last section. I don’t actually work in a 6-7 hour unbroken block. My experience has been that intermittent short breaks—5 minutes, not 15 or 30—help depth of concentration enormously.

The best explanation I have is that the work is simply too difficult, or that I’m simply not conditioned enough, to focus deeply and continuously for hours. If I try, then I rapidly burn myself out. What this feels like, in the moment, is that I’ll find I need to apply more and more willpower to maintain my concentration, until the pressure’s too much, and I give in to distraction to release it.

A short break lets my willpower muscles recharge. Five minutes works well for me; longer periods erode depth more sharply. If the break is done right, it won’t disrupt depth of concentration much. The important thing is to diligently avoid doing anything cognitive during the break: reading or thinking about another topic imposes higher switching costs. My favorite activity is to do simple household chores. I’ll pick things up, wash dishes, and prep vegetables for dinner. The temptation is to look at my phone during breaks, to read something, to answer messages—but that’s a terrible idea. It brings me all the way back to the surface, makes me claw my way back to depth again when the break timer ends.

But what should the working interval be between breaks? The standard “pomodoro” recommendation is a 25 minute working period, followed by a 5 minute break. When I’m really struggling to concentrate, a shorter working period does help—it feels like lifting lighter weights at the gym. But the breaks do impose a real switching cost. Sometimes they interrupt a smoothly-chugging train of thought; sometimes it takes a few minutes to resume in the next session. And of course there's a practical time cost. The typical pomodoro schedule consumes 17% of the working period in breaks. In practice, I find that I'll take a couple minutes to finish what I'm doing and settle back in at my desk when the timer goes off. I was surprised to measure the true cost at 28% of my total time spent on breaks!

Would longer working intervals be better? Or would the price paid in focus be higher than the benefit of less time spent on breaks? I ran an experiment. Every morning, I randomly chose a working interval: 25, 35, 45, or 55 minutes, with 5 minute breaks7. I recorded a time sheet with subjective focus ratings and task information. I did this for 40 days, 10 for each interval.

The aggregate findings were unsurprising. Time efficiency is much better for longer working intervals: the median 45 and 55 minute days had almost an hour of extra real working time. Conversely, longer intervals produced less reliable deep focus. Shallow focus was about twice as likely with 55 minute intervals by comparison to 25 minute intervals. But the most interesting result was that the effect of long working intervals on focus shifted throughout the day, as depicted in this figure:

For the first two hours of the day, longer working blocks don’t seem to impose a noticeable penalty on focus. But by the end of the day, long blocks have shallow focus half the time, while short blocks might actually have gotten more focused.

So for the past two months, my schedule has been: 55 minute intervals before 10 AM; 45 minute intervals between 10 AM and noon; 25 minute intervals afterwards. I hit higher depth-of-focus ratings more often now than I did before this change, and I’ve added ~45 minutes of clocked working time onto my morning blocks—without actually making them any longer.

The data also suggest that, as expected, some kinds of tasks are easier to focus on than others:

This figure suggests I might try varying the working interval with the difficulty of the task. I’d need to be careful there; for example, in the case of programming, the scores are poor in this sample because the programming I was doing was so mundane and annoying, and I didn’t want to be doing it—not because it was difficult.

Micro problem-solving

I’ve discussed some big changes I’ve made which have helped my depth of concentration. But day to day, much of the work is in training myself to notice small impediments to depth. Lots of tiny problem solving. Here’s a non-exhaustive list:

• One obvious category is around technology and internet use.
  • I use Focus to block obvious displacement activities during my working block (email, Twitter), but I noticed myself dodging difficult work by diving down literature rabbit holes, or endlessly chasing visual reference material, or pulling on some irrelevant technical thread. So now the WiFi is off by default on my computer in the morning.
  • I noticed that sometimes I’d re-enable the WiFi because I really actually needed to look something up… and then I’d forget to turn it off. So I made an Alfred workflow which turns it back on for an explicitly-specified period, so that I have to say “give me internet for 3 minutes.”
  • It goes without saying, but no internet on my phone before I sit down at my desk. I don’t want anyone else’s thoughts in my head before I start thinking my own.
  • I’d find myself looking at my phone during breaks, even though I know it harms focus. So now I use Forest in the morning. It lets me use my phone to play music but blocks just about everything else.
  • But then I had a problem: sometimes packages would arrive, and I’d need to buzz them in; my dog walker would tell me they’ll arrive soon. So I configured a “Focus” mode on my iPhone which only allows notifications from the couple sources I really want to be responsive to in the morning. This mode’s on a schedule, so that it’s automatically enabled during my working block.
• When I’m stuck, I’ll often find myself feeling sleepy. I think this is just a consequence of expecting more stimulation than I’m getting. When I notice this, I play energetic music, do a quick exercise, etc.
• If I spend a working interval flailing, never sinking below the surface, the temptation is to double-down, to “make up for it”. But the right move for me is usually to go sit in a different room with only my notebook, and to spend the next working interval writing or sketching by hand about the problem.
• Administrative tasks are a constant temptation for me: aha, a task I can complete! How tantalizing! But these tasks are rarely important. So I explicitly prohibit myself from doing any kind of administrative work for most of the morning. In the last hour or two, if I notice myself getting weary and unfocused, I’ll sometimes switch gears into administrative work as a way to “rescue” that time. Otherwise I do these tasks in the afternoon; I’ve trained myself not to mind if they’re very delayed or dropped. (My apologies to anyone waiting on a reply to an email.)
• I noticed that I’d often take a few minutes to transition fully from a break into a new working interval. After some experimentation, I’ve found that a 15-second meditation lets me make that shift immediately. Body scans work well for me.
• When I’m doing very difficult intellectual work, I find that being around other people harms it. When I’m doing work I don’t want to be doing (e.g. mundane programming), I find that being around other people helps my depth of focus.
• Music of some kind seems to help me focus, but the right kind depends on the work I’m doing. If what I’m doing is very difficult (tempting dullness), it should be something subdued, repetitive, and non-vocal; if it’s easy (tempting boredom and distraction), it should be active, something I’d want to sing along to.
• To preserve the sanctity of my morning working blocks, I almost never accept meetings before 2PM. This makes it tough to meet with Europeans, an awkward 7-9 hours away. I solve that problem by offering them weekend morning slots.
• If I want to make more progress on a difficult creative project, a good way to ratchet up the intensity of my work is to add weekend mornings. Trying to work more hours on weekdays rarely gets me anywhere with difficult intellectual work.
• I’ve noticed that unhealthy afternoon/evening activities can easily harm the next morning’s focus, by habituating me to immediate gratification.
  • For a while, I’d regularly listen to
    audiobooks and podcasts when walking. But this contributed to an internal expectation of constant stimulation. I’ve pulled back: better to let my mind wander. Now I listen to those things when exercising and cooking, since I find that the physical stimulus is distracting enough to keep me from doing much useful thinking.
  • Twitter, Mail, and so on are disabled on my phone so that even in the afternoon/evening, these automatic activities don’t scatter the next day’s focus.

Key to much of what I’ve described here is a journal, and a practice of weekly, monthly, quarterly, and annual review. I don’t think the details of those reviews is very important—most of the benefit just seems to come from regularly reflecting on what I’m trying and what’s happening as a result. It’s really about developing a rich mental model of what focus and perseverance feel like, and what factors seem to support or harm those states of mind.

On that note, I should mention: everything I’ve written here applies to the kind of work Thurston was talking about—work which requires staring hard enough and with enough perseverance at a fog of muddle and confusion. Sometimes what’s needed is to explore aimlessly. Sometimes a problem requires playful lightness and expansiveness. These modes need a completely different set of practices, and they’re often best with others. Sometimes I just need to execute; and then traditional productivity advice helps enormously. But deep insight is generally the bottleneck to my work, and producing it usually involves the sort of practices I’ve described here.

Thanks to Catherine Olsson, Kanjun Qiu, and Michael Nielsen for many helpful conversations on this subject. Thanks to Sara LaHue for suggesting that I look into interactions between time of day and working interval on depth of focus.

1 Michael Nielsen has some great notes on tactics for this in his Notes on creative contexts.

2 I notice that I really struggle to generate curiosity about problems in programming. Maybe it’s because I’ve been doing it so long, but I think it’s because my problems are usually with ephemeral ideas, incidental to what I actually care about. When I’m fighting some godforsaken Javascript build system, I don’t feel even slightly curious to “really” understand those parochial machinations. I know they’re just going to be replaced by some new tool next year. When I’m debugging some dataflow race condition, I can at least get a little interested in fundamental problems of program structure. Long ago, that was my full-time job! Unfortunately, most of my programming problems these days are the parochial kind. They make me resent programming, which is a shame: it was once the most joyful activity of my life!

3 You can make it ask about whatever you like (rather than energy level) and configure whatever response scale you like. But there’s a significant caveat: this app includes no data visualizations of any kind. It just gives you a JSON dump, which I analyzed using R.

4 I did an extended experiment with 20-30 minute afternoon naps around 2-3PM (~8th waking hour). They felt nice, so I still do them sometimes, but they didn’t substantially improve my afternoon mental energy levels. Null results also for mid-afternoon exercise.

5 I’ve also done experiments around how lunch affects my mental energy. I can’t distinguish a chopped salad, a quiche, and a protein-centric sandwich in my energy levels, but egregiously carb-y choices like burritos and pastries do hurt my energy. I usually eat a chopped salad for lunch these days.

6 I notice that some part of me feels ashamed to say that I’m “done” working at 2PM. This is probably because in my previous roles, I really could solve problems and get more done by simply throwing more hours at the work. That’s just obviously not true in my present work, as I’ve learned through much frustration. Reading memoirs of writers, artists, and scientists, I see that 2-4 hours per day seems to be the norm for a primary creative working block. Also, I don’t want to harp on this because I want this essay to be about quality, not quantity, but: I think most people are laughably misled about how much time they truly work. In a median morning block, I complete the equivalent of 12 25-minute pomodoros. When I worked at large companies, getting 8 done before 6PM was a rarity—even though I’d assiduously arrange my calendar to maximize deep work!

7 I just use the timer function on my Apple Watch. There are apps for this, but I’m often working with a paper notebook away from my computer. And I don’t want to use my phone.