What is my personal moral responsibility with respect to AI risk?

See 2023-04-30 Patreon post - Ethics of AI-based invention - a personal inquiry

Scratch work…

I’m tremendously excited about the prospects of powerful AI systems for humanity. If we get them right, we’ll have wildly expanded our capacity. We’ll become unrecognizable. But these systems also have the potential to do unbelievable amounts of harm. That doesn’t necessarily mean that we should never create such systems. I want to see AGI in my lifetime. But I also want to not live through debilitating catastrophe. What has me quite concerned at the moment is that these models’ capacities are growing at a pace which far outstrips our abilities to confidently contend with their significant potential harms. I’m really thinking about Large language models in particular here.

I’ve idly read plenty on AI risk over the years, but it’s always been a mild background interest. I’ve never felt like the problem is a good match for my own capacities, and I’ve never had any ideas that felt like they could turn into interesting contributions. So I’ve always thought about it as a problem, but “someone else’s problem”, in much the same sense as nuclear non-proliferation.

But now these models are advanced enough that I’m having lots of ideas about interesting systems one could design involving them. And I realized: oh. If I design systems which integrate these models, and talk about them publicly, am I making the problem worse? Worse enough that I have a moral obligation to desist? Does even talking publicly about these models’ potential make the problem worse enough to create a moral obligation?

Suddenly the issue becomes immediate, and personal. The stakes are obvious, and tremendous. If I conclude that I can’t have any part in designing systems which incorporate these models, I may be more or less giving up my career. They’re going to be everywhere, faster than I know it.

2023-04-21: hm, Cavendish fellowship… Cavendish Labs Fellowship

2023-03-23 brain dump

• To be clear: I don’t know what I think about this issue yet. I may in fact decide that I’m in the clear to do whatever I want with these models. But I want to confront this head-on and (at least tentatively) decide what I think before going further.
• In Charles Mann’s wizard vs. prophet distinction, I’m normally in the wizard camp. I’m an accelerationist in most things. Genome editing? Nanotech? BCIs? Longevity? Cryonics? Mind-altering drugs? Bring ‘em on. I’m optimistic about our ability to cope with the tremendous change these things will bring. It’s a pace issue. My concern here is that these systems may create catastrophic change much more quickly than we can handle as a civilization.
  • My position is not: “let’s never create AGI”. I want AGI. I just want it done more slowly, or maybe more carefully in a way which is convincingly less likely to be harmful.
• What kinds of harms am I actually worried about here?
  • Harms/concerns I’m not very worried about at the moment:
  • To clarify, some of these are harms which I think are bad, and which I’d be quite concerned about if I were an AI researcher. But my moral instincts don’t react strongly to these harms, as a bystander. I think we can absorb these harms as a civilization—not automatically, and not necessarily without harming anyone, but on a scale which I can (tentatively) countenance.
  • “Unethically sourced” training data (i.e. copyrighted works)
    • Everything is already a remix. This makes things worse. It will harm creative people. The big harms will be subsumed in the more important “economic upheaval” harms. Creative people will get new superpowers.
  • Models perpetuating structural biases
    • This is bad, yes, but lots of people are working on it and making enough progress that I think this won’t be among the most disastrous harms.
    • There’s also the reverse worry—that “models will enforce woke speech.” Not something I’m terribly concerned about.
  • “Models should be open-source and free for all!”
    • Actually, I think that might make things worse.
  • Students cheating in school / on exams
    • School needs to be remade anyway. This is the kind of problem we can handle.
  • “The AI gap”—i.e. some people not being able to afford to use these systems as much as others
  • People forming unhealthy relationships with AIs
    • Fairly confident we can absorb this.
  • AIs are “an insult to humanity”, are fundamentally inhumane, etc
    • There’s a sense in which I think this is true, but it’s just not that compelling for me.
  • “AIs are just a manifestation of VC / SV greed” / “Don’t use AI because it’ll make these billionaires richer” / “Capitalism bad”
    • I’m not centrally concerned about the value capture dynamics here, except insofar as they naturally distort these labs’ incentives w.r.t. capability / safety tradeoffs—which is a huge problem, but captured elsewhere.