AI risk

We’re creating and concentrating immense amounts of power. How can we avoid catastrophe?

Types of risk

• AI misuse
  • AI misinformation risk
• AI misalignment risk
• Social upheavals
  • Will generative AI cause catastrophic economic dislocation?

Related

• Many eminent AI scientists fear (various kinds of) AI risk
• AI capabilities research seems to be outpacing AI alignment research
• What is my personal moral responsibility with respect to AI risk?

So what? Slow down AI? Accelerate AI alignment work? Unclear.

---

Q. What’s Anthropic AI’s capability threshold for ASL-3?
A. “substantially increase the risk of catastrophic misuse” (i.e. CBRN) or (low-level) autonomous AI R&D capacity (source) ☉

Q. Which model caused the activation of Anthropic AI’s ASL-3 deployment and security standards?
A. Claude Opus 4 (2025-05-22, source)—as a precaution, due to leading indicators around virology assistance, not because it clearly crossed that threshold. ☉

Q. What new deployment and security commitments did Anthropic AI make in conjunction with Claude Opus 4?
A. Realtime monitors to detect and intervene on CBRN-related requests, new systems to detect and prevent universal jailbreaks, egress bandwidth controls on servers with model weights. ☉

References

Weidinger et al. - 2021 - Ethical and social risks of harm from Language Models
Threat Model Literature Review - LessWrong