The threat of AI-powered bioweapons serves as an alarm bell for the biotechnology industry

In recent weeks, executives at the largest AI companies have issued stark warnings that the technology they are creating may pose existential dangers. Anthropic CEO Dario Amodei said last weekend that AI carries serious risk and that development should slow down. OpenAI CEO Sam Altman replied on X that he agreed: "we need to pace the frontier." Behind these statements lies a specific fear shared by many researchers—that AI could help someone design and release a biological weapon. Understanding why that worry has grown is now central to the debate over how fast AI should advance.

When scientists sounded the alarm

The concern crystallized in 2022, when researchers at Collaborations Pharmaceuticals repurposed an AI "molecule generator" originally built to discover potential drugs. Within less than six hours, the model produced 40,000 molecules that could serve as chemical warfare agents—some more toxic than known nerve agents. The authors wrote at the time that, without being overly alarmist, the result should be a wake-up call for the AI drug discovery community.

It was exactly that for David Magnus, a professor of medicine and biomedical ethics at Stanford University, who has studied the misuse of biotechnology since the late 1990s. "That was very scary to me," he said, adding that everything since then has "just sort of blown up."

The unease has reached inside the labs themselves. AI researcher Jacob Coxon recently announced his departure from Anthropic, arguing that neither it nor OpenAI, where he had also worked, was acting responsibly. "The people building AI earnestly believe that it could kill us all by the end of the decade," he wrote on X. Anthropic employee Evan Hubinger publicly agreed: "I personally think it is >10% within the next decade."

Why the risk feels newly real

A bioweapon could take many forms: a highly lethal virus targeting people by their genes, a crop-destroying fungus that triggers food insecurity, or a tasteless, odorless toxin slipped undetected into a region's water supply.

What has changed is accessibility. Today's large language models, trained on the accumulated knowledge of "almost every scientist who ever lived on this planet," can answer questions across all fields of science and even provide experimental instructions, says Dunja Sabra, a biosecurity researcher at the University of Hamburg in Germany. Combined with cheaper gene editing and synthetic biology tools—and the rise of "DIY biology," which has already let many people build home labs—the situation looks potentially dangerous. "The chances are that someone determined would succeed eventually," Sabra says.

The defenses, and their limits

Safeguards do exist. DNA synthesis companies typically screen orders for suspicious sequences. Responsible researchers run risky work through "red-teaming," where independent scientists probe for misuse, and "blue-teaming," where others devise mitigations. AI companies have also adjusted their models to withhold dangerous scientific information.

But none of these protections is airtight. In a report published last week, Anthropic acknowledged that users had attempted to coax its models into exploring how to make the chikungunya virus more transmissible, engineer a bird flu more dangerous to humans, and compile an "atlas of venom toxin peptides," among other attempts.

"We've got a constant back and forth," Magnus says. "We have to build better surveillance and screening tools, [but] AI is really good at figuring out ways around them." He believes AI itself may be needed to restrict AI. On Wednesday, MIT biologist Kevin Esvelt went further, writing on X that a large language model had "disclosed a novel form of bioweapon that I hadn't realized was possible," and urging caution "for the love of God, children, the future of humanity, or whatever you consider holy."

Scientists don't all agree

Not every expert shares this level of alarm. At a recent media briefing, some biologists at Imperial College London argued that AI tools are simply not capable of fully developing bioweapons, noting that testing new pathogens requires difficult, time-consuming human labor. Some believe existing guardrails are sufficient.

Wendy Barclay, a professor of infectious disease at Imperial, added a further point: the greatest pandemic risk today comes not from bioweapons but from pathogens already circulating. H5N1 bird flu, which has killed millions of birds and spread widely through US dairy cattle, was detected last month in captive mink at a Utah farm.

Sabra prefers to look five to ten years ahead, arguing that countries should strengthen health-care systems, prepare antidotes to known toxins, and stockpile medicines. "We need to be prepared," she says.

What to watch next: the gap between AI capabilities and safeguards will keep narrowing, so expect continued clashes between acceleration and safety at frontier labs—and growing pressure for independent testing of what models can and cannot reveal.

This article first appeared in The Checkup, MIT Technology Review's weekly biotech newsletter.

Meta description: AI CEOs warn the technology could aid bioweapons. Inside the 2022 molecule-generator scare, Anthropic's findings, and why scientists disagree on the risk.

Tags: AI safety, bioweapons, Anthropic, biosecurity, large language models

Featured image: abstract dark digital artwork of molecular structures dissolving into streams of data, no people, no logos, no text.