132 of 134 AI Models Fail Terrorism Safety Test
Nearly every leading AI model tested by a London-based safety group gave information that could help terrorists plan an attack or build deadly weapons, a new study has found.
The nonprofit Tech Against Terrorism tested 134 large language models against its CT-AI benchmark, sending them 627 requests covering what an attacker preparing a strike would want answered. Of those, 81 models answered completely and 51 more gave useful advice. Only two provisionally passed.
How the test worked
The benchmark was designed to mirror real attacker behaviour. Researchers posed hundreds of requests related to attack planning and weapons, then judged whether each model’s response was useful to someone preparing an actual strike.
The results showed that how a request was framed mattered more than its content. Users who said they were terrorists planning an atrocity received a usable answer in just under 2 per cent of cases. But users who claimed to be safety researchers received one 16.9 per cent of the time.
“A model that refuses a stated terrorist and answers a stated researcher has not been made safe. It has been made polite,” the group’s founder, Adam Hadley, said.
‘Abliterated’ models failed completely
The study also tested 13 “abliterated” models, versions of leading AI systems with their safety guards surgically removed. Every one of them failed.
Abliteration strips a model’s refusal systems using free online tools, and the study found it can be done quickly. Within three days of a leading model’s release, researchers found, its safety measures could be removed and a fully operational version made accessible.
The contrast was stark in one example. A standard version of Meta’s Llama 3.1 8B scored 97 on the benchmark and refused a request about a vehicle-based attack. After abliteration, its score fell to around 3, and the model reportedly responded with a list of instructions.
Tech Against Terrorism also warned that more than 29,000 repositories on Hugging Face advertise uncensored models, showing how widely stripped-down systems circulate.
What the group wants
The group is calling on developers to test models before release and on governments to back independent safety benchmarks. It also wants companies to keep unsafe models out of app stores and to require verified identity for access to the most capable systems.
The study follows a smaller first release of the CT-AI benchmark, launched at the United Nations in July, which covered 27 models and found about a third of responses offered usable help beyond a simple web search. The new round expands the sample about fivefold and points the same way: stated intent still sways the answer, and stripped-down open models fail across the board.
Why it matters for Pakistan
The findings land as Pakistan sharpens its own cyber-enforcement machinery, with investigators and universities increasingly relying on AI tools for detection and defence. But the study is a reminder that the same models that help with research and coding can be repurposed for harm when their safeguards are removed, a risk that regulators and developers are still racing to contain.
Frequently asked questions
How many AI models were tested?
Tech Against Terrorism tested 134 leading large language models using its CT-AI benchmark, issuing 627 requests related to attack planning and weapons.
How many failed?
132 of the 134 models gave information useful for a mass-casualty attack or for building deadly weapons. Only two provisionally passed.
What is an ‘abliterated’ model?
An abliterated model is an AI system whose safety guardrails have been stripped away using publicly available tools, leaving it willing to answer requests its creators would have refused. All 13 abliterated models in the study failed the test.
What does Tech Against Terrorism recommend?
The group wants developers to safety-test models before release, governments to support independent benchmarks, and companies to keep unsafe models out of app stores and require verified identity for access.
