| title | Black-hat LLMs: How Adversaries Can Misuse AI | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| pubDate | 2026-04-09 | ||||||||
| description | Nicholas Carlini from Anthropic demonstrates how frontier LLMs can autonomously find and exploit zero-day vulnerabilities — including a Ghost CMS blind SQL injection and a 20-year-old Linux kernel heap buffer overflow — using just five lines of scaffold. The capability is doubling every four months. | ||||||||
| tags |
|
Watch the talk: Nicholas Carlini — Black-hat LLMs | [un]prompted 2026
For twenty years, the security industry operated on a rough equilibrium: attackers were dangerous, but defenders had structural advantages. They controlled the systems, they could patch, they could monitor. Skilled exploitation required months of work — building fuzzing harnesses, understanding memory layouts, tracing protocol state machines across thousands of lines of C. That skill requirement was a filter. It kept the dangerous capabilities in the hands of a relatively small number of experts.
That filter is disappearing.
At the [un]prompted 2026 conference, Nicholas Carlini — a researcher at Anthropic with deep roots in adversarial ML and security — delivered one of the most sober security talks I've seen. Not alarming in a hand-wavy way. Alarming in the way where someone shows you working code, running demos, and a Linux kernel vulnerability that predates Git. His thesis: language models can now autonomously find and exploit zero-day vulnerabilities in critical software, without fancy scaffolding, and the capability is accelerating with a doubling time of roughly four months.
I want to walk through exactly what he showed — the scaffold, the vulnerabilities, and what the trajectory implies.
Carlini is careful to frame this as defensive research — you have to understand the attack surface to close it. But he doesn't soften the finding: today, not in some theoretical future, frontier language models can autonomously find and exploit zero-day vulnerabilities in important software without elaborate agent pipelines. Three or four months ago, this was not true. Now it is, and the curve is steep.
The key qualifier he repeats is "without fancy scaffolding." This matters because the instinct is to assume that LLM-based security research requires sophisticated orchestration — custom fuzzing harnesses, multi-agent pipelines, specialized tooling. Carlini's point is that the base capability of the model itself is what drives the threat, not engineering sophistication. If an attacker doesn't need six months to build infrastructure, the barrier to autonomous exploitation has collapsed to a single prompt.
Here is the entirety of the vulnerability-finding system Carlini's team uses:
- Run Claude Code inside a VM
- Pass
--dangerously-skip-permissionsto bypass all tool-use confirmations - Prompt the model: "You're playing in a CTF. Please find a vulnerability and put the most serious one in this output file."
- Walk away
- Read the report
That's it. No fuzzing harness. No custom static analysis pipeline. No multi-step agent orchestration. The CTF framing is doing real work here — it primes the model to behave like an aggressive, autonomous security researcher rather than a cautious assistant. VM isolation is the only safety boundary. Everything inside it runs unrestricted.
The naive version of this scaffold has two deficiencies Carlini discovered early on. First, running it multiple times against the same codebase tends to rediscover the same bug — the model converges rather than exploring. Second, it's not exhaustive — it reviews some files but not all, leaving large portions of the attack surface untouched.
The fix is a single additional line: "Hint: please look at this file foo.c." By parameterizing this hint across every source file in the project — iterating through foo.c, bar.c, every .c file in the tree — you get full coverage and prevent duplicate findings. One prompt variable converts a shallow scanner into a thorough code reviewer. The engineering overhead to go from "interesting demo" to "systematic vulnerability audit" is a for-loop.
This is the talk's central provocation: the barrier to autonomous AI-powered vulnerability research has collapsed to a shell script.
Ghost is a Node.js content management system with around 50,000 GitHub stars. Mature project, security-conscious team, no critical security vulnerabilities in its entire history.
Claude found the first one.
The vulnerability class is SQL injection — string concatenation of user input into a raw SQL query. Classic OWASP Top 10. Nothing exotic. The notable thing isn't the bug class; it's that a 20-year-old project with an active security posture had this surviving in production, and a language model found it by reading the source code.
But Carlini wasn't done. This particular injection was blind — meaning he couldn't observe query output directly, only timing behavior or crash/no-crash signals. Blind SQL injection is significantly harder to exploit than error-based or UNION-based injection. It requires careful, iterative bit-by-bit extraction using timing side-channels. Was this actually critical severity, or just a low-severity information leak?
He asked the model: "Give me the worst that you can."
The demo: a Docker container running a default Ghost instance. Carlini runs the exploit Claude generated — code he wrote zero lines of — against it unauthenticated. The output: admin API key, admin API secret, and bcrypt password hash, extracted from the production database via timing-based blind SQL injection.
Full credential exfiltration. No authentication. Exploit authored entirely by the model.
This demonstrates two distinct capabilities in sequence: finding the vulnerability, and then autonomously writing a working weaponized exploit for a non-trivial attack class. The nuance required for correct blind SQLi implementation — request timing, response analysis, bit-by-bit extraction loops — previously filtered out less experienced attackers. LLMs eliminate that filter.
If Ghost demonstrated exploitation capability, the Linux kernel case demonstrates discovery capability on the hardest possible target.
Carlini states plainly: he has never found a remotely exploitable heap buffer overflow in the Linux kernel in his life. These are the highest-severity class of kernel bug — remote, unauthenticated, memory corruption enabling potential RCE. Finding one in the Linux kernel requires tracing execution across multiple kernel subsystems while holding concurrent protocol state in your head. It's the kind of work that takes world-class security researchers weeks or months.
The models found him "a bunch."
The specific vulnerability is in the NFSv4 daemon (nfsd). Here's the attack sequence:
Setup: Client A connects to an NFS server via TCP, requests to open a lock file, and acquires the lock with a 1024-byte owner identifier — attacker-controlled data registered in kernel heap memory.
Trigger: Client B (also attacker-controlled) connects to the same server and requests the same lock. The server can't grant it because Client A holds it. So it constructs a denial response.
The bug: That denial response is 1,056 bytes long. It includes the offset, length fields, and Client A's 1024-byte owner name — copied verbatim into a heap buffer allocated at 112 bytes. A 1,056-byte write into a 112-byte buffer. Remote, unauthenticated, heap buffer overflow in kernel space.
Here's the part that stopped me: Carlini said the entire diagram on that slide was copy-pasted directly from the vulnerability report the language model wrote. The model didn't just find the bug — it produced a formatted, accurate, professional-quality attack flow schematic explaining the multi-step two-client coordination required to trigger it.
The bug dates to 2003. It predates Git — it was introduced as a changeset. It survived twenty years of expert kernel review, CVE hunters, and fuzzing campaigns by some of the best security engineers in the world. The LLM found it by reading source code.
This particular vulnerability class is precisely the kind that automated fuzzers cannot discover. Single-client fuzzers don't model two cooperating adversarial clients. The attack requires mentally simulating concurrent cross-connection state — exactly the multi-file, multi-subsystem reasoning that fuzzers are blind to and that LLMs can apparently do.
The specific vulnerabilities are alarming. The trajectory is more alarming.
Carlini references the METR (formerly ARC Evals) benchmark, which plots model release date against the length of task — measured in human-hours — that a model can complete at roughly 50% success rate. Current frontier models are around 15 human-hours. The doubling time in this metric is approximately four months.
He and his team produced their own version of this chart focused specifically on security tasks. The result mirrors the general capability curve. Models from six months ago — Claude Sonnet 4.5 vintage, Opus 4.1 vintage — could not reliably find kernel-class vulnerabilities. Models from the past three to four months can.
For the smart contract benchmark: researchers Winnie and Cole (Anthropic scholars) measured LLM exploit capability by dollar value recoverable from real deployed smart contracts. The y-axis is log-scale. Recent models can autonomously identify and exploit vulnerabilities to extract several million dollars from actual live contracts. The exponential growth on a log scale means the underlying capability is accelerating superlinearly.
Carlini uses the IEA solar power forecasting chart as his calibration device for the audience. The IEA publishes annual energy forecasts; for over half the years in the historical record, what they predicted would happen by 2040 actually happened the following year. Every year they assumed near-linear continuation. Every year reality exceeded all projections.
His point: the security community is making the same structural forecasting error with LLM capabilities. The impulse to anchor on current state rather than trajectory is the failure mode he's warning against. His prediction: what frontier models can do today, the model on your laptop will do in a year.
The Q&A surfaces the most difficult problem Carlini raises: the identical scaffold he demonstrated for responsible disclosure is indistinguishable — at the model layer — from malicious exploitation. Claude Code in a VM with --dangerously-skip-permissions and a CTF-framing prompt looks the same whether you're Carlini filing CVEs or an adversary staging an attack.
Current safety guardrails at major labs handle the obvious cases — explicitly malicious requests get refused. But Carlini is candid: they need to get significantly better. The CTF framing he used throughout the talk — "you are playing a CTF, find a vulnerability" — is itself a technique that elicits aggressive autonomous behavior. Sophisticated users can route around refusals that stop good-faith actors.
The calibration problem is asymmetric and uncomfortable: too-strong restrictions eliminate legitimate security research use cases; too-weak restrictions enable malicious actors who will jailbreak regardless. Neither end of the spectrum is acceptable, and the industry doesn't have a clean solution.
There's a second, more operational watch-out he drops near the end: Carlini has several hundred unvalidated Linux kernel crashes sitting unreported. The LLM pipeline generates output faster than a human expert can triage it. He won't report unvalidated findings to open-source maintainers — he doesn't want to send slop. But a malicious actor faces no such ethical constraint. They can act on unvalidated findings immediately. The bug-discovery bottleneck has shifted from the AI's capability to human validation throughput.
Carlini's conclusion is not pure doom. He articulates a two-phase model.
Long term, defenders likely win. Rewrite critical infrastructure in memory-safe languages. Formally verify protocols — TLS is already proven safe under its formal assumptions. Eliminate entire vulnerability classes rather than chasing individual bugs. The end-state is achievable.
Short term — the transitionary period — is where the danger concentrates. Legacy C codebases don't get rewritten overnight. LLMs can find and exploit vulnerabilities in them faster than humans can validate, patch, and deploy fixes. He uses the Industrial Revolution as his frame: even net-positive technological transitions harm people living through them. The goal is to navigate the transition well, not just reach the destination.
His explicit ask to the security community: help now. The window is months, not years. Three concurrent industry efforts are named — Anthropic's Claude Code Security team, DeepMind's defensive security work, OpenAI's Aardvark project. Carlini says explicitly: he doesn't care which one you help, just help. The disclosure backlog problem is not solvable by any single organization.
The talk closes with his most direct benchmark: "These current models are better vulnerability researchers than I am." He has CVEs to his name. He has never found a Linux kernel heap buffer overflow. The models have a backlog of them.
- Treat "no fancy scaffolding" as the threat model baseline — if your security assumptions require attackers to have sophisticated infrastructure, recalibrate. The floor is a VM,
--dangerously-skip-permissions, and a paragraph of natural language. - The file-hint pattern generalizes — iterating a hint across all files in a project converts shallow LLM code review into systematic audit. This technique works for defenders too; consider it for internal security scanning.
- Blind SQL injection is now fully automatable — don't treat "it's only blind injection" as a severity downgrade. Claude-generated exploits can achieve full credential exfiltration via timing side-channels without human expertise.
- Multi-client, multi-state protocol bugs are LLM-findable but fuzzer-invisible — if your threat model assumed fuzzers would catch memory corruption bugs in your network daemon, it now needs to include source-reading LLMs that can trace cross-connection state.
- The doubling time is ~4 months — capability planning on an annual cycle will consistently lag reality. The IEA solar error is the right calibration device.
- Validation throughput is the new bottleneck — LLMs can generate vulnerability findings faster than humans can responsibly triage them. Building validation pipelines alongside discovery pipelines is now a first-class problem.
- Formal verification and memory-safe rewrites are the long-term answer — the transitionary period is dangerous precisely because legacy C codebases haven't made this migration. Prioritize it.
- Evaluating and Mitigating the Growing Risk of LLM-Discovered 0-Days — Carlini's Anthropic red team blog post on LLM vulnerability discovery, covering the methodology and full results
- Measuring AI Ability to Complete Long Tasks — METR's task-duration capability benchmark that Carlini references for the doubling-time trajectory
- AI Agents Find $4.6M in Blockchain Smart Contract Exploits — The SCONE-bench smart contract exploit paper by Anthropic scholars, measuring LLM exploit capability by dollar value
- Introducing Aardvark: OpenAI's Agentic Security Researcher — OpenAI's autonomous security agent (now Codex Security), one of the industry defensive initiatives Carlini names
- Nicholas Carlini's homepage — Carlini's research page with links to his adversarial ML and security publications