Could a chatbot turn into a weapon that finds previously unknown software bugs? According to Google’s Threat Intelligence Group, the answer is yes — and they believe they just prevented one of the first real-world examples.
Google published a report saying a criminal hacking group was preparing a "mass exploitation event" that relied on a zero-day vulnerability in an open‑source, web‑based system administration tool. The exploit would have let attackers bypass two‑factor authentication — the kind of failure that turns a single flaw into a wide‑reaching compromise. Google says it intervened and disrupted the operation, but the company also found signs in the exploit code that an AI model likely helped build it.
The clues that pointed to AI
Researchers at Google flagged stylistic and substantive markers inside a Python script used to weaponize the flaw. There were oddities like a "hallucinated" CVSS score — a made‑up severity rating presented as if authoritative — and structured, textbook formatting that reads like model training data. Those breadcrumbs, alongside the pace and scale of the attackers' workflow, gave Google "high confidence" that a large language model (LLM) supported both discovery and weaponization.
Importantly, Google said it does not believe its own Gemini model was used. Instead, the company described a broader ecosystem of criminal experimentation with off‑the‑shelf and viral tools — names like OpenClaw have circulated as explorers of agentic workflows that can be repurposed for illicit vulnerability hunting.
What the attackers were doing — and how AI helped
The vulnerability itself was a classic logic flaw: a developer hardcoded a trust assumption in a 2FA flow. An LLM can accelerate finding and exploiting these kinds of semantic bugs by rapidly scanning code, suggesting attack paths, drafting proof‑of‑concept scripts and iterating payloads in controlled environments to improve reliability. Google’s report also describes "persona‑driven jailbreaking," where adversaries prompt models to play the role of a security researcher to coax out technical advice that would otherwise be gated.
State‑linked groups and criminal gangs alike seem to be experimenting. Google and other observers named actors tied to China, North Korea and Russia as among those demonstrating growing interest in AI‑assisted vulnerability discovery. The speed at which these techniques went from theoretical to industrial‑scale surprised some: "There’s a misconception that the AI vulnerability race is imminent. The reality is it’s already begun," said John Hultquist of Google’s threat intelligence team.
Why this matters beyond one disrupted attack
Zero‑day bugs are valuable because they're unknown to the software maker and can be weaponized before a patch exists. Historically, finding such flaws required deep expertise and time. AI lowers that barrier: models can compress months of manual review into hours, find holes across massive codebases and generate working exploit code.
That flips the arms race. Defensive teams must now consider AI both as a tool for attackers and an asset for defenders. Companies and governments are scrambling — some models have been withheld from full release amid these concerns, and others are being distributed to vetted cybersecurity teams in limited previews.
Practical changes are already appearing. Big vendors are experimenting with detection and containment layers that lean on AI to spot anomalous behavior or to triage incidents faster. Google itself has been rolling AI features into services designed to detect ransomware and speed recovery; those same defensive techniques will matter more with AI‑augmented attackers. See how the company has been adding protections to its file services in recent updates like AI ransomware detection and recovery Google Drive adds AI ransomware detection and easier file recovery — here’s what it means.
On the product side, there's also talk of building constrained, enterprise‑grade agents that retain the productivity of agentic tools without the freewheeling risks of others. That conversation shows up in efforts to design safer, always‑on copilots that borrow OpenClaw‑style convenience while trying to avoid its pitfalls Microsoft Eyes an Always‑On Copilot — Think OpenClaw, but Safer.
Not all AI is the same — and regulation won't be a silver bullet
Google’s disclosure stopped short of naming the model used, but the episode complicates policy debates. Regulators and vendors are weighing whether to limit model capabilities, require security testing for release, or implement stricter access controls for models able to generate exploit code. Meanwhile, attackers will keep adapting, mixing open models, private fine‑tunes and simple automation to reach their goals.
Security researchers point out a mixed opportunity: the same LLM techniques can help defenders discover vulnerabilities, prioritize fixes and automate tedious testing. The difference will be governance — who gets access, under what controls, and how quickly fixes can be deployed once a flaw is found.
The incident is a reminder that AI’s arrival in cybersecurity isn’t a future threat; it’s happening now. For organizations that care about protecting accounts, data and infrastructure, the practical steps are mundane but urgent: patch quickly, assume 2FA implementations can be tested against logic errors, and invest in detection systems that can respond at AI speeds. The gap between attack and defense is narrowing — and the next exploit may not give defenders the same lucky break Google says it had this time.
