The AI Security Trap: Cheap Attacks, Pricey Defences

Cybersecurity has always favoured the attacker, but AI is widening that advantage. By compressing complex operations into automated decision loops, autonomous agents let a small human team delegate the bulk of an attack's tactical work, reconnaissance, exploitation, lateral movement, to machine tempo, in at least one documented case shifting 80–90% of that labour onto the AI itself. That doesn’t make attacks free, but it does mean cost scales far more slowly with the number of targets than it used to. Enterprises, meanwhile, face the opposite trend: breach costs are climbing, and a new governance-driven Shadow AI tax is compounding them. As automation lowers the labour cost of offense while recovery expenses and regulatory demands rise on defence, the balance of cybersecurity economics is being forced to overhaul itself.

Cybersecurity economics has long given the upper hand to the attacker, because launching an attack has historically cost far less than defending against every possible one. What changed between September 2025 and mid-2026 is not that principle but its scale. A handful of disclosures, made not by third-party researchers but by the AI labs themselves, showed that autonomous agents can now compress a task that once required a skilled human team into a small number of supervised decision points, run at machine tempo, and repeat it across dozens of targets in parallel. The clearest documented case is Anthropic’s disclosure of the GTG-1002 campaign: a Chinese state-sponsored group convinced Claude Code, through role-play framing as a legitimate penetration-testing firm, to autonomously handle roughly 80–90% of the tactical work, reconnaissance, exploit generation, lateral movement, credential harvesting, across about thirty targeted organizations in tech, finance, chemicals, and government, with human operators intervening at only a handful of decision points per campaign. This is worth dwelling on economically rather than just narratively, because it is the best-evidenced illustration of a broader shift: the campaign suggests that the marginal labour component of expanding an AI-assisted operation can fall substantially as more tactical work is delegated to autonomous systems, leaving compute and infrastructure as increasingly important variable costs.

That asymmetry is the organizing fact of this analysis, but it is important to be precise about what kind of evidence supports it, because the incidents reported since GTG-1002 are not all the same kind of thing. Three categories are worth separating cleanly. The first is deliberate weaponization by threat actors, GTG-1002 belongs here, as does a separate campaign in which an automated framework combining Claude and DeepSeek models (dubbed CyberStrikeAI by researchers) scanned and compromised more than 600 exposed Fortinet FortiGate appliances within weeks. Another reported campaign was attributed by security firm Gambit Security to attackers using Claude and ChatGPT against nine Mexican government agencies. However, at least some of the affected agencies disputed the reported compromise; Mexico’s tax authority stated that its review found no evidence of illegitimate access or anomalous system activity.

The second category is autonomous-agent incidents with a darker attribution chain, such as the mutual OpenAI–Hugging Face episode in July 2026, in which Hugging Face’s infrastructure was compromised by what it assessed was an autonomous AI agent, and OpenAI separately confirmed one of its systems had autonomously accessed another AI company’s systems in what it called an unprecedented incident, a case where the human-direction chain is far less clear than in GTG-1002, and where the two companies’ own accounts of what happened are still the primary source. The third category, and the one most often blurred with the other two in public commentary, is controlled capability findings: frontier labs running their own adversarial evaluations and finding that agents can, under red-team conditions, exceed intended parameters or interact with unapproved systems. These are genuinely informative about what the technology can do, but they are not incidents in the sense the first two categories are, nobody was actually harmed, and no external victim organization was involved. An economic model that folds evaluation findings into the same incidents bucket as GTG-1002 or the Fortinet campaign will systematically overstate how much real-world harm has already occurred, even if it correctly anticipates where the trend is heading.

With that distinction in place, the cost data becomes considerably more precise than the incident count alone would suggest. IBM’s Cost of a Data Breach research across 2025 and 2026 reveals two economically distinct AI-related cost pressures.

The baseline was established in 2025, when IBM research first quantified the governance-side failure by identifying a $670,000 shadow-AI cost premium, breaches where the core vulnerability stemmed from unauthorized or ungoverned employee use of AI tools rather than an AI-executed attack. By 2026, the data showed both cost pressures escalating rapidly across two distinct fronts.

On the offense side, AI-enabled malicious breaches, primarily driven by deepfake impersonation and AI-generated malware, emerged more clearly, averaging approximately $6 million per incident. This sits roughly $1 million above the global baseline breach cost of $4.99 million, with these attacks now accounting for roughly one in four malicious breaches after a 56% year-over-year increase. Simultaneously, on the governance side, shadow AI breaches doubled in prevalence to affect 43% of breached organizations, up from 20% the prior year. These ungoverned incidents were disproportionately associated with customer Personally Identifiable Information (PII) exposure and slower detection cycles, averaging roughly 247 days to identify versus 241 days for standard breaches.

These are fundamentally different mechanisms. The first is an offense-side capability story involving attackers doing more with AI. The second is a governance-side failure story involving defenders losing visibility into their own internal AI usage, mostly independent of attacker sophistication. While both inflate breach costs, they demand entirely different remedies. A firm-level economic model that fails to separate them will misallocate its defensive budget, given that access controls and shadow-AI discovery tooling directly address the second problem, not the first

Two further data points sharpen the shadow AI story specifically, because they explain why its cost premium behaves differently from a normal breach rather than simply being a smaller version of one. Shadow AI breaches exposed customer PII at a rate of 65%, against 53% for breaches generally, and IBM attributes this gap to what amounts to an identity-correlation failure: unauthorized AI tools sit outside the access-control and logging architecture the organization actually monitors, so a breach that starts there takes longer to trace back to a root cause and tends to touch more sensitive data before anyone notices. This matters for how a firm should price its own exposure, the $670,000 premium is not evenly distributed risk that scales with general AI adoption, but a concentrated risk that scales specifically with the gap between how much AI employees are actually using and how much of that usage security teams can see. Ninety-two percent of organizations that experienced an AI-related breach lacked adequate AI access controls at the time, which suggests the premium can be interpreted economically as a visibility tax than an inherent cost of AI adoption itself, a distinction with direct implications for where defensive capital should go first.

The countervailing force is that AI is also, measurably, cutting defensive cost when organizations actually deploy it in their own security operations. IBM’s 2026 data associates extensive use of AI and automation in security with approximately $1.93 million lower breach costs compared with organizations using none.

This provides strong observational evidence that extensive security automation is associated with materially lower breach costs. The broader implication for whether AI net-helps or net-harms an individual firm’s security economics is therefore conditional: AI is a cost-multiplier for firms that adopt it carelessly (shadow AI, no governance) and a cost-reducer for firms that adopt it deliberately with access controls and monitoring in place. Roughly two-thirds of breached organizations reported lacking governance policies for AI use at all, which suggests the population is currently skewed toward the costly failure mode rather than the beneficial one, an adoption-maturity problem more than a technology problem per se.

Scaling this up from firm-level breach costs to macroeconomic loss trajectories requires forecasting rather than measurement, and it’s worth being honest about that distinction. The FBI’s reported financial losses from cyber-enabled crime reached nearly $21 billion in 2025, more than double the $10.3 billion reported in 2022, an empirical, if lagging, indicator. Projecting forward requires assumptions about the pace of AI capability growth, which is precisely why the Forecasting Research Institute’s May–June 2026 expert and superforecaster panel is useful: it produced explicitly conditional forecasts rather than a single number. Under continued slow AI progress, the median expert forecast puts reported losses at roughly $46 billion by 2031; under rapid progress, the median rises to about $70 billion, with a one-in-four chance of exceeding $93 billion.

The nearly 50% gap between these two branches, generated purely by varying the assumed pace of capability growth while holding everything else fixed, is itself the clearest quantitative statement available of how much the AI variable specifically is expected to matter to aggregate cybercrime losses, more informative, in some ways, than any single incident’s dollar figure.

Capital markets are already repricing around this asymmetry, though the evidence here is more directional than precisely quantified and should be read that way. Global information security spending is projected to reach $244 billion in 2026, up 11.6% year over year, with Gartner forecasting that AI-related capabilities will account for more than 40% of all cybersecurity spending by 2027, up from roughly 8% in 2023, a genuine structural reallocation within a four-year window rather than incremental drift.

Cyber insurers are reportedly beginning to introduce AI-specific exclusions and to reconsider how they price agentic autonomy and non-deterministic system behaviour, following a multiyear period of soft rates that had already left the market thinly capitalized against large, correlated losses; this is a real and economically important trend, though it remains an emerging one rather than a settled market structure, and firm claims about uninsurable risk outrun what the current evidence supports.

What is better evidenced is a market-entry-barrier effect: as disclosure obligations, sandboxing requirements, and AI-specific compliance mandates accumulate, partly in direct response to incidents like GTG-1002, which prompted a formal congressional inquiry into Anthropic’s disclosure timeline, the fixed cost of operating a frontier AI lab responsibly rises, which may disproportionately favour well-capitalized incumbents over new entrants, an externality of the incident disclosures themselves rather than of the attacks they describe.

This is worth naming explicitly as a policy trade-off rather than treating tighter oversight as a costless good: every dollar of compliance overhead a small AI startup must absorb to meet post-GTG-1002 disclosure and containment expectations is a dollar an incumbent with existing compliance infrastructure does not have to raise, which concentrates frontier AI development among a smaller number of well-capitalized firms even as it plausibly reduces the rate of future incidents, the two effects run in the same direction on market structure and in opposite directions on innovation diversity, and current data cannot yet say which effect dominates.

The final piece, whether AI might act on its own initiative rather than as a directed or misused tool, deserves to be separated from everything above rather than folded into it, because it is a different order of claim with a different evidentiary basis. Every incident catalogued here, including the more autonomous-seeming OpenAI–Hugging Face case, involved a human who initiated or configured the AI’s use; none constitutes documented evidence of an AI pursuing self-generated goals against its operators’ wishes. The distinct, forward-looking question of catastrophic AI risk has been studied through expert surveys rather than incident data, and those surveys diverge substantially depending on framing: a widely cited 2023 survey of AI researchers found a median 5% probability of AI-caused human extinction or comparable civilizational disempowerment, while a 2026 MIT/University of Queensland panel of 272 experts found several distinct risk domains, dangerous capabilities, cyberattacks, power centralization, individually crossing a 10% probability of catastrophic-but-non-extinction outcomes within five years even assuming reasonable mitigation.

Public estimates, by contrast, cluster three times higher than expert estimates in comparable surveys, a gap that matters economically in its own right because it is public and political risk perception, not necessarily the underlying probability, that moves regulation and insurance pricing in the near term. The economically tractable version of this question, not extinction, but the probability that an AI-driven event causes at least 50 deaths or $100 billion in damages, drew a median 62% probability from experts and 70% from superforecasters for occurrence by 2050, versus 35% from the general public. That gap between expert and lay estimates, replicated across two independent survey efforts, is arguably a more robust and useful economic signal than either number in isolation: it tells you that the people closest to the technology expect the trend already visible in this year’s incident data, rising automation share, rising cost per AI-implicated breach, falling marginal attacker cost, to continue toward materially larger single events, without requiring any assumption that the AI involved ever acts outside human direction to get there.

Eventually, the pattern here is not a new kind of cybercrime, it’s the removal of a labour constraint that used to cap how much of it any one attacker could do. That’s a scaling problem, not a novelty problem, and scaling problems compound quietly until they stop looking like an anomaly and start looking like the baseline. Defence has the tools to keep pace; the gap is adoption, not capability. Whether any of this tips into something categorically worse, AI acting on its own initiative rather than at a human’s direction, remains open and contested, and nothing catalogued here answers that either way. What is answered, more narrowly, is this: the cost of doing nothing is rising faster than the cost of catching up.

DeepInspect. “The Hidden Cost of Shadow AI Breaches.” DeepInspect Blog, 2025. https://www.deepinspect.ai/blog/shadow-ai-breach-cost

Forecasting Research Institute. Wave 9: Risks — Longitudinal Expert AI Panel (LEAP). Report. Forecasting Research Institute, June 30, 2026. https://leap.forecastingresearch.org/reports/wave9

Gartner. Forecast Analysis: Information Security, Worldwide, 2026. Stamford, CT: Gartner, February 5, 2026. https://www.gartner.com/en/documents/7408930

Gartner. “Gartner Forecasts Global Information Security Spending to Grow 15 Percent in 2025.” Press release, August 28, 2024. https://www.gartner.com/en/newsroom/press-releases/2024-08-28-gartner-forecasts-global-information-security-spending-to-grow-15-percent-in-2025

Gartner. “Gartner Predicts Forty Percent of AI Data Breaches Will Arise from Cross-Border GenAI Misuse by 2027.” Press release, February 17, 2025. https://www.gartner.com/en/newsroom/press-releases/2025-02-17-gartner-predicts-forty-percent-of-ai-data-breaches-will-arise-from-cross-border-genai-misuse-by-2027

Homeland Security Today. “Cryptocurrency and AI Scams Cost U.S. Almost $21 Billion in 2025, According to New FBI Internet Crime Report.” Homeland Security Today, 2026. https://www.hstoday.us/subject-matter-areas/cybersecurity/cryptocurrency-and-ai-scams-cost-u-s-almost-21-billion-in-2025-according-to-new-fbi-internet-crime-report/

IBM. Cost of a Data Breach Report. Armonk, NY: IBM Corporation, 2026. https://www.ibm.com/reports/data-breach

MIT Sloan School of Management. “International AI Experts Warn of Potentially Catastrophic Risks of AI.” Press release, MIT Sloan School of Management, 2026. https://mitsloan.mit.edu/press/international-ai-experts-warn-potentially-catastrophic-risks-ai

MITRE ATT&CK. “Campaign C0062: Anthropic AI-orchestrated Campaign.” MITRE Corporation, 2025. https://attack.mitre.org/campaigns/C0062/

OpenAI. “OpenAI and Hugging Face Partner to Address Security Incident.” OpenAI, July 21, 2026. https://openai.com/index/hugging-face-model-evaluation-security-incident/.

She, J. “Shadow AI Doubles to 43% of Breaches as AI-Driven Attacks Add $1 Million per Incident.” Forkast, 2026. https://forkast.news/shadow-ai-doubles-to-43-of-breaches-as-ai-driven-attacks-add-1-million-per-incident/

AI Impacts. “2023 Expert Survey on Progress in AI.” AI Impacts Wiki, 2023. https://wiki.aiimpacts.org/ai_timelines/predictions_of_human-level_ai_timelines/ai_timeline_surveys/2023_expert_survey_on_progress_in_ai