Everything About AI
NextFin News - In July, an autonomous AI agent broke into one of the technology industry's most important platforms without a human hand on the keyboard. It found a code-execution path in a dataset loader, escalated to node-level access, harvested cloud credentials, and moved laterally through internal clusters over a weekend, executing thousands of actions across a swarm of short-lived sandboxes. The company that disclosed the intrusion, Hugging Face, said its own forensic analysis was blocked by the safety guardrails of the frontier models it first tried, forcing it to run an open-weight model on its own infrastructure to keep pace with the attacker. Its conclusion was blunt: "Autonomous, AI-driven offensive tooling is no longer theoretical."
That incident, disclosed in the same month that Anthropic reported intervening in multiple attempts by scientists and other actors to use its models to help develop biological weapons, marks the point at which artificial-intelligence risk stopped being a philosophical debate and became an operational, financial, and regulatory fact. Yet the market continues to price AI almost exclusively as a revenue story. The gap between the two is where the risk now sits.
The New Baseline: Three Developments That Changed the Threat Model
Three developments in 2026, taken together, define the new baseline.
First, the threat model changed. The Hugging Face intrusion was not a jailbroken chatbot or a prompt-injection parlor trick. It was an end-to-end autonomous agent campaign that treated the data and model surface as a first-class attack surface, running "many thousands of individual actions across a swarm of short-lived sandboxes, with self-migrating command-and-control staged on public services." More than 17,000 attacker events were recorded. The defender's dilemma it exposed is structural: the attacker was bound by no usage policy, while the defender's access to the most capable hosted models was blocked by the very safety systems designed to prevent misuse.
Second, the misuse cases moved up the harm ladder. Anthropic's misuse report, released September 10, covered December 2025 to August 2026 and cited biological misuse as "one of the most serious risks of frontier AI models." The company said it highlighted five case studies of actors using its models in ways that could support biological weapons development, and disrupted activity across six other harm areas: cyber operations, influence operations, surveillance, scams and fraud, conventional weapons development, and illicit distillation — the unauthorized replication of one model's capabilities into another. Notably, none of the misuse cases involved the company's most powerful Mythos-class models, with the exception of one distillation instance. The implication is uncomfortable: the models already in wide circulation are capable enough to be dangerous in the wrong hands.
Third, the people closest to the technology are walking away. In September, a researcher who said he spent three years at Anthropic and OpenAI resigned with a warning that "Neither Anthropic nor OpenAI is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." In February, Anthropic's safety lead, Mrinank Sharma, resigned with a letter stating "the world is in peril." Evan Hubinger, Anthropic's alignment-science lead, responded publicly that he personally assesses a greater than 10 percent chance that AI could kill all humans within the next decade, adding that the company is "trying its best, but we do not yet have a plan to solve alignment for superintelligence." When safety staff inside frontier labs conclude that caution is losing to competition, that is a governance signal, not a publicity stunt.
Against this backdrop, the money keeps flowing. The Bank for International Settlements, in its Annual Economic Report 2026, warned that the five largest hyperscalers are on pace to spend more than $1 trillion on AI-related capital expenditure across 2025 and 2026 combined — a sum already outpacing their earnings and free cash flow, forcing some to issue debt to cover the gap. Goldman Sachs Research estimated global AI investment in 2026 at approximately $1.019 trillion, with about $581 billion in the United States. Anthropic itself is preparing an initial public offering that backers say could value the company at $2 trillion or higher, after reporting second-quarter revenue that would more than double to around $10.9 billion and its first operating profit.
The question this piece pursues is not whether AI will keep advancing — it will. It is whether the market, regulators, and corporate boards have correctly priced a technology whose risk profile has structurally shifted while its governance remains largely voluntary.
The Mechanism: Why This Time Is Different
The first-order read of the Hugging Face incident is simple: AI makes attacks cheaper and faster. The second-order point is more important. Autonomous tooling inverts the traditional economics of defense. A human attacker is limited by time, skill, and the need to sleep; an agent swarm is not. It can run thousands of actions, migrate its command-and-control across public services, and operate at machine speed across a weekend while defenders work business hours. Hugging Face matched that speed only by deploying its own AI analysis agents over the full attacker log — doing in hours what would usually take days.
This creates an asymmetry that compounds. The attacker needs to succeed once; the defender must succeed every time. And the defender's best tools — the most capable frontier models — come with safety guardrails that refuse to process real exploit payloads and attack commands, because they cannot distinguish an incident responder from an attacker. The attacker, using a jailbroken hosted model or an unrestricted open-weight one, faces no such constraint. The result is a structural advantage for offense that no amount of patching fully closes.
The same asymmetry runs through the biological-risk cases. The International AI Safety Report 2026 — a scientific assessment guided by more than 100 experts from over 30 countries and published in February — found that general-purpose AI can produce laboratory instructions, troubleshoot experimental procedures, and answer technical questions, reducing barriers to harmful workflows. The report is careful to note uncertainty about how much these capabilities increase real-world risk given practical barriers to producing weapons. But Anthropic's five biological case studies show the pathway is no longer hypothetical: actors are already attempting to walk it.
This is the mechanism behind the cyclical-versus-structural call. A cyclical risk is one that mean-reverts: a bad earnings quarter, a contained breach, a model hallucination caught in testing. What is unfolding in AI is structural. Three features will not self-correct on their own. First, capability is a one-way ratchet — computing power has grown exponentially since 2012, with the largest training runs likely surpassing 10^26 FLOP in 2025, and there is no technical path by which a model unlearns what it knows. Second, the dual-use nature of the technology means the same capability that designs a protein therapy can design a pathogen; you cannot remove one without removing the other. Third, the deployment model — open weights, APIs, agents with tool access — puts powerful capability in the hands of actors whose incentives are not aligned with safety. None of these reverts.
The Governance Gap: Voluntary Regimes Meet Mandatory Consequences
If the risk is structural, the regulatory response is not keeping pace. The United States' latest framework, Executive Order 14409 signed June 2, 2026 — "Promoting Advanced Artificial Intelligence Innovation and Security" — directs agencies to develop a classified benchmarking process to assess the advanced cyber capabilities of AI models and designate "covered frontier models." It explicitly does not impose licensing or preclearance. In the European Union, the AI Act's transparency rules take effect in August 2026 and a further prohibition in December 2026, with a call to increase model-evaluation capacity expected to be operational by 2027.
The pattern is consistent: rules arrive after capability, and the most consequential frontier models sit in a voluntary zone. The June executive order signals recognition that frontier models with advanced cyber capabilities may require closer public-private coordination — but coordination is not the same as constraint. Meanwhile, the International AI Safety Report reiterates that although developers have made it more difficult to bypass model safeguards, new attack techniques are constantly being developed and attackers still succeed at a moderately high rate. Its prescription is "defense-in-depth" — layering evaluations, technical safeguards, monitoring, and incident response so that no single failure leads to significant harm. Sensible, but voluntary, and dependent on the very companies racing to ship.
The consequence is that AI risk is being reclassified as fiduciary risk by other routes. In the United States, the Securities and Exchange Commission has made clear that the use of advanced analytics or AI does not exempt firms from existing obligations related to fiduciary duty, disclosure, and market integrity. The message across jurisdictions is consistent: lack of legal clarity does not equal lack of responsibility. Boards and senior executives are expected to understand where AI is used, what decisions it influences, and what controls exist to intervene when things go wrong. That is a material shift: AI safety is becoming a directors'-and-officers' liability question, not a research-lab one.
The Market Is Pricing Upside, Not Tail Risk
Here is where the financial story sharpens. The AI investment boom is being underwritten by a specific assumption: that spending of more than $1 trillion across two years will compound into durable productivity and earnings. The Bank for International Settlements does not call the boom a bubble outright. Its concern is different and more subtle — that a major repricing of AI-related stocks "could have more pronounced wealth effects and sharper consumption pullback than in the past," and that the U.S. market's global footprint would propagate the destruction internationally. The five largest hyperscalers are spending ahead of earnings and free cash flow, issuing debt to cover the gap. That is a financing structure that depends on continuous confidence.
Consider the valuation arithmetic at the frontier. Anthropic's last hard mark was a $965 billion post-money valuation on its $65 billion Series H round; backers now discuss a $2 trillion IPO target. Nvidia, the picks-and-shovels beneficiary, trades at a market capitalization above $5 trillion on fiscal net income of $120.1 billion from revenue of $215.9 billion. These prices embed a world in which capability keeps scaling, demand keeps compounding, and nothing materially interrupts the build-out.
The tail risk the market is not pricing is not that AI fails technically. It is that AI succeeds in ways that trigger a governance or incident shock severe enough to force a repricing of the entire complex: a biological near-miss traced to a frontier model, an autonomous agent incident with physical-world harm, or a regulatory intervention that constrains deployment just as debt maturities come due. The Bank for International Settlements frames the transmission channel precisely: if inflation proves stickier than expected and central banks are forced to raise rates, the same tightening needed to contain macro risks "could be what pops the AI-financed debt bubble." Add an AI-specific shock to that rate path, and the wealth effects compound.
This is the second-order implication most investors are missing. The conventional wisdom is that AI risk is a problem for the companies building models — a reputational or compliance cost line. The less-priced view is that AI risk is a financing-risk multiplier for the entire complex. When capital expenditure is debt-funded and confidence-funded, anything that shakes confidence raises the cost of capital for everyone building on the same assumptions. The Hugging Face incident, the Anthropic misuse cases, and the safety resignations are not isolated news items; they are data points that the probability of such a shock is higher than the last time the market marked these assets.
The Strongest Counter-Thesis — and Why It Only Partly Holds
The bullish counter-argument is serious and deserves its due. AI safety is improving, not deteriorating. The International AI Safety Report notes that since the previous report in January 2025, AI systems have generally become more reliable and have seen greatly expanded deployment as a result. Developers have made it more difficult to bypass safeguards. Anthropic's own report is evidence that monitoring works: the company detected and disrupted misuse before it caused harm, across biological, cyber, and influence domains. The resignations of a handful of researchers, on this view, reflect the preferences of a safety-maximizing minority, not the judgment of an industry that is collectively reckless. And the economic payoff — if even a fraction of the promised productivity materializes — would dwarf the costs of caution.
There is truth here. Defense-in-depth works when it is funded and layered. Incident response is improving. And it is entirely possible that the worst-case scenarios remain just that — scenarios.
But the counter-thesis rests on a premise the evidence is eroding: that capability growth and safety growth move in tandem. The Hugging Face intrusion shows the opposite — the attacker's capability (autonomous, multi-stage, machine-speed) outran the defender's governed capability (guardrail-blocked forensic analysis). Anthropic's report shows misuse attempts climbing the harm ladder even as safeguards held in these specific cases. The safety report itself warns that no combination of current methods eliminates failures entirely, and that AI agents compound reliability risks because they operate with greater autonomy, leaving humans fewer chances to intervene.
The decisive point is asymmetry again. Safety must succeed every time; an attacker, or a malfunction, needs to succeed once. A system that is "generally more reliable" can still produce a catastrophic outlier — and in a debt-financed, confidence-dependent market, the outlier is what reprices the complex. The counter-thesis is right that the industry is not helpless; it is wrong to conclude that the trend is sufficient.
So what would prove this judgment wrong? One specific, observable signal: if the next two annual cycles pass with no frontier-model-linked incident causing physical-world harm, no successful large-scale autonomous-agent breach of critical infrastructure, and no regulatory intervention that constrains deployment — while hyperscaler AI revenue grows faster than AI-related capital expenditure — then the structural-risk premium argued here should compress, and the market's current pricing would be vindicated. Conversely, any one of those events printing true would confirm that the risk is being underpriced.
Conclusion: Who Benefits, Who Is Exposed, and What to Watch
The mechanism cashes out into a concrete asymmetry. The beneficiaries of taking AI risk seriously are the operators who build defense-in-depth before an incident, keep a capable model on their own infrastructure vetted and ready, and treat the data and model surface as a first-class attack surface. It is the boards that can demonstrate meaningful human oversight rather than symbolic committees. It is the investors who distinguish between AI revenue that is already contracted and AI capital expenditure that is debt-funded against assumptions.
The exposed are equally clear. Any frontier lab whose valuation depends on uninterrupted scaling into a $2 trillion public market while its own safety staff resigns is carrying a governance discount that an IPO roadshow will struggle to price. Any hyperscaler funding AI build-out with debt while spending ahead of free cash flow is exposed to a confidence shock it cannot patch. Any enterprise that has deployed AI agents into decision loops without an intervention mechanism is one malfunction away from a fiduciary breach.
The forward look splits by horizon. In the short term — the next six to twelve months — sentiment will be driven by the Anthropic IPO, the next wave of model releases, and whether the June U.S. executive order produces concrete benchmarking designations. A clean IPO and steady model launches would keep the bid under the complex. In the medium term — one to three years — the binding constraint shifts to returns: whether AI revenue grows faster than the capital expenditure supporting it, and whether the debt raised to fund the build-out can be serviced without dilution. In the long term, the structural question dominates: whether governance can be made as scalable as capability, or whether the asymmetry between offense and defense hardens into a permanent feature of the technology.
Three scenarios frame the path. The base case is continued scaling with contained incidents — safety holds often enough that the market muddles through, but volatility spikes around each disclosure. The upside case is that defense-in-depth and evaluation capacity mature fast enough to keep pace with capability, turning AI risk into a manageable cost line and validating today's valuations. The downside case is a frontier-model-linked incident with physical-world harm, arriving into a higher-rate environment, triggering both regulatory constraint and a repricing of debt-funded capital expenditure. The falsifying signal for the base case is the one named above: two clean annual cycles with revenue outpacing capex.
The closing judgment: AI risk has graduated from a research-lab debate to a balance-sheet and boardroom reality. The companies and investors that treat it as a compliance checkbox will learn, as Hugging Face did, that the attacker is not bound by the same rules. The ones that survive will be those that price the tail risk before the market forces them to.
"Autonomous, AI-driven offensive tooling is no longer theoretical." — Hugging Face, security incident disclosure, July 2026
The attacker does not need your model to be safe — only your defense to fail once. The market is still pricing AI as if failure were optional.
Explore more exclusive insights at nextfin.ai.
