Before any team argues about vendors, dashboards, or demos, the argument should be settled by arithmetic. AIOps for Security is an editorial label still settling into usage — attached to machine-learning and LLM-assisted security operations, backed by no formal standard, and lacking any founding milestone — and it is best understood through a short list of numbers that recur across every serious treatment of the field. Seven of them do most of the explanatory work. Each one below is stated, sourced to the published record, and translated into a plain conclusion a reviewer can carry into any procurement conversation. None of them requires a product trial to appreciate, which is precisely why they belong at the front of the discussion rather than the end.
Two in a Hundred Thousand: The Rate That Rules Everything
Screening for something rare is mathematically punishing, and the punishment was quantified long before modern tooling arrived. Axelsson’s 2000 worked example took a condition affecting two individuals in every hundred thousand and a test that raises a false alarm once per hundred thousand examinations — and the flags still landed on true cases only two times in three. Relax the false-alarm rate a thousandfold, to once per thousand, and the picture inverts: roughly one flag in seventy points at something real. The lesson generalises directly to attack detection. When the thing you hunt is scarce, the accuracy of the test matters far less than how seldom it cries wolf, and any conversation that discusses sensitivity while ignoring the alarm rate has not started honestly.
50 Million Events, 50 Attacks: The Queue Arithmetic in AIOps for Security
Scale the same arithmetic to a modern estate and the queue becomes brutal. Picture a monitoring stack ingesting fifty million events per day in which fifty are genuinely malicious. Let the detector err once per ten thousand events. The resulting flag pile contains barely one genuine attack in every hundred entries — roughly 112 alerts examined per attack actually caught, with analyst attention gone long before then. Push the false-alarm rate down two more orders of magnitude, to one in a million, and the same detector’s flags become useful: nearly half point at genuine attacks. Nothing about the detection model changed between those two outcomes. The base rate did the work, which is why AIOps for Security claims deserve scepticism until the alarm arithmetic is shown, not asserted.
2.8% Measured, 8.6% Bounded: A Good Trial That Failed Its Gate
One published-style triage evaluation shows how demanding honest measurement is. A mail-triage agent was assessed against 1,240 blind-labelled messages containing 71 true positives; it missed two of them. The observed miss rate of 2.8% looks respectable until the statistics enter: with so few positives, statistical honesty pushes the 95% ceiling to roughly 8.6% — nowhere near a 2% gate. The same evaluation reported closing precision of 99.8% and a 93.5% share of workload that could safely be removed — impressive figures that do not rescue the miss bound. A separate “true positive” verdict read from the agent’s own reasoning reached precision of just 86%. Precision, workload relief, and miss control are three different claims; only the last one permits autonomy, and it was the one that failed.
236 and 313: The Sample Price of Proof in AIOps for Security
How much evidence does a safety claim actually cost? The rule of three gives the cheap answer: zero errors observed across n cases bounds the underlying error rate near three divided by n. To argue a miss rate at or below 2% when even one miss has been observed requires 236 evaluated cases; with two misses on record the requirement rises to 313. These are not academic figures. They are the reason the autonomy ladder gates each step on labelled-case counts — roughly 200 labelled alerts with at least 30 true positives before a system may merely recommend, and no auto-closing until a miss bound of 2% or better has been demonstrated on at least 150 true positives with canaries and a blind holdout behind it. Vendors rarely volunteer sample sizes. Ask, and watch how quickly the conversation changes.
42.3% Behind 9.1%: The Pooling Trap That Hides Failures
Averages across alert categories can conceal almost anything. In one documented evaluation, a pooled miss rate of 9.1% looked tolerable until the categories were separated and cloud-IAM decisions turned out to have been missed 42.3% of the time — a failure rate more than four times the headline, hidden by mixing denominators. The pattern generalises: an overall figure of 1% can sit on top of a 40% failure rate in whichever category happens to be hardest. Any evaluation of AIOps for Security tooling that reports a single blended number is, whether deliberately or not, burying its worst result. Insist on a per-category table. The categories where a system performs poorly are predictable in direction — unusual access patterns, identity anomalies — and they are exactly the ones an attacker probes.
25.2%: What Agents Actually Resolve on the Bench
Independent benchmarks deflate marketing quickly. ITBench, presented at ICML 2025, saw autonomous agents finish just 25.2% of the security-and-compliance tasks in its suite. CyberSOCEval, published in 2025 by Meta and CrowdStrike, measured agent success at roughly 23–34% on malware analysis and about 43–53% on threat-intelligence reasoning — against random baselines of 0.63% and 1.7% respectively. Real capability, in other words, but far from autonomy, and a wide margin below the fluency their outputs suggest. Gartner’s decision to seat AI SOC agents on the 2026 Peak of Inflated Expectations, alongside its warning about “AI washing” — accuracy figures polished past what measurement supports — fits this data neatly. A demo that answers smoothly is not evidence; a benchmark table with the failure cases listed is.
50 Canaries a Quarter: Cheaper Than One Wrong Auto-Close
Verification after deployment has a known weak spot: sampling closed decisions at random barely constrains anything. A 5% audit of roughly 1,845 quarterly auto-closes that finds zero misses still leaves the miss rate bounded near 38% — an assurance figure indistinguishable from knowing nothing. Canaries fix this because they are planted, not sampled: fifty seeded cases per quarter, each a known true positive, with a withdrawal trigger the moment an agent closes three or more of them. The cost is trivial; the sensitivity is orders of magnitude better than random review of real traffic. A program that cannot show its canary results is asking for trust it has not earned, and the ask should be declined politely but firmly.
Reading the Terms Before You Sign
Numbers of this kind reward the same discipline a traveller applies when vetting anything unfamiliar — the habit of checking the source before trusting the promise, the way one compares rental terms before booking quality vehicles for an unfamiliar road trip. A short list serves as the takeaway:
- Demand the base rate and the alarm rate behind any accuracy figure, and compute how many alerts one real catch costs.
- Treat every pooled percentage as suspect until it is broken out per category, worst case first.
- Ask for sample sizes against the 236-and-313 standard, and for canary results rather than random spot checks.
Preparation beats improvisation here, just as planning matters more than confidence once the road turns remote — a point made practically in a fuel-planning routine for long, empty roads. Teams that internalise these seven numbers stop buying fluency and start buying evidence, which is the only currency this field actually honours. The field notes on AIOps for Security kept by the Ops Atlas collect the arithmetic in one place, and it has been published for a quarter century; the only open question is whether buyers will read it.
