Everyone wants more autonomous agents. The teams shipping reliable ones are doing the opposite.

Everyone wants more autonomous agents.

The teams shipping reliable agents are doing the opposite.

“Agentic” got conflated with “autonomous” sometime last year, and the AI crowd has been chasing the wrong thing ever since.

What actually happens in production

The more autonomy you give an agent, the more decisions it makes per task. Every decision is a roll of the dice. Your reliability collapses fast — a 95% reliable agent making 10 decisions is right about 60% of the time.

That’s not an agent. That’s a coin flip with extra steps.

The agents shipping real work look almost boring:

• Narrow scope. One workflow, not “your AI assistant for everything.”
• Tool calls gated by confirmation where reversibility is low.
• Deterministic routing wherever it can replace LLM judgment. The model decides what’s hard, not what’s obvious.
• Failure modes designed before features. What does the agent do when it doesn’t know? Most teams haven’t answered this.
• Human-in-the-loop is a feature, not a fallback.

The most reliable agentic systems I’ve built had the model making the fewest possible decisions. Everything else was plumbing, contracts, and guardrails.

The math isn’t quite right — and that’s the point

One thing worth flagging: the 0.95^10 ≈ 60% calculation assumes independent failures, which isn’t quite how LLM errors compound in practice. Some failures correlate (one bad context decision poisons the next five), some recover (a re-prompt fixes the previous misstep).

But in production, the correlated failures are the ones that actually kill you. Independent errors you can catch with retries. Correlated errors look like an agent confidently going off the rails for eight steps before anyone notices. That’s where the reliability story gets ugly.

Where to draw the line

The autonomy budget isn’t a binary setting. It maps to reversibility:

• High autonomy for decisions that are cheap to undo.
• Near-zero autonomy for decisions that aren’t.

The interesting design constraint in domains like finance: reversibility isn’t binary either. Some actions are technically reversible but operationally catastrophic — a trade you can unwind but only after PnL impact. That changes where you draw the human-in-the-loop line. It’s not just “can we undo this,” it’s “what does undoing this cost.”

“Autonomous” sells better in keynotes. “Constrained” ships better in production.

Where do you draw the autonomy line in your agents?