Ship the agent with a human in the loop. Always.
Every agentic project we've shipped has gone live with a human in the loop. Every single one. Not because we lack confidence in the agent — because the alternative is recoverable in theory and unrecoverable in practice.
A v1 agent making autonomous decisions in production has three problems at once.
Problem 1: You don't know what it's wrong about. A model that's right 95% of the time fails 5% of the time, and the 5% is not random. There's a distribution of failures, and you don't know its shape until you've watched a lot of them. Pre-launch evals catch some of it. They don't catch all of it.
Problem 2: The blast radius is uncapped. "The agent sent the wrong email to a customer" is recoverable. "The agent issued a refund to the wrong account" is recoverable with a phone call. "The agent took an action that compounded across other agents" is the kind of thing that ends up in a Wall Street Journal headline. The right time to figure out the blast radius is before you put the agent in front of customers.
Problem 3: You learn nothing. When the agent runs autonomously and silently, you have no signal on which decisions were good. A queued action that a human approved or edited is a label. A silently-fired action is a guess.
The shape we ship looks like this:
1. The agent runs end-to-end in shadow mode for at least one week. Real inputs, real reasoning, real proposed actions — but the actions go to a queue, not to the world.
2. Operators review the queue. They approve, edit, or reject. Every decision is logged with the operator's reasoning when they edit.
3. We instrument the queue. Approval rate, edit rate, reject rate, time-to-decision. All visible on a dashboard the founder can look at.
4. Once the approval rate is consistently above a threshold for a class of decisions, we let those decisions auto-approve with a 60-second delay (operator can still intercept).
5. After more time, those decisions go fully automatic. New decision classes start back at step 1.
This is slower than "ship and watch the metrics." It's also the difference between an agent that's been live for a year and a project that got killed in week three after the first incident.
The human-in-the-loop layer is part of the v1 deliverable, not a transitional scaffolding to be removed at launch. It's how the agent earns the right to autonomy.
If a vendor or consultant is selling you a fully autonomous v1 agent, ask them about their incident playbook. The answer will tell you everything you need to know.
— Wash Candido