The AI Operator | October 2, 2026
An AI agent can write code, update records or finish a task you would not know how to do yourself. That can save time. It also creates a problem: how do you know the work is right?
On October 1, a writer described this challenge. People can ask AI to do work they cannot repeat on their own. But when the result seems wrong, they may struggle to find the mistake.[1]
One person's account does not tell us how common this is. Still, the question matters. An agent can give a clear, confident explanation without getting the job right.
Before the work begins, decide what proof you need. The agent should deliver a result you can check, evidence you can follow and a clear account of anything it could not finish. You should not have to do the whole job again to know whether it worked.
What's Happening This Week
Anthropic makes a helpful distinction in its guide to testing AI agents. Watching the steps an agent takes is different from checking the result.[2]
A record of those steps can explain what the agent tried to do. Checking the final result tells you whether it worked. If the agent was meant to update a file, is the right change in the file? If it was meant to change a record, does that record now show the right information?
A good plan and a polished progress report do not answer those questions. The proof needs to come from the place where the work happened.
NIST is also developing ways to test the facts in agent answers and keep records that people can check later.[3] The lesson is simple: build the checks into the job, rather than adding them after something goes wrong.
Why This Is a CFO Problem
The cost of an agent is more than the cost of running it.
Imagine an agent finishes a task in five minutes, but a skilled employee needs forty minutes to check it. The agent's speed alone does not tell you how much work it saved.
Mistakes add more costs. Someone may need to find the original request, read the logs, locate the changed record and work out what happened. That time is part of the price of using the agent, even if the AI bill is small.
Decide what counts as a good result before you see the answer. Otherwise, a convincing explanation may get an easy pass, while a clumsy one gets a closer look.
Different parts of the work need different checks. Software can check exact values and confirm that a change happened. Sources can support factual claims. People can review choices that depend on judgement. Use the right check for each part of the job.
The Operator's Check
Choose one task your agent does often. Look at what it handed back last time.
Could someone new check the work from that handoff? Would they find proof, or mostly an explanation from the agent? If it changed a record, could they open that record and confirm the change? If it made an important claim, could they follow the source? If the work depended on judgement, did someone else review it?
Then look at what happens when the task fails. Can you tell whether the agent never started, stopped partway through or finished without reporting back?
That matters before you try again. If a message was already sent, a retry could send it twice. If a record was already created, a retry could create another one. The system needs a way to check what happened and avoid repeating work.
Finally, measure how long the review takes. Include the time spent checking results, fixing problems and piecing together missing details. Those hours belong in the business case.
Money Move
Before the next run, choose one repeated task where a mistake would be easy to fix. Decide how you will check it.
Start by naming the proof that the job is complete. That might be the updated file, a record opened in the destination system or a test that confirms the result.
Next, choose one important claim the agent might make. Decide what evidence would support it.
Then name one problem that should make the agent stop and ask for help. “Blocked” should be an honest result, not something the agent hides behind a confident answer.
Run the task and time the review. Compare that with the time it takes to do the job by hand. If checking the agent's work still means doing most of it yourself, improve the handoff before using the agent more widely.
Make the evidence easy to find. Make the unfinished parts easy to see. A good handoff should help a person check the work, understand what remains and decide what to do next.
-dg