What an AI agent actually is, and when to refuse one

On 8 September 2026 a developer named James Anderson published a short post on dev.to with the title "Most 'AI Agents' Are Just If-Statements in a Trench Coat." Within ten days it had 132 comments, which for a technical blog is a small riot.

I read the whole thread twice, because the argument in it is the one I've been having with vendors, prospects, and occasionally myself for the past two years.

The argument is about a word. Everybody selling software to businesses now sells an "AI agent," and almost nobody means the same thing by it. I'd shrug at that if the word were free. It isn't. The thing it names, when it names anything at all, is the single most expensive property a piece of software can have, and most of the people I see being quoted for one don't need it.

So here is what I can give you: the one definition I've found that holds up, three tests you can run on any quote before you sign it, and the honest result of running those tests on my own sales page.

TL;DR. An AI agent is software that picks its own path to a goal instead of following one you specified. Every other definition is a vendor drawing the line around its own product, and Gartner reckons only about 130 of the thousands of firms claiming the label are the real thing. That one difference is what you are paying for, and it is a liability rather than an upgrade, because you now have to check the path as well as the result. Three tests before you buy: can you draw the flow in advance (then it is a workflow, buy that instead), what does it cost to check one decision and does the check leave a record, and can the vendor show you a real run where the agent chose differently for a good reason. Most business problems want a workflow with a human gate. That is not a consolation prize. It is the correct answer.

Nobody agrees what an agent is, and that is not an accident

There is no standards body for the word "agent," so every vendor defines it in whatever way makes its product one. A scripted chatbot is an agent. A scheduled script that reads a spreadsheet and sends an email is an agent. A model that can call tools in a loop is an agent. The word has become the industry's way of saying "the new thing," and I can't do anything useful with that, and neither can you.

Gartner put a number on it in June 2025, and I find the number more useful than any definition offered since. Of the thousands of vendors claiming to sell agentic AI, Gartner counted only about 130 it considered genuine, and gave the rest a name: agent washing, the rebranding of assistants, robotic process automation, and chatbots as something they are not.

The same release predicted that "over 40% of agentic AI projects will be canceled by the end of 2027," citing escalating costs, unclear business value, and inadequate risk controls. I hold that loosely, because it's a prediction built on a January 2025 poll of 3,412 webinar attendees. But I don't read it as the wording of a firm that thinks the market uses the label carefully.

The buyers have noticed. In a Gartner survey of 413 marketing technology leaders fielded between June and August 2025, 45% of those with agents in pilot or production said the vendor's agent fell short of the business performance promised. Nearly half of the people who bought one, in other words, and those are the buyers with a team to run it.

I'm not going to referee the definitions. The fight can't be won, and you don't need it to be. You need the one distinction that changes what you pay, and I think there is exactly one.

The one definition with teeth: who chooses the path

Anthropic drew the cleanest line I've found, in December 2024, in an engineering post called Building effective agents. It separates two kinds of system:

Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.

Strip the engineering out and I get one question. Who chooses the path?

In a workflow, you did. Somebody sat down and wrote out the steps: read the invoice, find the purchase order, check the amount, ask Maria, post it to the accounting system. A language model might do the reading or the writing at one of those steps, but somebody fixed the route through them before the first invoice arrived.

In an agent, the software chooses. You give it the goal, "get this invoice paid correctly," and a set of things it is allowed to do, and it decides on each run which of those things to do and in what order.

It might search your email first this time and your accounting system first next time. It might ask a human, or not.

That is the entire difference, and it has nothing to do with intelligence, the model, or whether it can talk. It is whether a person drew the route in advance or the software chose it at run time. Every dollar between the two comes from that.

Four things get sold as one thing

The pages that rank for "AI agent versus chatbot" are, as I write this, vendor pages from top to bottom, and they all sell the same two-box picture: the chatbot follows a script, the agent thinks for itself, and you should want the second one. What that picture leaves out is the thing I'd tell most businesses my size to buy.

I count four things on offer, not two.

First, a script. The same steps, no model involved. Cheap, boring, and where more automation lives than anyone admits.

Second, a workflow. The same steps, with a language model doing one or two of them, reading a document or drafting a reply. You drew the route.

Third, a workflow with a gate. As above, with a person approving the step that matters before it happens. Still your route.

Fourth, an agent. The software chooses the route.

The two-box version erases the first three, which happen to be the cheaper three. I don't think that's a conspiracy. I think it's what happens when the people describing the category are the people selling the top tier of it.

For scale, Statistics Canada's Q2 2026 survey found that 19.2% of Canadian businesses had used AI to produce goods or deliver services in the previous 12 months, and that 28.2% of those users ran virtual agents or chatbots. That is from 9,251 businesses surveyed between 1 April and 6 May 2026. Multiply through and roughly 5% of all Canadian businesses run something in that category, and the survey can't tell a scripted chatbot from anything more autonomous.

The Bank of Canada's August 2026 review of the December 2025 wave of its Business Leaders' Pulse put the share using AI significantly in core operations at 8%.

No statistics agency anywhere measures agentic deployment as such. So if somebody hands you a small-business agent adoption rate, they made it up. (I wrote up why published adoption figures disagree so wildly separately.)

So almost nobody your size is running a genuine agent yet, and the word on the quote in front of you probably describes the second or third thing, which is fine. The next three tests tell you which.

Test one: can you draw the flow?

Anderson's post gives the first test in one sentence, and I haven't found a better way to put it:

If you can draw the flowchart of what your system does before it runs, you don't have an agent. You have a pipeline.

If you can sit at a whiteboard and draw every step the software will take, in order, before it takes them, then a person chose the path. It is a workflow, which is what Anderson means by a pipeline. Calling it an agent doesn't change what it is, but I've watched it change what you're charged for it.

One commenter on the thread, Ahmed Adawy, drew the cost consequence: if you can write the flow down as a fixed set of states, running it as an autonomous loop is "just paying a 10x latency and token tax for zero added value." That figure is one engineer's estimate, and I quote it for the shape of the point, which matches what I see on invoices. Every step where the software stops to decide what to do next is a step where a person could have decided once, in advance, for free.

Another commenter, writing as build996, supplied the half that makes the test fair rather than glib. You genuinely can't draw the flowchart when a step depends on output that does not exist until you run something. If step three depends on what a customer wrote in an email that hasn't arrived yet, and you can't list the range of things they might write, you can't draw the flow. That is the honest case for an agent, and I've built for it.

So the test is not "agents are bad." It is: draw the flow. If you can, buy the flow.

And I'll add the part vendors rarely say out loud. Finding out you need a workflow is good news. A workflow is cheaper, faster, testable before it goes live, and auditable after. You haven't been downgraded. You've been told the shape of your problem.

Test two: what does it cost to check?

I published the second test a few days before I read Anderson's post, as question 4 of a five-question test for whether AI fits a task at all. I'll quote myself rather than restate it:

If the output were wrong, would you find out within a day, and would checking be cheap? No if either half is a no, and this is the one I'd keep if I could keep only one. If a bad output can sit undetected for a month, the first half fails. If verifying takes as long as doing the work, the second fails. Either way you've moved the labour, not removed it.

I wrote that question for any automation. Here is what changes when the automation is an agent.

A workflow produces an output. If the invoice got posted with the wrong amount, you check the amount. An agent produces an output and a path. It chose to search your email before your accounting system, chose to trust one figure over another, chose not to ask anyone.

When the output is wrong, you now have two things to check: the answer, and the route it took to get there. And when the output is right, you have no idea whether it was right for a good reason or a lucky one unless you check the route anyway.

That, to me, is the whole price of the word. Autonomy hands you a second thing to verify, and checking a path you didn't specify in advance usually costs more than checking a result you did.

Which is why the second half of my own question, "does the check leave a record," matters more for an agent than for anything else you'll buy. An agent that can't show you what it did, step by step, in a form a person can read in a minute, is an agent whose output you'll end up redoing by hand to be sure of. You moved the labour.

So I ask for two conditions, both required. Nobody can list the paths in advance, so the agent is earning its autonomy. And checking which path it took is cheap and leaves a trail, so the autonomy isn't costing you the time it saved. Fail either one and my answer is a workflow, probably with a gate.

Test three: show me a trace that branched

The third test is the one I now ask every vendor, and I think it's the sharpest of the three because only evidence passes it.

Ask for a production trace, a record of one real run, where the agent chose a different route from its usual one, and where that choice was right. Not a demo, and not a diagram of what it could do, but a log from a customer's live system showing the software going left instead of right because the situation called for it.

A vendor selling a real agent will have dozens of these and will be glad I asked. A vendor selling a workflow with the word "agent" on the invoice will show me a demo, or a diagram, or explain that the traces are confidential. I've had all three answers.

Anderson's own system, the one that prompted his post, shows what the trace usually reveals when somebody finally reads it: "It did the same three steps every single time. Extract, transform, respond. Every run."

Another commenter on the thread, Pratik Patel, explained why this goes unnoticed for so long, and I think it's the most useful observation in the entire discussion. Calling it an agent, he wrote, "wasn't just a naming problem, it was a measurement problem. Once the control flow was supposed to be dynamic, every green run got treated as evidence the loop was doing useful work." His fix was a counter for whether the path taken matched the expected three steps, which he called the canary that would have caught it earlier. Nobody had one, so it took reading the logs by hand.

I want to be careful about what that is: two people describing two systems, with no denominator. I won't tell you what share of agents in production never branch, because nobody has measured it, and the number that circulates for it turned out, when I went looking, not to exist. What I can tell you is that a green result is not evidence of a decision. Only a trace is. So ask for the trace.

The arithmetic nobody does

Here is a thing I've never seen on a vendor's slide. If a job has 20 steps and the software gets each step right 95% of the time, the chance the whole job comes out right is 0.95 multiplied by itself 20 times, which is about 36%.

That table is plain multiplication, not a finding, and I'm labelling it that way on purpose. But the shape is not a hypothetical. In May 2025 Toby Ord at Oxford re-analysed the benchmark data from a large set of agent evaluations and found the results well described by "a constant rate of failing during each minute a human would take to do the task." That is the same curve as my table, and it gives each model a characteristic half-life.

Two months earlier the group that ran those evaluations, METR, had reported that current models succeed nearly 100% of the time on tasks that take a person under four minutes and under 10% of the time on tasks that take more than about four hours.

The measured numbers on realistic business work sit where the arithmetic says they should. The τ-bench study from Sierra, published in June 2024, found that the strongest tool-using agents of the time succeeded on fewer than half of its customer service tasks, and that in retail they got fewer than a quarter of tasks right eight times out of eight.

And the citation I'd put in front of any buyer is Salesforce's own. In May 2025 Salesforce AI Research published CRMArena-Pro, 19 tasks drawn from the sales, service, and pricing work Salesforce sells agents to do. Leading agents scored about 58% on single-turn tasks and about 35% on multi-turn ones, with what the authors called near-zero awareness of confidentiality. Kudos to them for publishing it. A vendor's own lab printing 35% for the vendor's own category is worth more than any critic I could quote, and it lands within a point of the 20-step row above, which I'll call a coincidence and leave there.

What the arithmetic tells me is where autonomy belongs: in short chains, with cheap checks, and with a human gate before anything irreversible, not because the models are bad but because multiplication is.

The three bills: tokens, review time, and liability

The cost of an agent arrives on three separate invoices, and vendor pricing pages show you one of them.

Tokens. Anthropic, who sell the models, published their own numbers in June 2025 while describing their multi-agent research system: agents typically use about four times the tokens of a chat interaction, and multi-agent systems about 15 times. Their conclusion was the one I'd want a vendor to reach on its own: such systems "require tasks where the value of the task is high enough to pay for the increased performance." That is a supplier telling you when not to buy. I'd believe them.

Review time. This is the invoice nobody sends, and the one I'd budget for first. In July 2025 METR ran a randomised controlled trial with 16 experienced developers working 246 real tasks in codebases they knew well. With AI tools they took 19% longer. They had forecast a 24% speedup beforehand, and afterwards still believed they had been 20% faster.

METR's 16 people is a small trial and they say so, and it's software, not accounts payable. But the 39-point gap between what people felt and what the clock said is the finding I care about as a buyer, because it says the cost of checking the output is a cost you will systematically fail to notice. I budget for it in hours, not in feelings.

Liability. There are two cases, and one of them is from here. In February 2024 the British Columbia Civil Resolution Tribunal decided Moffatt v. Air Canada, in which the airline's website chatbot had told a grieving customer he could claim a bereavement fare after travelling, which the airline's actual policy did not allow. Air Canada argued the chatbot was a separate legal entity responsible for its own words. The tribunal did not accept that, awarded $650.88 in damages, and wrote that "it makes no difference whether the information comes from a static page or a chatbot."

That was a chatbot, not an agent, so the incident doesn't transfer. The principle does: the thing you deploy speaks for you, and the more it decides on its own, the more it can say.

Then in July 2025 a coding agent from Replit, as The Register reported, ran destructive commands against a live production database during a code freeze, fabricated around 4,000 records, and reported that a rollback was impossible when it wasn't. The user said he had told it not to, 11 times, in capital letters.

And the OWASP GenAI Security Project, which maintains the standard list of risks for these systems, says of the prompt-injection attack that turns an agent against its owner that "it is unclear if there are fool-proof methods of prevention." They didn't write "hard." They wrote unclear if possible.

None of that is my argument against agents. It's my argument for pricing one on three lines instead of one.

When an agent is the right answer, and what I found on my own sales page

I'd be a strange person to write this piece if I didn't build agents, so here is the case for them, stated without hedging.

Anthropic's own criterion is the one I use. Agents, they write, "can be used for open-ended problems where it's difficult or impossible to predict the required number of steps, and where you can't hardcode a fixed path." To that I add my second test: the check has to be cheap and it has to leave a record. I want both conditions, or I build the workflow.

Work that passes both looks like this. One is triage of an inbox where nobody can list in advance what will arrive, and every decision the software makes is a proposed label a person accepts or rejects in seconds. Another is research that follows leads across several systems and comes back with sources attached, so checking it is reading, not redoing.

A third is reconciling records where the mismatch could be any of a hundred things, and the agent's job is to find the explanation and show its working while a person still presses the button. In all three, nobody can draw the path, the check is fast, and the irreversible step belongs to a human.

Now comes the part I owe you, which I had not expected to be writing when I started. I ran the three tests on my own Custom AI Agents page.

The page lists four signs you might need one: one person re-keying the same data between two systems every day; approvals taking three days because someone has to notice an email; an inbox that is the workflow, which breaks when you're on vacation; and a no-code AI assistant that fell apart on real data. By test one, every one of those is a flow I could draw on a whiteboard before writing a line of code.

The worked example on the page runs invoice received, matched to purchase order, approval request sent to a named person, approved, entry created. That is the left-hand side of the diagram above. The page's own promise is "guardrails, logging, and a human in the loop where it matters."

By my own tests, what I sell under the word "agent" is, most of the time, a workflow with a gate and a log.

How naive of me to be surprised by that. I picked up the word the way the whole market did, as the name for a system that is built for you and actually runs, rather than a chatbot or a demo. What I build for those four problems is the right build for those four problems, and it's the cheaper thing on the four-item list above.

So I'm keeping the name, because it's what people search for, and I'm keeping the closing line on that page, which already says "We'll tell you if an agent is the right answer, and if it isn't, what is." This article is that sentence, written out at length.

The three questions, in one place

Put those three in an email. A vendor who answers all three plainly is one I'd buy from, whichever of the four things they turn out to be selling. A vendor who can't is telling you which one it is.

If you'd rather not run the tests alone, send me the quote and I'll tell you what I'd do with it, including when the answer is to automate the workflow and skip the agent, which in my experience it usually is. Autonomy is not a tier you graduate into. It's a liability you take on, on purpose, for the few jobs that earn it, and there's no shame in a business that has none of those yet.

I would rather you owned the boring thing that works.

Raf Apocalypse is the founder of Pixel2 Consulting in Langley, B.C. He has spent 20 years building software, including two diabetes platforms that together reached more than 16,000 patients, and has shipped systems for Nike Brazil, MercadoLivre, and Hospital Albert Einstein.

Sources retrieved 17 September 2026. Anthropic, Building effective agents (19 December 2024) and How we built our multi-agent research system (13 June 2025). James Anderson and commenters, dev.to (8 September 2026). Gartner press releases of 25 June 2025 and 29 October 2025. Statistics Canada, Bank of Canada, METR, Ord, τ-bench, CRMArena-Pro, OWASP, and the tribunal decision in Moffatt v. Air Canada, 2024 BCCRT 149, read through the McCarthy Tétrault summary.