A Five-Role Diagnostic For Any AI Agent

Most Agent Problems Have A Diagnostic

When an AI agent fails in production, the conversation almost always turns to the model. The team waits for a bigger model, a better-tuned variant, a different vendor, a new context window. Sometimes the team turns to prompts instead: more careful instructions, more examples, a sharper system message. Most failures are neither. They are failures of structure, and the structure question is a thirty-year-old framework most engineers have never read.

This piece turns that framework into a working diagnostic. It is the kind of checklist you can apply to any agent system in about an hour and come out with a clear picture of where the system is structurally weak. Where the answer to one of the five questions is empty or vague, that is almost always where the system will eventually break.

The Diagnostic, In Outline

Davis, Shrobe, and Szolovits published their paper, What Is a Knowledge Representation?, in 1993. It defined what a knowledge representation actually is by listing the five roles such a representation plays. The first piece in this series argued that those five roles describe what a modern AI agent harness does, with surprising precision. This piece turns the description into questions.

The rest of this piece works through each question, what good looks like, and what failure looks like.

Role One: The Surrogate

The surrogate is the agent’s stand-in for reality, the part of the system it consults when it needs to know what is the case. In a useful agent, the surrogate is current, consistent, and visible. In an unreliable agent, the surrogate is none of those things.

What good looks like: a clear, structured representation of the entities and state the agent cares about. Knowledge graphs, structured memory, retrieved documents with provenance, world models that update on observation. The system can be asked what it currently believes about any relevant fact, and the answer is consistent across turns.

What failure looks like: the agent forgets information across turns, hallucinates state that was never asserted, gives different answers about the same fact in different sessions, or has no way to be queried about its current beliefs. The team trying to debug the system cannot inspect what the agent thinks; they can only watch what it does.

The diagnostic question: What is the agent’s surrogate for the world it acts on, and how does that surrogate stay current? When the answer is “the conversation history” or “whatever the model retrieved this turn”, the system has no real surrogate, and behaviour will be inconsistent in ways that are hard to debug.

Role Two: Ontological Commitments

The ontology is the declaration of what exists in the agent’s world. The cleanest place to look at it is the tool list. Tool schemas declare what entities and operations the agent can name, and the agent cannot reason about anything not expressible in those schemas.

What good looks like: tool definitions with explicit types, strict parameter constraints, and clear semantics for each operation. A tool list that forbids invalid operations as a matter of structure, not as a matter of the model choosing not to call them. When a junior engineer reads the schema, they can tell what the agent can and cannot do.

What failure looks like: free-form arguments, loose schemas, overlapping tools with unclear semantics, no commitment about what is and is not in scope. The model invents calls that look plausible and the system either accepts them and fails downstream or rejects them at execution. Either way the ontology is being defined by what the model happens to attempt rather than what the team has committed to.

The diagnostic question: What does your tool list forbid? An ontology is defined by exclusion. If everything is permitted, nothing has been committed to, and the system will be unreliable in proportion to the lack of commitment.

Role Three: Fragmentary Theory Of Reasoning

The harness does not just encode what the agent knows. It encodes how the agent should reason. This is the part of the system that has a theory about which steps come first, what checks happen between them, and what to do when something goes wrong.

What good looks like: explicit planning structures, recovery logic for predictable failure modes, verifier steps between actions that could go wrong, an audit-able trace of how the agent arrived at each decision. Even when the model is doing the underlying reasoning, the harness has a view about what reasoning shape is acceptable.

What failure looks like: “loop until done” as the only structure, no recovery from intermediate failures, no verification between steps, plans that drift after step seven because nothing in the harness was holding them. The team has no way to debug a multi-step failure because there is no concept of a step.

The diagnostic question: What theory of reasoning does the agent enact, beyond the language model’s token-level choices? If the answer is “we trust the model to figure it out”, the answer is “nothing”, and reliability will scale with how forgiving the use case is.

Role Four: Medium For Efficient Computation

The harness makes some operations cheap by structuring memory, indexing, schemas, and caches. The choices about what to make cheap are also choices about what to make hard or impossible.

What good looks like: structured memory with retrieval by key, type, or relationship rather than by similarity alone. Cached plans for common task shapes. Schemas that make valid operations easy to construct and invalid ones syntactically wrong. The system is fast at the things it does often, by design.

What failure looks like: every lookup is a semantic similarity search, every reasoning step starts from scratch, every retrieval risks the same kinds of errors, no caching of anything that could be cached. Performance gradually becomes a tax on every decision.

The diagnostic question: What does the harness make cheap, and what does it make impossible? If everything is equally possible, nothing has been optimised for, and the system will be both slow and brittle in places that should be neither.

Role Five: Medium Of Human Expression

The harness is what humans actually read, edit, and reason about. The model is opaque. The prompt is a thin and over-loaded surface. The harness is where domain experts and engineers express what the system should do, in a form that can be inspected, reviewed, and versioned.

What good looks like: a clear separation between domain knowledge and engineering plumbing. Tool definitions that read like a domain expert wrote them. Verifier rules that a subject-matter expert can read and edit. Audit trails that someone responsible for the system can understand.

What failure looks like: the entire system is opaque except the prompt. Domain experts have no way to contribute except by editing prompts. Every change goes through engineering. Nobody outside the engineering team can audit what the system is doing or why.

The diagnostic question: Where does domain understanding live, in a form a non-engineer can read and edit? When the answer is “in the prompt” or “in the model”, the system has no medium of human expression, and it will be difficult to maintain in proportion to how specialised the domain is.

Using The Diagnostic

The five questions can be asked in any order. The pattern we see most often in client work is that teams have answers for roles two and four (the tools are typed, the memory is indexed) and struggle on roles one and five (no clear surrogate, no editable layer for domain understanding). When agents fail in production, the failure usually traces to one of the gaps.

Once a weak role is identified, the work is structural, not prompt-tuning. Building a real surrogate. Tightening the ontology. Putting an explicit reasoning layer over the model. Creating a place where domain experts can express what the system should do without becoming engineers. These are the harness improvements that pay off across model upgrades, vendor changes, and shifts in the underlying technology.

The diagnostic is older than most of the engineers using it. That is not a flaw. It is a sign that the questions about what makes intelligent systems work are durable in a way that the answers, year by year, are not.

This is the second piece in a short series on neurosymbolic AI and practical agent systems. The first piece argued that the modern agent harness is an unrecognised return of symbolic AI, with the symbolic content concentrated in specific components rather than spread evenly. The next piece turns to retrieval architecture, and the case for moving from retrieval augmented generation to knowledge augmented generation.

If your team is building agent systems and the diagnostic surfaces gaps you would like a second opinion on, our fixed-fee advisory sessions are structured around exactly this kind of harness-first review. Get in touch.