Why the Same Task Does Not Produce the Same Solution

Ask a coding agent the same thing five times and you may get several different, yet individually plausible, solutions.

That does not mean five attempts will inevitably produce five completely different implementations. Large parts will often look similar, especially when the task is narrowly defined and the existing code already imposes strong conventions. Nor does it mean Large Language Models are random machines that arbitrarily take a different turn on every run.

The more important technical point is this:

There is no technical guarantee that the same task will always produce the same solution path and the same implementation.

This property has long been familiar from simple text generation. With coding agents, however, it takes on another dimension. An agent does not merely generate text or code. It searches a repository, reads files, chooses Tools, runs tests, interprets their results, and adjusts its next steps accordingly.

An agent therefore does not just produce an answer. It constructs a path to a solution.

That is exactly where the idea that agentic software development behaves like a new, much faster assembly line starts to break down.

The Myth of Assembly-Line Work

A large part of the discussion around AI productivity can be reduced mentally to a very simple model:

Ticket
  ↓
Agent
  ↓
predictable processing
  ↓
Patch

If that picture were accurate, a coding agent would mainly be a new form of automation. We could keep feeding it more tickets and would essentially only need to measure how quickly and cheaply it turns them into code.

For some tasks, reality can come surprisingly close to this model. The more narrowly a task is defined, the more strongly the solution space is constrained by existing structures, and the fewer decisions the agent has to make itself, the more similar separate runs can become.

Agentic work becomes interesting precisely where the path to the solution is not fully predefined.

Anthropic therefore distinguishes between Workflows, where models and Tools are orchestrated along predefined code paths, and Agents, where the model itself determines how to approach a task and which Tools to use. OpenAI describes Agents similarly as systems in which a model controls the workflow, makes decisions, and dynamically selects Tools based on the current state.

That is more than a terminological detail.

A hard-wired Workflow does not reconsider during execution which route might be more useful. An Agent does exactly that.

An Agent is not a ticket-to-patch machine with a guaranteed production pattern.

Agentic Work is therefore less like an assembly line and more like a sequence of decisions made under changing Context.

What Determinism Means in Conventional Software

In software development, we are used to systems in which the path between input and output is completely defined by code.

In simplified form:

Input
  ↓
Step A
  ↓
Step B
  ↓
Step C
  ↓
Output

If the same inputs are present and the relevant state has not changed, we expect the same sequence and the same result.

Of course, conventional software also has many sources of variation: time, concurrency, external services, network state, random values, or changed databases. The important distinction is that we can usually model this variability as part of the explicit system state. The algorithm itself does not spontaneously decide that today it would rather try step B2 than step B1 because that route currently looks more plausible.

Many forms of automation depend on exactly this property.

A formatter should produce the same formatting for the same file. A compiler should produce the same program from the same source code under the same conditions. A CI pipeline should not reconsider on every run which quality checks it feels are useful this time.

For these problems, determinism is extremely valuable.

That also explains why the assembly-line mental model is so attractive for Agents. For decades, software development has turned more and more repetitive work into deterministic processes.

A coding agent, however, is not simply another fixed step in that progression. It introduces a model into the control flow that can choose, within certain boundaries, what appears to be the most useful next step.

Why LLMs Do Not Behave Like a Conventional Function

From an application perspective, a Large Language Model can certainly be invoked like a function:

const result = model(input);

But this notation should not lead us to assume that the system conceptually behaves like a conventional function that must return exactly the same output for exactly the same input.

A language model calculates probabilities for possible continuations of the current Context. In simplified terms, there is not just one next Token but a distribution of possible continuations. Some are highly likely, others less so, and many are practically irrelevant.

The model therefore does not behave arbitrarily. If we ask for a TypeScript function, it will not be equally likely to produce a recipe or a medieval poem. The existing Context strongly constrains the space of sensible continuations.

Within that space, however, several continuations can be plausible.

Sampling parameters such as temperature or top_p influence how selections are made from these probability distributions. They can reduce or increase variation, but they are not the focus of this article. Settings that improve repeatability should not be confused with a universal guarantee of identical outputs either. For agentic work, what matters more is what happens when small differences affect not just wording, but the next work step the model chooses.

For our mental model, this statement is enough:

Non-determinism does not mean that a model behaves arbitrarily. It means that several plausible next steps can exist within a solution space.

For a single answer, that may lead to different wording. In code, two implementations may structure the same requirement differently.

With an Agent, however, something more interesting happens.

From Token Variation to Decision Variation

A coding agent usually does not produce the final patch in a single step.

It may first inspect the directory structure. Then it might search for an existing similar feature. Next, perhaps it opens a facade, a store, and a test. Or it starts with the API contract. Maybe it runs an existing test first. Maybe it searches for a symbol. Maybe it looks at Git history.

All of these actions can be reasonable.

A typical Agent loop can be simplified like this:

Task
  ↓
Decision
  ↓
Action / Tool
  ↓
Observation
  ↓
new Context
  ↓
next decision

Loops like this are among the central architectural patterns of today’s Agents. Based on the current state, the model decides what action should come next, observes the result, and continues with the additional information that has become available.

This turns the variation of a language model into a new kind of variation.

The difference is no longer limited to whether one line of code is written in two different ways. Variation can already occur in

• which file is inspected first,
• which search is started,
• which test is run first,
• which hypothesis is formed about the existing architecture,
• which failure is judged relevant,
• which Tool is used next.

One of those decisions may expose information that does not appear at all, at least initially, in another run.

For an Agent, variation is not only output variation. It can become process variation.

For understanding agentic systems, that is much more important than the simple observation that language models generate probabilistically.

Context Is Not a Static Input Block

The previous article in this series explained that Context is not simply everything a model could theoretically know. What matters is which information is actually available during a particular work step.

For an Agent, that Context is also not static.

After one action, for example, a search result may be added:

Search Result

After the next, a piece of source code:

loaded file

Then perhaps:

Compiler Error

or:

Test Output

or:

Git Diff

Each observation can change the assessment of which action should come next.

Anthropic therefore describes Context Engineering for Agents not merely as designing a good initial prompt, but as the ongoing question of which information should be present in Context during a multi-step process. Agentic systems can also load information “just in time” instead of placing everything in the initial Context.

That leads to an important consequence.

Imagine two Agent runs with the same task. Both begin under almost identical conditions. In the first run, the Agent decides to search for an existing implementation of the same pattern. In the second, it opens the files of the new feature directly.

After that first decision, the two situations are already no longer identical.

Run A may now contain an existing architecture example in Context. Run B instead contains details of the local implementation.

The next decision is made on the basis of that different Context.

Early differences in the solution path change the later Context.

And the changed Context in turn influences the next decisions.

This creates path dependence.

Path Dependence: A Small Turn, a Different Route

The term path dependence sounds more theoretical than it is in this context.

It simply means that the future course of a process depends on the steps that have already taken place.

Imagine a repository with two similar features. The older feature uses a structure that is now outdated; the newer one follows the current architecture.

One Agent finds the newer feature first and adopts its pattern.

A second Agent encounters the older feature during its search and interprets that as the local convention.

Both decisions can be understandable in the moment. The Agent does not automatically know the historical evolution behind the two implementations.

From that point on, the available clues differ. Further searches may use different terms. Different files appear relevant. Different dependencies are examined. The resulting patch takes a different shape.

A small difference at the beginning can amplify over several steps.

That does not mean two Agents must inevitably keep drifting further apart. A test, compiler error, or explicit architecture rule may later pull both paths back toward the same solution space.

But there is no guarantee.

That is why it is misleading to look only at the final output and ask why “the same input” suddenly produced two different solutions. With an Agent, the process consists of a whole sequence of decisions and observations.

The Same Task Is Not Automatically the Same State

There is another point that is easy to miss when comparing runs.

Two identical prompts do not necessarily mean that two Agent runs actually start or proceed under mathematically identical conditions.

Differences may include

• files that are already loaded,
• search results,
• results from external Tools,
• repository state,
• existing session history,
• available Memories,
• activated Skills,
• cached information,
• test or compiler output.

With web-based Tools, even the outside world may have changed between two runs.

This argument should not be pushed too far. It would be equally wrong to conclude that Agent runs are fundamentally incomparable. Good evaluation deliberately tries to control the initial state as carefully as possible. Anthropic, for example, recommends isolated, clean environments for each Trial in Agent evals so that leftover files, caches, or shared state do not introduce additional variation.

Even when we control these external influences reasonably well, however, the core point remains:

Even under very similar initial conditions, the solution path does not have to be identical.

An Agent still has room to make decisions within the permitted solution space.

An Observation from My Architecture Lab

I do not encounter this behavior only in theory. In my architecture lab, I deliberately have coding agents build many small applications and recurring features. Part of that work is simply about trying out technologies and architecture approaches. At the same time, I use these projects to understand how precise the guardrails for coding agents actually need to be.

I am especially interested in Agent Files and Skills.

Agent Files describe architecture rules, for example: Which layers exist? Where does State belong? Which dependencies are allowed? Which patterns should be used, and which should be explicitly avoided?

Skills describe recurring ways of working: How is a particular Slice built? Which steps belong to a CRUD feature? Which checks should run after a change?

In one demo app for a training seminar, I had reached a state I was genuinely happy with from an architecture perspective. The structures were clear, several features followed the same principles, and the relevant Agent Files and Skills were already in place.

Then I had the Agent build another feature. There was nothing particularly exotic about it, and structurally it was another CRUD case — exactly the kind of task for which the project already contained examples and work instructions.

Yet the Agent suddenly produced an architecture that I would never have designed in that form myself. What made this interesting was not that the Agent had created obvious chaos. Quite the opposite: the solution was internally plausible, and individual decisions were understandable. It was not a random pile of code, but a consistent solution path that simply did not lead to the architecture I expected.

Somewhere early in the process, the Agent had taken a different turn. Perhaps it had used another existing structure as its first reference, or an early interpretation made different files look relevant later on. Which exact turn it was matters less than what followed: that decision created different Context, and the next decisions were then made on the basis of that Context.

This observation is not scientific proof of general behavior. It is one practical example. For me, however, it was instructive because the obvious explanation was specifically not that the Agent lacked guardrails.

The Agent Files, Skills, and existing architecture examples were all there. Even within those boundaries, room for decisions remained.

Guardrails can constrain the solution space, but they do not automatically make the decision path deterministic.

That distinction matters a great deal in practical work with coding agents.

Non-Deterministic Does Not Mean Uncontrollable

At this point, it is easy to draw the wrong conclusion.

If an Agent’s path cannot be predicted exactly, one might conclude that agentic systems are fundamentally uncontrollable.

That does not follow.

Non-determinism and arbitrariness are two different things.

Software architecture already gives us a useful analogy. A team is normally not told in which order every developer must open files. Yet we can still define very precise rules for what an acceptable solution should look like.

Controlling an Agent works in a similar way.

We can constrain its solution space with clear Requirements. Architecture rules can forbid unwanted dependencies. Agent Files can expose project-specific conventions. Skills can describe proven workflows.

Then come the harder constraints from conventional software engineering.

A Type System simply rejects certain states. A compiler can reject invalid code. A linter can enforce architecture or quality rules. Tests can verify domain behavior. Architecture tests can detect forbidden dependencies. Evals can check recurring behavioral requirements. Reviews can assess aspects that are not practical to automate completely.

This is a fundamental difference between controlling a solution path and controlling a solution space.

We cannot guarantee which path an Agent will take. But we can define very precisely which paths are acceptable.

For me, that is the more useful engineering perspective. The goal does not have to be turning an Agent into a badly reconstructed deterministic Workflow through ever longer prompts that makes every solution path identical. The goal is to define the space of acceptable solutions.

The better that space is described and technically enforced, the less problematic it becomes that different runs choose different routes within it.

Multiple Solutions Are Not Automatically a Problem

Variation is often discussed only when two results differ. That gives it a negative tone very quickly.

Yet different solution designs are nothing unusual in software development.

Give five experienced developers the same ticket and you are unlikely to get five identical pull requests. One may extend an existing abstraction, another may consider a local implementation simpler. One starts from the data model, another from the Use Case. Some differences would be discussed in Review. Others would simply represent legitimate trade-offs.

The comparison with humans is not meant to anthropomorphize model behavior. It only makes visible that, in a problem with several valid solutions, variation is not automatically the same thing as poor quality.

Two Agent runs can produce different implementations that are both correct. They can expose different trade-offs, pursue different hypotheses about a bug and still reach the same cause, or propose alternative architecture variants that make a discussion possible in the first place.

In such cases, variation is not a defect. It is part of an open solution space.

It becomes problematic when those differences cross boundaries that matter to the system: when functional Requirements are met only sometimes, when architecture rules are not followed consistently, when security constraints vary from run to run, or when a process requires reproducibility that the agentic system cannot provide reliably.

The relevant question is therefore not:

“Why did the Agent do something different this time?”

but:

“Is what it did differently still within our acceptable boundaries?”

That is a much more productive quality question.

What This Means for Evaluation

From this perspective, it also becomes easier to understand why a single impressive Agent run has limited evidential value.

If a system behaves probabilistically, makes several decisions, and changes its later Context through earlier actions, then one successful run initially shows one thing: that this run succeeded.

It does not automatically prove that comparable tasks will be solved just as reliably.

In its current guidance for Agent evals, Anthropic therefore explicitly describes each individual attempt at a task as a Trial. Because model outputs can vary between runs, multiple Trials are used to assess behavior more robustly. Particularly useful is the distinction between the probability of succeeding at least once and the stricter question of how consistently an Agent succeeds across repeated attempts.

For normal software development, that does not mean every small ticket suddenly has to be implemented fifty times by an Agent.

It is about the mental model.

If I want to know whether a new Agent File reliably communicates an architecture rule, one successful attempt is only a weak signal.

If I develop a Skill for recurring CRUD work, I should not test it only with the task I happened to use while writing it.

If I evaluate an Agent model or a new Harness for my repository, I need representative tasks and more than one lucky run.

Evaluation should also avoid prescribing an unnecessarily exact solution path. Here again, the distinction between path and outcome matters. Anthropic notes that rigidly checking a specific sequence of Tool Calls in Agent evals can punish legitimate solution paths. Where possible, the evaluation should therefore check whether the desired result and relevant Constraints were satisfied rather than dictate every intermediate step.

That maps well to practical software development.

For a coding agent, such criteria might include:

• behavior covered by tests,
• public API unchanged,
• no forbidden layer dependencies,
• Type Check successful,
• linter successful,
• security rules satisfied,
• functional Use Case fully implemented.

How the Agent reached that state can be secondary as long as the path itself did not introduce relevant risks.

When a system is probabilistic and path-dependent, a single successful run is rarely enough as a quality statement.

This is not a demand for unlimited evaluation. It is simply the consequence of not confusing Agent quality with one good demo.

When a Workflow Is the Better Solution

That brings us to a question that is surprisingly easy to forget amid enthusiasm for Agents:

Does this task need to be agentic at all?

For LLM-based systems, Anthropic explicitly recommends starting with the simplest approach that fits the problem. For clearly defined tasks, Workflows can provide more predictability and consistency; Agents become interesting when flexibility and model-driven decisions are actually required.

That is not a limitation of Agents. It is ordinary architecture work.

If I know exactly that every commit must always be followed by

Format
→ Lint
→ Test
→ Build
→ Deploy

I do not need an Agent to reconsider each time which step might be useful.

If a file can be transformed according to clear rules, a script is probably better.

If a generator can produce reproducible boilerplate code from a known schema, a generator is exactly the right tool.

An Agent is strongest where decisions are necessary.

Which parts of the system are relevant to this bug?

Which existing implementation is the best reference?

Which test helps verify a hypothesis?

Is a local fix sufficient, or does the bug point to a structural problem?

Which of several possible solutions best fits the existing Constraints?

A fully deterministic process for questions like these would only be possible if we had already encoded the answers in software.

That leads to a simple engineering rule:

The more precisely the correct solution path is already known, the less reason there is to leave that path to an Agent’s discretion.

Automation and agentic work are therefore not competing concepts.

A good system combines both.

Deterministic mechanisms secure what is known and verifiable. The Agent receives room to decide where that flexibility actually adds value.

Agentic Work Is Not Assembly-Line Work

The assembly-line metaphor promises a comfortable world: a ticket goes in at the top, the Agent processes it, and a patch comes out at the bottom. Once that works, we would only need to add more tickets and more Agents.

But that picture underestimates the very property that makes an Agent interesting. It interprets its current Context, decides on the next step, acts, observes the result, and decides again. Early branches can expose different information; the changed Context can in turn make different next steps look sensible. That is how path dependence emerges.

This explains why the same task does not have to produce the same implementation without making that variation automatically negative. Several paths can lead to correct results. Architecture rules, Agent Files, Skills, Tests, compilers, Type Systems, and Evals can constrain the solution space strongly enough that different paths still end within the same quality boundaries.

Agentic software development therefore requires a different idea of control: not every step has to be predefined. What matters is knowing which outcomes we accept and which boundaries must not be crossed on the way there.

Agentic Work is not assembly-line work.

And non-determinism initially explains only why two plausible runs can take different routes.

It does not yet explain why an Agent can be remarkably convincing while being wrong on one of those routes. Nor does it explain why certain assumptions or architecture decisions can become entrenched over multiple changes even though they were never intended in the first place.

Different solutions would, after all, be relatively harmless if all of them were correct.

Why Agents can hallucinate and how individual deviations can turn into long-term Drift is therefore the next question in this series.

Sources and Further Reading

• Anthropic: Building effective agents — Distinguishes predefined Workflows from model-driven Agents and discusses when to choose either approach. Building effective agents
• Anthropic: Effective context engineering for AI agents — Describes Context as dynamic state in multi-step Agent loops and information loaded “just in time.” Effective context engineering for AI agents
• Anthropic: Demystifying evals for AI agents — Covers Trials, variation across runs, isolated eval environments, and evaluating outcomes rather than unnecessarily rigid trajectories. Demystifying evals for AI agents
• OpenAI: A practical guide to building AI agents — Describes Agents as systems in which an LLM steers workflow execution and dynamic Tool selection. A practical guide to building AI agents