What Changed in Three Years with LLMs

When I started seriously experimenting with LLMs for software development, my most important question was fairly simple:

Can this tool actually help me in day-to-day software development?

My answer in 2023 was somewhere between “sometimes astonishing” and “still fairly exhausting.” The models could explain things convincingly, generate code, and produce technical answers that looked surprisingly competent at first glance. At the same time, they produced a lot of noise: APIs were invented, framework features were assumed incorrectly, and current developments were mixed with outdated knowledge. Especially with fast-moving technologies, you had to know the domain well enough to recognize when an answer merely sounded plausible.

For small, clearly bounded tasks, that was already useful. Explaining a function, refactoring a method, or generating a manageable example often worked surprisingly well. Investigating an unfamiliar, grown application, finding the relevant parts, understanding the existing architecture, implementing changes across several layers, running tests, and handling errors autonomously was a different category entirely.

Today, in 2026, I work with these tools very differently. Not because they have suddenly become flawless, but because the kinds of tasks I can usefully delegate to them have changed.

For me, that development looked roughly like this:

This is not a universal evolution curve of artificial intelligence. It is the evolution of my own usage. And that experience is exactly where this series began.

Two very different realities

My experience with LLMs comes from two contexts that could hardly be more different. In my professional work, I deal with large software systems, some of them in environments with highly sensitive data. There, the question “What can the agent technically do?” is not enough. Just as important is what information a system is allowed to see, which tools it may access, which decisions it should make on its own, how changes can be traced and reviewed afterwards, and who ultimately remains responsible for the result.

An agent that could technically analyze an entire application is therefore not automatically allowed to analyze the entire application. Information boundaries, privacy, governance, security, and traceability are just as much part of professional agentic work as the model itself.

In my private projects, I had much more freedom. Over the past few years, I have used LLMs and coding agents in almost 40 projects of different sizes and levels of complexity – from small experiments to frontends, backends, infrastructure, and applications in which AI itself was part of the product.

That gave me room to experiment much more aggressively. I could give agents access to complete repositories, change agent files and skills, use memories, refine specs, switch models, or observe what happened when context was expanded or reduced. This is not a scientific sample. But it is enough practical work by now to see a clear difference compared with my first experiments.

At first, the language was the most impressive part

The early broadly available LLMs had an interesting effect: they often appeared competent faster than they actually were in many technical situations. The language was convincing, the proposed solution sounded plausible, and the code looked like something a developer might genuinely have written.

That was part of the problem. If you knew the framework or API in question, you usually noticed fairly quickly when a function did not exist, a parameter was wrong, or the suggested solution was based on knowledge that had already become outdated. If you did not know, you still received a very confident answer.

That did not make the systems useless to me. Quite the opposite: even then, they were good sparring partners for small, clearly bounded problems. They could generate variants, reduce boilerplate, explain code, and help when getting started with unfamiliar technologies. But the actual integration work largely remained with the human developer.

A typical workflow looked more like this:

AI was a tool inside a development process still driven by a human. And the crucial technical work was often not writing a single function. The harder part was finding out where a change belonged, which parts of the system were affected, and what side effects might follow. That was precisely where the early systems felt much more limited to me.

The most important progress was not better code

At some point, my perception shifted. Of course code generation improved as well. But something else became more important to me:

LLMs became more valuable to my work because they got better at reading existing code.

Suddenly, the interesting unit was no longer just one function. Several files could be considered together, dependencies could be recognized, unfamiliar features could be explored, bugs could be traced across layers, existing patterns became visible, and different solution paths could be compared against the architecture that was already there.

That also changed how useful LLMs became when working with unfamiliar technologies. Previously, a new library or an unknown subsystem often meant first reading documentation, finding entry points, searching through code, and slowly building a mental model from examples. Today, an LLM can significantly accelerate that first orientation step.

It does not do the thinking for me. It narrows the search space. An agent can find entry points, collect relevant files, follow dependencies, formulate hypotheses about a bug, or prepare a proof of concept. The result is not complete system understanding, but the path toward my own understanding can be much shorter.

For me, that development mattered far more than whether the next generated method became a little more elegant.

From assistant to agent

Tool use changed the workflow once again. A classic chat initially knows what I give it. A coding agent can – within its permissions – search the repository, read and modify files, run tests, see compiler or linter errors, and react to the results.

That creates a different workflow:

That may sound like a gradual difference. In practice, it is a substantial one.

In my own projects, it is now realistic to describe a clearly bounded feature to a coding agent and receive a coherent implementation across several technical layers. Depending on the system, that can start at the data model or backend and continue through the API and state all the way to the UI.

That does not work for every task or every model. It requires sufficient context, does not automatically result in a good solution, and above all does not replace review. But the fact that such a workflow has become practically useful at all fundamentally distinguishes my work today from my earliest experiments with LLMs.

Research is beginning to examine this distinction between classic assistance systems and more agentic tools in greater detail. A controlled CHI study on copilots and coding agents, for example, reports that more automated agents were able to take over tasks that classic assistance systems did not go as far with, reducing human effort in the process. At the same time, this created new requirements around understanding and evaluating what an agent is actually doing. [1]

That matches my practical experience well: more autonomy does not necessarily remove human work. It changes it.

An agent does not need to be perfect to be valuable

Part of the debate around AI in software development uses a strange benchmark, in my view. The question often becomes whether AI can fully and autonomously replace an experienced developer. If the answer today is no, the technology is quickly dismissed as overrated.

For my daily work, a different question matters much more:

How much high-quality work can I reliably take responsibility for with these tools?

A coding agent does not need to be perfect to be extremely useful. If it accelerates the exploration of an unfamiliar area, prepares a proof of concept in an hour instead of an afternoon, or performs repetitive changes consistently across many files, that already creates substantial value. The same is true when it traces a bug across several layers, works out plausible causes, or largely implements a well-specified change whose architecture and result I review afterwards.

The important point is not the absence of human work. What matters is how the work I spend my time on shifts.

My own perception is clear: for many tasks, I am considerably faster with modern LLMs and coding agents. But that does not justify a universal claim about productivity. The research so far paints a much more differentiated picture.

Three randomized field experiments with a total of 4,867 developers found a combined increase of roughly 26 percent in completed tasks for developers using AI coding assistance in 2026. Larger effects were observed especially among less experienced developers. The studies examined coding assistants, however, not the full range of today’s agentic workflows. [2]

By contrast, a widely discussed experiment with 16 experienced open-source developers examined 246 real tasks in projects they had known for years. In that setting, being allowed to use the AI tools available at the time increased task completion time by 19 percent. Particularly interesting was that even after the experiment, the developers still believed they had been faster with AI. [3]

A 2026 meta-analysis covering 23 studies again found a moderate positive productivity effect, but with substantial variation between the situations studied. Effects tended to be larger in controlled experiments than in open-source and enterprise contexts. [4]

To me, that is not a contradiction that should be resolved too quickly. It may be the more interesting finding: AI assistance can improve productivity, but the size and perhaps even the direction of that effect depend heavily on context, task, tooling, and the developer.

That is less spectacular than “AI makes developers 50 percent faster.” As an engineering statement, it is much more useful.

Fast-generated code is not automatically a good solution

Another reflex has not changed for me because of LLMs: I do not accept code merely because it works. That was true before, and with coding agents it matters even more.

An agent can make a test pass and still choose the wrong system boundary. It can implement a requirement correctly while making an architectural decision that gets in the way of the next requirements. It can reproduce an existing workaround cleanly, create local consistency while reinforcing global inconsistency, or place a technically correct solution in the wrong part of the system.

Research on the quality of AI-generated code also provides reasons not to treat verification as an optional final step. A systematic review on the security of AI-generated code documents known weaknesses in generated solutions and highlights the importance of appropriate review and verification processes. [5]

That does not mean AI-generated code is insecure while human-written code is secure. That would obviously be too simple. Code does not gain quality because a human or a model authored it. It has to pass the same domain, architectural, functional, and security-related quality boundaries.

This is also where I repeatedly encounter an observation in my private projects that I find particularly interesting. In some of them, I work with very explicit constraints: architectural rules, agent files, skills, memories, examples, and clear boundaries for what should belong in components, application logic, state, or infrastructure. Even so, certain patterns reappear.

After several changes, for example, a component starts orchestrating a domain flow again. A clean reactive flow gains an imperative intermediate step. A layer boundary is relaxed locally because the immediate solution appears simpler. This does not necessarily happen right away, and not with every agent, but it happens often enough to be a recurring practical observation for me.

Personal observation: Explicit architectural rules substantially reduce problematic decisions. They do not automatically prevent coding agents from drifting back toward familiar, locally plausible patterns over longer stretches of work.

Why that happens is one of the questions this series will examine. It would be too easy to conclude from it that LLMs are fundamentally incapable of understanding architecture. The more interesting question is how the model, context, existing code, local examples, specification, and tooling jointly influence which solution appears plausible as the next step.

AI as a catalyst

Many of the problems we now discuss in connection with coding agents are surprisingly old software-engineering problems. Humans copy existing code, orient themselves around local examples, interpret unclear requirements differently, inherit workarounds, and build on decisions whose original context may have disappeared long ago.

Coding agents did not invent these mechanisms. But they can accelerate them dramatically.

Imagine that a system contains an unfortunate workaround. It works, it is close to the new task, and it looks like an established pattern. A later change follows it, causing the workaround to appear a second time. That makes it more visible in the repository and can make it look even more like an intentional convention when the next similar task arrives.

This resembles a broken-window effect. I am explicitly not claiming that this mechanism has already been established as a universal law for coding agents. For now, it is a hypothesis emerging from a recurring observation: if existing code is an important part of the context for new decisions, frequently visible local patterns may increase the likelihood of their own further reproduction.

The interesting part is that this dynamic is not necessarily negative. Good patterns can become more visible in exactly the same way, be reproduced, and thereby increase consistency across the system.

That is why I find the idea of AI as a catalyst or amplifier more useful than treating it as an entirely new cause of software problems.

AI does not invent many of software engineering’s familiar problems. It can drastically increase their speed and reach.

The same applies to good engineering practices. Architecture does not become less important. Its effects can simply become visible much faster.

Humans can still steer surprisingly effectively

Up to this point, it would be easy to get the impression that coding agents inevitably drift architecturally sooner or later and therefore need to be watched with permanent suspicion. That is not my experience.

The important point is that speed works in both directions. An agent can implement a great deal in a short time. An experienced developer can often recognize just as quickly that a solution is moving in an unfortunate direction. Then the entire change is not discarded. The spec is refined, a boundary is made more explicit, a pattern is named as a reference, or a wrong abstraction is removed. After that, the agent keeps working.

agent works quickly
↓
result is largely useful
↓
problematic direction is recognized early
↓
context or constraint is corrected
↓
agent continues implementation

For me, this is currently one of the most productive forms of agentic work. Not: “Do everything and tell me when you’re done.” More like: “Work largely independently inside a clearly defined space, while I pay particular attention to the decisions where that space might be crossed.”

That is why, as of 2026, I still see experienced developers with deep system understanding as a crucial part of professional agentic work. This is not a claim about what models will be able to do in five or ten years. It describes the situation I work with today.

The more implementation work I delegate to an agent, the more important it currently becomes to have someone who can judge whether the emerging solution actually fits the system.

This is also why repository-wide changes and complete software autonomy should not be treated as the same thing. Newer benchmarks are increasingly trying to evaluate not just individual code changes, but complete workflows involving environment setup, implementation, and verification. The 2026 SWE-Cycle preprint, for example, reports a substantial drop in performance when these previously separate tasks have to be completed as a full end-to-end process without human scaffolding. [6]

An agent can perform impressive, coherent changes today. That does not mean it can autonomously take over an arbitrary software project. There is a large space between those two statements – and that is where I currently find the practical work most interesting.

The real question has changed

In 2023, I asked myself:

Can AI actually program in a useful way?

That question hardly interests me anymore. Of course it can program – not always correctly, not always well, and not without sufficient context. But the existence of useful code generation is no longer an open question for me.

The more interesting question has become:

How do we use this speed in a way that produces good software?

That opens a much larger field. We need to understand why results are not always reproducible, what context really means, what memory and skills can and cannot do, and what benchmarks such as SWE-bench actually measure. We need to ask why agents can drift despite explicit rules, how precise requirements have to become when implementation gets cheaper, how architecture constrains the available solution space, and how verification should work when software can be generated faster than humans can comfortably review it.

There are economic and organizational questions as well. When is a large frontier model actually necessary? When is a smaller or local model enough? What does agentic work cost once we count not just tokens, but review, infrastructure, and wrong decisions? And what happens to our own roles when more and more implementation work can be delegated?

This series will examine some of those questions. Not because AI has suddenly replaced every old problem in software engineering, but because it makes many of them visible under new conditions.

An article from 2023 would already be historical

If I had written this article in 2023, a significant part of it would already be a historical snapshot today. That is remarkable, but it is no reason to become just as careless in the opposite direction.

We do not know how large the next leaps will be. We do not know where additional scaling will produce diminishing returns or which of today’s limitations will disappear with better models. Nor do we know which problems will turn out to be structural properties of the approaches we use.

That is why I consider two statements important at the same time:

We should not turn the limitations of 2026 into eternal limitations.

And equally:

We should not extrapolate three years of unusually rapid progress into an unlimited future.

Both would be speculation.

For software engineering, the current state is already more than enough to matter. The technology is already capable enough that we need to learn how to work with it professionally.

Why this series exists

I therefore do not want to write an AI marketing series, nor a collection of reasons why all of this is overrated. A model leaderboard does not interest me very much either. It would probably be outdated before the series was finished.

What interests me is the engineering question: What actually works today? Where does it work unexpectedly well? Where do risks emerge? Which old problems are simply accelerated? Which workflows help, which rules help less than expected, what can we measure, and what remains experiential knowledge? Most importantly, which principles still make sense if the next generation of models is once again significantly better than the current one?

My perspective on LLMs has changed several times over the past few years: from an impressive text generator, to a useful coding assistant, to a tool for analysis and exploration, and finally to agents that can, under suitable conditions, take over substantial parts of a concrete implementation.

That is neither “just noise” nor the autonomous software developer that can reliably take over a complex system without human guidance. Both descriptions have become too convenient for me.

The reality in between is much more interesting – and already practical enough that software engineering needs an answer to it.

This series is therefore primarily a snapshot:

observe, understand, experiment, verify – and derive the most robust engineering principles we can from the results.

Before we can examine why agents sometimes make surprisingly good decisions and sometimes surprisingly strange ones, however, we need to take a step back.

What actually happens inside an LLM? What do training and inference mean? What is a token, what is context, and what do we actually mean when we say an output is probabilistic?

That is where the next article begins.

Sources

[1] Valerie Chen, Ameet Talwalkar, Robert Brennan, Graham Neubig: Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, 2026. DOI: 10.1145/3772318.3790850.

[2] Kevin Zheyuan Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, Tobias Salz: The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science, 2026. DOI: 10.1287/mnsc.2025.00535.

[3] Joel Becker, Nate Rush, Elizabeth Barnes, David Rein: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. Preprint, 2025. arXiv: 2507.09089. DOI: 10.48550/arXiv.2507.09089.

[4] Sebastian Maier, Moritz Gunzenhäuser, J. Schweisthal, Manuel Schneider, S. Feuerriegel: A meta-analysis of the effect of generative AI on productivity and learning in programming. Preprint, 2026. arXiv: 2605.04779.

[5] Claudia Negri-Ribalta, Rémi Géraud-Stewart, Anastasia Sergeeva, Gabriele Lenzini: A systematic literature review on the impact of AI models on the security of code generation. Frontiers in Big Data 7, 2024. DOI: 10.3389/fdata.2024.1386720.

[6] Hao Guan, Lingyue Fu, Shao Zhang, Yaoming Zhu, Kangning Zhang, Lin Qiu, Xunliang Cai, Xuezhi Cao, Weiwen Liu, Weinan Zhang, Yong Yu: SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle. Preprint, 2026. arXiv:2605.13139.