The Architecture Nobody Wants to Talk About

It rarely takes more than the first week.

You are new to the project, working your way through onboarding tickets, build processes, names, and responsibilities, when someone casually says the first sentence:

“Whatever you do, don’t touch Service X.”

You nod, remember the name, and at that moment you have no idea how often you will hear that sentence over the coming months. Sometimes it is said with a small smile. Sometimes it sounds more like a warning.

A few weeks later, you experience your first release. That, too, feels different from what you may know from other projects.

Certain people had better be available that day. After deployment, nobody simply moves on to the next ticket. People watch the logs, check critical flows, and wait a little longer than usual before someone finally says:

“Looks good.”

Not necessarily because anyone knows that something must break. But because nobody can say with complete confidence what might break.

And eventually, the other sentence is spoken. Maybe by management. Maybe by a long-time specialist.

“I could have done that in a day.”

And that may even be true.

From a functional perspective, the change is small. A new field. An additional condition. A small adjustment to an existing rule. For someone who has known exactly this part of the system for years, it really may be a matter of hours.

Other developers need several days or weeks.

From the outside, an obvious picture quickly emerges:

The system is not the problem. The developers are too slow.

Except that this picture does not quite fit with what everyone in the project already knows.

The problematic service is known. The dangerous areas are known. Testers know the typical side effects. Product owners know which seemingly small tickets regularly escalate. Developers know which changes require an unusual amount of analysis. And managers have probably seen more than one presentation about Technical Debt.

The problem is not hidden.

So why does a state like this remain remarkably stable for years?

That is what this article is about.

Not yet about how exactly a Big Ball of Mud is defined technically. That comes in the next part.

For now, I am interested in something else:

Why do organizations develop the ability to keep working with a structurally problematic system instead of fundamentally changing that state?

Not a State of Emergency, but a Stabilized State

When people think of a Big Ball of Mud, they often imagine chaos: constant outages, firefighting, production incidents, a system on the verge of collapse.

In reality, the situation is often far less spectacular.

Daily work continues. Features are delivered. Releases happen. Production incidents are resolved. Customers keep using the product.

The system works.

It is just that keeping it working increasingly requires things that are not part of the actual business change:

• additional tests,
• additional reviews,
• additional coordination,
• specialist knowledge,
• informal rules,
• special release processes,
• certain people who are brought in as a precaution,
• and areas everyone knows should preferably not be touched.

At that point, an organization has begun adapting itself to its architecture.

There is nothing inherently unprofessional about that. Quite the opposite.

If changes frequently cause unexpected side effects, additional tests make sense. If only two people truly understand a critical area, asking one of them before making a change is reasonable. If releases are risky, a particularly careful release process is sensible. If a particular service causes problems whenever it changes, caution is justified.

The problem does not necessarily lie in any one of these decisions.

The problem can emerge from the combined effect of many locally reasonable decisions.

Every successful compensation makes it a little easier to keep living with the structural condition:

• The additional test prevents the defect.
• The review catches the side effect.
• The specialist finds the production issue.
• The release succeeds.
• The workaround works.

From the perspective of that day, every one of these measures was successful.

And that is precisely where the paradox lies:

The organization becomes increasingly good at compensating for the consequences of its architecture.

At the same time, the immediate pressure to change the architecture itself can decrease.

The compensation works — and that is precisely what can help the structural condition persist.

When Deviation Becomes Part of Normal Operations

An interesting organizational-sociology perspective on this comes from Diane Vaughan.

Vaughan studied NASA’s decision-making processes before the Challenger disaster in 1986. In her analysis, she described how technical deviations could gradually become part of what an organization considered acceptable operations.

Deviations occurred. They did not immediately lead to disaster. They were analyzed, explained, and incorporated into existing decision-making processes. Further missions succeeded.

Over time, this could change what the organization perceived as an acceptable level of risk.

Vaughan coined the term Normalization of Deviance for this process.

An important distinction is necessary here.

Vaughan’s analysis is not about Big Balls of Mud. And I have not found robust empirical research from which one could seriously conclude:

Big Balls of Mud are a case of Normalization of Deviance.

That would be too strong a transfer.

But the concept offers an interesting lens through which to view certain organizational behaviors.

Jeffrey Pinto later used interviews with 21 project managers to examine how Normalization of Deviance can also appear in project organizations. His work covered planning, governance, and problematic practices that gradually become a seemingly normal part of project work.

Applied cautiously to software, this at least allows us to ask a question:

What happens when a release only works reliably because of three additional safety measures?

At first, that is exceptional. The next time, the same measures are repeated. Then again.

Eventually, nobody asks:

Why does our release need this special treatment in the first place?

The operational question becomes:

Who is doing the additional check this time?

In the same way, this:

This service has a structural problem.

can gradually become:

We just have to be careful with this service.

The problem is still known. But the response to it has become part of normal operations.

The organization has learned to live with it.

Technical Debt Creates Work That Does Not Exist in the Feature

There is much more direct software-engineering research on the additional effort created by systems like these.

Terese Besker, Antonio Martini, and Jan Bosch studied how Technical Debt affects developers’ day-to-day work.

In their longitudinal study, 43 developers documented their work; the study was complemented by interviews and an independent replication.

On average, the developers reported losing around 23 percent of their development time because of Technical Debt.

Additional effort was particularly common in testing, but also appeared in analysis and refactoring. The study also found that existing Technical Debt sometimes forced developers to introduce new Technical Debt.

Those 23 percent do not describe a Big Ball of Mud. The study looks at Technical Debt in general — and Technical Debt is hardly completely avoidable in long-lived software systems. Nor does it have to be inherently problematic: technical debt can be taken on deliberately when the short-term benefit justifies the expected future cost. For this series, the exact percentage therefore matters less than the mechanism Besker and colleagues observed: Technical Debt consumes working time that is not part of the functional change itself. In a Big Ball of Mud, I am interested in the point at which this additional effort is no longer occasional but begins to dominate a significant part of everyday development work. To stay with the debt metaphor: the problem is no longer that debt exists. The interest payments begin to consume the actual development capacity.

That allows us to ask two questions that sound similar but are fundamentally different.

The first is:

Why do our developers need six weeks for this change?

The second is:

Why does our system require six weeks of human effort for a functionally small change?

The functional size of a change tells us little about how much work a difficult-to-change system will turn it into.

The first question quickly looks for the cause in people: experience, motivation, productivity, seniority.

The second looks at the system.

Perhaps nobody spends six weeks writing feature code. Instead, people analyze impact, search for implicit dependencies, involve colleagues, extend regression tests, reconstruct historical edge cases, inspect areas that should have nothing to do with the change, and safeguard additional release risks.

All of that takes time.

And much of it is entirely professional work in a system that is difficult to change safely.

A slow development process can therefore arise, paradoxically, precisely because developers are trying to act responsibly in a risky system.

It is still expensive. But the cause looks different.

The System Produces Its Own Experts

The longer a complex system exists, the more valuable experience with it becomes.

That is normal.

Software development is knowledge work. Not every relevant piece of knowledge is documented or can be fully reconstructed from the code.

It becomes problematic when experiential knowledge increasingly has to compensate for the system’s lack of structural comprehensibility.

At that point, a person does not merely know a lot about a business area. They may also know:

• which seemingly unrelated component must be tested after change X,
• which old edge case only occurs for one particular data constellation,
• why a strange line of code should probably not be removed,
• which service is officially responsible but is actually bypassed,
• and which error message really points to a completely different problem.

Part of the architecture then no longer exists only in the repository.

It exists in people’s heads.

Software-engineering research attempts to make this concentration of knowledge tangible through concepts such as the Truck Factor or Bus Factor.

Avelino, Passos, Hora, and Valente analyzed 133 popular GitHub projects. In 65 percent of the systems they studied, the estimated Truck Factor was no more than two people. The authors additionally validated their automated estimates with developers from 67 of the projects.

That does not mean every project with a low Truck Factor is a Big Ball of Mud.

Even excellently structured systems can concentrate critical knowledge in a small number of people.

But it does show:

A software project’s dependence on a small number of knowledge holders is a real and measurable risk.

In a system that is difficult to understand, that same knowledge can become the most important mechanism keeping the system apparently manageable.

The Hero and the Wrong Signal

The specialist who holds this knowledge is not automatically the villain of the story.

The opposite may be true.

Perhaps this is the person who has taken responsibility for years, finds a production defect in twenty minutes after others have searched for a day, remains available during critical releases, takes on difficult changes, and helps colleagues.

Again and again.

When this person says:

“I could have done that in a day.”

that can be factually correct.

But it does not automatically follow that:

This change takes one day.

It may simply mean:

One particular person with years of accumulated, partly implicit knowledge can make this change in one day.

That is different.

The organization does not necessarily possess the general capability to make this change within a day.

It possesses one person who can.

And that means the individual strength of a specialist can produce a misleading organizational signal.

The system appears manageable. After all, it keeps being managed. The incident is resolved. The feature is finished. The release is saved.

Three different lessons can then be learned at the same time:

• The organization: “When things get critical, we bring in Person X.”
• The team: “We should not work in this area without Person X.”
• Person X: “If I do not step in, things may go wrong.”

None of these reactions requires egoism or a desire for power. They can arise entirely from experience, responsibility, and helpfulness.

That is precisely why this dynamic is so persistent.

The exceptional performance of individual people can compensate for structural deficiencies in the short term. And in doing so, it makes those deficiencies less immediately painful for the organization.

What this dependence eventually does to key people themselves and to their teams deserves an article of its own.

For now, only one point matters:

A system can compensate for structural weaknesses through human expertise — and thereby appear more stable than it actually is.

Costs That Never Appear on an Invoice

Technical Debt has another unpleasant characteristic.

Its costs rarely appear as a single line item.

There is usually no invoice containing:

Architectural erosion: €417,000

Instead, the costs are distributed:

• One developer spends an extra day analyzing.
• Two colleagues sit in an additional meeting.
• Another regression test is written.
• Another review takes place before the release.
• A specialist is pulled away from another initiative.
• A change is split across two releases as a precaution.
• Onboarding takes longer.
• A refactoring is postponed again.

Every individual activity can be justified. And every individual activity may be sensible.

Only in aggregate does the economic effect emerge.

In their research, Besker, Martini, and Bosch accordingly describe Architectural Technical Debt not merely as a question of poor code quality, but as something that can impair maintenance and evolution over the long term.

The invisibility of these costs matters organizationally.

New features have tickets. Projects have budgets. Migrations have business cases.

The two additional hours of analysis required today because an old architectural boundary no longer exists usually have none of those things.

They disappear into normal development work.

And when costs like these disappear a little bit everywhere, a structural problem can look like a collection of individual productivity problems for a very long time.

Why You Cannot Simply Pay Off the Debt

At this point, it would be easy to reduce the story to short-sighted management.

That would not do reality justice.

Technical Debt can be taken on deliberately. A pragmatic technical solution may be economically sensible today if it gets an important product to market faster.

Even in a heavily eroded system, the alternative to continuing operations is not free.

The system has customers. It contains data. Other systems depend on it. It implements business processes. It may have to satisfy regulatory requirements. It has to keep running during a migration. And nobody can guarantee that a rewrite will be faster, cheaper, or better in the end.

The decision therefore is rarely:

Bad architecture or good architecture?

It is more often:

Do we spend the next few months on a feature with visible customer value — or on a structural improvement whose benefit is harder to make immediately visible?

That is a real economic decision.

Freire and colleagues surveyed 653 software practitioners from six countries about why Technical Debt items are not paid off.

The reasons were often not purely technical. The authors identified numerous obstacles; reasons for non-payment were predominantly associated with planning and management.

That does not imply:

Management prevents good software.

It shows something else: Technical Debt does not disappear through better programming alone.

It competes with planning, resources, and other organizational goals.

The technical question becomes an economic one.

And that point will become considerably more important later in this series.

Sometimes Organizations Keep Investing

Another possible explanatory framework comes from research on Escalation of Commitment.

As early as 2000, Mark Keil, Joan Mann, and Arun Rai studied software and information-systems projects in which organizations continued to invest resources despite problematic developments.

The authors did not find one simple mechanism. Different theoretical explanations helped distinguish escalating projects from non-escalating ones.

Again, caution matters.

A decades-old productive system is not the same thing as a failed software project. And it would be too simplistic to explain the continued operation of a legacy system exclusively through sunk costs.

There are real switching costs, real modernization risks, real dependencies, and sometimes no realistic alternative.

The more interesting lesson from this research is another one:

The fact that a path has become problematic does not automatically mean that leaving that path is the more rational decision.

Organizations can therefore remain in situations for a very long time even though nobody considers them ideal.

Not necessarily because nobody understands the problem, but because every available alternative also carries costs and risks.

That is what makes a Big Ball of Mud so difficult to address economically.

The question is not only:

How expensive is it to keep operating this system?

But also:

How expensive and risky is it to stop?

When Everyone Knows and Nobody Talks About It Anymore

Another question remains.

What happens when the problems have long been known?

Perhaps Service X has already been discussed five times. Perhaps there are architecture diagrams, tickets, workshops, and refactoring proposals. Perhaps the problem has been prioritized repeatedly and then postponed again.

At some point, the question is no longer whether people recognize the problem, but whether they continue to raise it.

Organizational-psychology research examines exactly this under the concepts of Employee Voice and Employee Silence.

Elizabeth Morrison describes Voice as the voluntary communication of suggestions, concerns, or information about problems to people in higher organizational positions. Silence refers to withholding such information.

One point is particularly important:

Silence does not mean that knowledge is absent.

People can understand a problem very well and still decide not to raise it again.

Whether employees speak up depends, among other things, on whether they perceive doing so as safe and potentially effective.

Again, this is not research specifically about Big Balls of Mud. But it provides a useful interpretive framework for an observation that can easily emerge in problematic projects.

At first, people say:

“We really need to solve this properly.”

Then perhaps:

“We should genuinely make time for this.”

Later:

“The issue is already in the backlog.”

And eventually:

“There is no point anyway.”

The problem has not disappeared.

Perhaps only the expectation has changed that raising it again will make any difference.

A technical insight can therefore turn into silent organizational knowledge.

Everyone knows Service X.

And precisely because everyone knows it, hardly anyone has to explain anymore why Service X is a problem.

“We Have Technical Debt” Is the Easier Sentence

Technical Debt is now a completely established concept.

Almost every larger software system has some.

Technical Debt can be identified, prioritized, documented, added to a backlog, and perhaps even measured.

That is useful.

Still, there is a linguistic difference that interests me in heavily eroded systems.

The sentence:

“We have a lot of Technical Debt.”

is relatively easy to say.

The sentence:

“When we change a central part of our system, we can no longer reliably predict the consequences.”

is more uncomfortable.

Because the second sentence is no longer only about code quality.

It is about control, predictability, risk, and the organization’s ability to deliberately change its own product.

At that point, a technical topic becomes a governance question.

I have not found robust research showing that organizations deliberately use the term Technical Debt to linguistically obscure such a loss of control.

I do not want to claim that.

But the conceptual distinction matters.

A system can have Technical Debt and still be highly manageable.

With a Big Ball of Mud, I am interested in a further question:

How confidently can the organization still say what consequences a change will have?

Once that predictability is lost, saying “we have Technical Debt” eventually stops being a sufficient description.

At that point, part of the system’s structural control has been lost.

And admitting that is organizationally much more uncomfortable than adding another ticket to the Technical Debt backlog.

The Paradox of Successful Compensation

That brings us back to the beginning.

Service X was not changed without consulting the specialist first. The additional review happened. The regression tests ran. The specialist was available during deployment. The workaround worked. The release succeeded.

From the perspective of that day, this was a success.

And it genuinely was.

Customers could continue working. The company could deliver. A risk was managed.

That is precisely why I do not think situations like these are well explained by simple blame.

Developers can act professionally. Architects can warn early and clearly. Specialists can be extraordinarily helpful. Product owners can pursue legitimate customer value. Management can prioritize in economically understandable ways.

And yet all of these locally reasonable decisions can collectively stabilize a condition that nobody consciously chose.

Perhaps that is one of the most interesting characteristics of a Big Ball of Mud:

It does not necessarily survive because nobody recognizes its problems.

It can survive because an organization becomes increasingly good at working despite those problems.

The compensation mechanisms work:

• Additional tests reduce risk.
• Specialist knowledge accelerates critical changes.
• Reviews prevent defects.
• Workarounds keep operations running.
• Release rituals create safety.
• Additional coordination compensates for missing structural boundaries.

Every one of these measures can be sensible.

And every successful compensation can also remove a little of the immediate pain that would otherwise force fundamental change.

This is not a law of nature.

Not every organization develops this dynamic. Not every compensation mechanism worsens Technical Debt. And not every large, old, or complex system is a Big Ball of Mud.

But several independent lines of research make at least parts of this dynamic plausible:

• Technical-Debt research shows additional development effort and difficulties in paying debt down.
• Research on knowledge concentration shows the real dependence on individual knowledge holders.
• Organizational research shows how problematic practices can become normalized.
• Research on Employee Voice and Silence shows that known problems are not necessarily raised indefinitely.
• Research on Escalation of Commitment shows that organizations can continue problematic technical paths for a variety of psychological, organizational, and economic reasons.

In my own professional life, I have experienced many of these patterns at the same time.

That does not turn my experience into evidence.

It is the reason the underlying question interests me.

Not:

Who broke this system?

But:

How can an obviously problematic system keep functioning for years?

One possible answer is:

Precisely because people have learned to compensate for its problems successfully every single day.

Before we can talk about how to deal with such a state — or even how to get out of it — we first have to define more precisely what we are actually talking about.

Because a large system is not automatically a Big Ball of Mud. Neither is an old system. And a monolith certainly is not one by definition.

That is what the next article is about:

What Exactly Is a Big Ball of Mud?

---

Sources

Avelino, Guilherme Amaral; Passos, Leonardo; Hora, André; Valente, Marco Tulio: A Novel Approach for Estimating Truck Factors. 24th IEEE International Conference on Program Comprehension (ICPC), 2016. DOI: 10.1109/ICPC.2016.7503718.

Besker, Terese; Martini, Antonio; Bosch, Jan: Managing Architectural Technical Debt: A Unified Model and Systematic Literature Review. Journal of Systems and Software 135, 2018, pp. 1–16. DOI: 10.1016/j.jss.2017.09.025.

Besker, Terese; Martini, Antonio; Bosch, Jan: Software Developer Productivity Loss Due to Technical Debt – A Replication and Extension Study Examining Developers’ Development Work. Journal of Systems and Software 156, 2019, pp. 41–61. DOI: 10.1016/j.jss.2019.06.004.

Freire, Sávio; Rios, Nicolli; Pérez, Boris; Castellanos, Camilo; Correal, Darío; Ramač, Robert; Mandić, Vladimir; Taušan, Nebojša; López Herrera, Gustavo; Pacheco Hernández, Alexia; Mendonça, Manoel; Falessi, Davide; Izurieta, Clemente; Seaman, Carolyn; Spínola, Rodrigo: Software Practitioners’ Point of View on Technical Debt Payment. Journal of Systems and Software 196, 2023, 111554. DOI: 10.1016/j.jss.2022.111554.

Keil, Mark; Mann, Joan; Rai, Arun: Why Software Projects Escalate: An Empirical Analysis and Test of Four Theoretical Models. MIS Quarterly 24(4), 2000, pp. 631–664. DOI: 10.2307/3250950.

Morrison, Elizabeth W.: Employee Voice and Silence. Annual Review of Organizational Psychology and Organizational Behavior 1, 2014, pp. 173–197. DOI: 10.1146/annurev-orgpsych-031413-091328.

Pinto, Jeffrey K.: Project Management, Governance, and the Normalization of Deviance. International Journal of Project Management 32(3), 2014, pp. 376–387. DOI: 10.1016/j.ijproman.2013.06.004.

Vaughan, Diane: The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA. University of Chicago Press, Chicago, 1996.