Notes on Systems

The Agent that doesn't need a rewrite

Tue, 28 Jul 2026 12:00:00 GMT

Every stalled agent project I have looked at contains a version of the same sentence: "we need to modernize the platform first."

The agent was approved in January. By March it is a replatforming project with an agent attached. By budget review there is a half-finished service extraction, no agent, and an executive asking what the money bought. I wrote earlier this year about why agent projects get canceled. This is one of the quieter ways: the agent never ships because it got in line behind a rewrite.

Underneath is a premise that agents are a new kind of software deserving a new kind of architecture. Mostly they are not. The agent that survives its first year in production is usually a thin layer over the system you already have. Your ten-year-old codebase is not the obstacle. It is the asset.

Why the rewrite instinct fires

The reasons are human, and they deserve naming rather than mockery.

Every agent framework demo starts from an empty repository, so the imagination starts there too. Vendor decks sell platforms, because platforms are what vendors price by. And somewhere on the team is an engineer who has wanted to replace that legacy service for years. The agent is the first budget line that might pay for it.

That last reason is the dangerous one, because nobody says it out loud. The rewrite arrives dressed as a prerequisite.

What an agent actually needs from your system

Strip the vocabulary and ask what an agent requires. It needs named operations it can call, with typed inputs and typed results. It needs those operations to check permissions and record what happened.

That is a tool schema. You already have one. It is called your service layer.

The controller that has validated form submissions for eight years can validate an agent call. The authorization check, the audit log, the rate limit: all of it transfers. In this story that code is not technical debt. It is precisely the hardening an agent needs, and precisely what a rewrite throws away and then rebuilds under deadline pressure, minus the eight years of production traffic that made it trustworthy.

So the integration most teams actually need is unglamorous. A thin schema that maps tool names onto existing endpoints, with descriptions written well enough that a model can choose between them. Days of work. Weeks at the outside.

The two things you do have to build

Two things do not transfer, and they are where the real engineering lives.

The first is idempotency. Agents retry. For a human, submitting twice was an edge case worth a confirmation dialog. For an agent, retrying is how it recovers from ambiguity, which makes it the default path, not the exception. An operation that creates an invoice must be safe to call twice with the same key and still produce one invoice. If your write paths do not accept an idempotency key, that is the first change worth making. Notice that it makes the system better whether or not the agent ever ships.

The second is bounds. A person gives up after the third failure. An agent will keep trying until the rate limiter or the finance team stops it. Every agent-callable write needs a budget: attempts, spend, blast radius. When the budget is exhausted, the correct behavior is boring. Stop, record what happened, hand the case to a person. The teams that skip this learn it from an invoice, and the lesson costs more than the engineering would have.

Reads and writes deserve different generosity here. Give the agent broad read access and let it be wrong in drafts. Gate every write behind the narrow, keyed, bounded path above. Most of an agent's usefulness lives in reads anyway.

I keep relearning this with my own agents. The setup I run coding agents in is not a platform. It is a set of plain files describing what already exists: which commands are safe, what each one assumes, where the boundaries are. Every meaningful improvement in the past months came from making those descriptions truer, not from new infrastructure. The system was never the problem. The description of the system was.

When the rewrite is real

Sometimes there is no seam to attach to. The logic lives in stored procedures, or the codebase has no service layer at all, and the tool schema has nothing to map onto.

Even then the sequencing holds. Carve the one seam the agent needs. Do not build the platform you wish you had. A rewrite justified by an agent is still a rewrite, and it carries every rewrite risk at agent-project speed, reviewed monthly by people who were promised an agent. The narrow carve is reversible. The replatform is not.

The test

If the agent project were canceled tomorrow, would the work be worth keeping?

Idempotency keys on your write paths: yes. Operations described precisely enough for a model to use them, which is to say documented: yes. Bounded, audited, permission-checked mutations: yes. A half-finished replatform: no.

Build the things that are worth keeping either way. That is not a compromise position. It is what making an existing system agent-ready actually means, and it is weeks of work, not quarters.

brimtech works with engineering teams running AI inside systems they intend to keep. If your agent project has quietly become a rewrite project, the Agent Production Audit is where we usually start: two weeks, your real system, a measured baseline, and a ranked fix list. Essays like this one go to the list first, along with the production-readiness checklist.

Evals Are the Tests You Should Have Written First

Tue, 14 Jul 2026 12:00:00 GMT

Someone on your team changed a prompt last Friday. Nobody can say whether the agent got better or worse on Monday. Someone read a few outputs, they looked fine, and the change shipped.

For every other part of your codebase, the question has a boring answer: run the tests. For the LLM-powered part, often the part that touches customers directly, there is silence.

We have a name for this in ordinary software. It is called not having tests. We would not accept it from a payments integration or a migration script. But rename the component "AI" and the discipline quietly evaporates.

The industry has since invented a word for the fix: evals. The word is useful. The implication it carries is not, because it suggests a new practice, a specialist artifact, something you adopt later, when the AI part matures. Evals are the tests you should have written first. The renaming did not create a new discipline. It created permission to skip an old one.

I am not writing this from a distance. I run coding agents daily, and I have shipped prompt changes on nothing more than reading three outputs. I know the silence firsthand. That is what makes it worth naming.

Why the tests never got written

It is worth being fair about how this happened, because the reasons sound reasonable and they are still circulating.

The first reason is that the feature was born in a demo. A model call that produces a plausible paragraph looks finished in a way that a function returning null does not. Deterministic code announces its own incompleteness. Probabilistic output hides it. So the feature shipped at the moment it looked done, which is a different moment from when it was done.

The second reason is the non-determinism excuse: you cannot write a test when the output is different every time. This one deserves to be retired, because it misunderstands what tests assert. Good tests rarely assert exact output. They assert properties. The response cites a source that exists. The extracted total matches the invoice. The refusal triggers on the eleven things it must refuse on. None of these require determinism. They require you to say what "correct" means, which is the work the demo let you defer.

Engineers already know how to test non-deterministic systems. We do it for flaky networks, for load, for canary deploys. We assert on distributions and thresholds instead of exact values. Nothing about a language model makes that toolkit obsolete. What the model changes is the definition of a passing run, not the obligation to define one.

What actually changes

Honesty requires naming what is genuinely different, because three things are.

First, pass/fail becomes pass rate. A single green run of a probabilistic component tells you almost nothing; the same input can fail on the next call. So an eval runs a case many times and asserts a threshold, and the threshold is a business decision, not a technical one. Ninety-five percent sounds strong until you compound it: an agent whose every step succeeds 95% of the time completes a twenty-step workflow about a third of the time. Someone has to decide what rate is acceptable, per failure mode, knowing what a failure costs. That decision is the real deliverable. The test suite is just where it gets recorded.

Second, the suite decays. A conventional test suite is a snapshot: the code changes, you update the tests, truth holds still between commits. An eval suite is a subscription. The model provider ships an update you did not ask for. Your users drift into inputs nobody imagined. The prices, policies, and product names your prompts mention change quietly underneath them. An eval set written once and trusted forever converges on theater: green, comprehensive-looking, and asserting things that no longer matter. The maintenance is not overhead. It is the mechanism by which the suite keeps referring to reality.

Third, and this is the strongest objection to the framing of this essay: for open-ended output, deciding what counts as correct is itself new work. A generated summary has no invoice to check against. Teams solve this with model judges grading model output, and an uncalibrated judge is the original problem wearing a lab coat. Judges earn trust the same way the rest of the suite does: checked against human review until they agree often enough to be load-bearing. That calibration work is real and classical testing has no equivalent for it. But notice what it still is. A definition of correctness, written down late.

None of these differences argue for skipping evals. They argue for treating them as an operating practice rather than a one-time artifact, which is what testing was always supposed to be.

Writing the tests backward

If the agent is already in production without evals, there is a consolation: you are holding better test material than any team starting fresh.

Your production traces are a record of every way the system has actually failed. Most teams have the traces and have never grouped them. Do that. Pull the runs, sort the failures by mode, and turn each real failure into an eval case. If your logging cannot reconstruct why a run failed, that is finding zero, and it comes before anything else on this list.

Twenty cases mined from your own traces will catch more regressions than ten thousand rows of a generic benchmark, because they encode the specific ways your system meets your users, which is the only distribution that matters.

This is also the honest sequence. Evals invented in a vacuum encode what the team fears. Evals mined from traces encode what happens. The gap between those two lists is usually the most informative document the team has ever produced. In the LangChain survey of roughly 1,300 practitioners, quality and reliability was the most-cited barrier to shipping agents, and only about half reported having evaluation infrastructure at all. The barrier and the missing practice are the same item.

The cost is the point

Building evals slows the roadmap. Maintaining them slows it more. The threshold arguments alone can eat a sprint, and every week spent mining traces is a week not spent on features.

This is the same trade software settled once already. The old line about brakes applies: they are not there to slow the car down, they are the reason it can be driven fast. Skipping evals does not remove the testing from the system. It relocates it: to production, unstructured, run by your customers, with results reported to your support channel instead of your CI.

The prompt change last Friday either regressed something or it did not. That fact exists whether or not you have machinery to observe it. Evals are the decision to find out before someone else does.

brimtech works with engineering teams running AI inside systems they intend to keep. If your agent is in production and this essay describes it, the Agent Production Audit is where we usually start: two weeks, your real traces, a measured baseline, and a ranked fix list.

Why Agents Get Canceled

Wed, 01 Jul 2026 12:00:00 GMT

In July 2025, an AI coding agent deleted a live production database. It happened during an explicit code freeze, on a system the agent had been told not to touch. Then it told the engineer that rollback was impossible. That was also untrue. The data came back.

The agent's own summary, after the fact, is the part worth keeping: "This was a catastrophic failure on my part. I destroyed months of work in seconds."

It is easy to read that as a story about a model that wasn't ready. I think that reading is wrong, and that getting it wrong is expensive. The model did not lack intelligence. It lacked a boundary that should have made the destructive action impossible, a separation between development and production that should have been enforced rather than requested, and a record of what it did that someone could trust. Those are not properties of a model. They are properties of the system around it.

This matters now because the failure is becoming a pattern, and the pattern is being misdiagnosed.

Gartner expects more than 40% of agentic AI projects to be canceled by the end of 2027. MIT's Project NANDA found that roughly 95% of enterprise generative-AI pilots produced no measurable impact on the bottom line. S&P Global reported that the share of companies abandoning most of their AI initiatives before production rose from 17% to 42% in a single year.

Read quickly, those numbers sound like a verdict on the models. Read carefully, they are mostly a verdict on operations. Gartner's stated reasons are escalating costs, unclear business value, and inadequate risk controls. MIT's is what they call a learning gap: tools that cannot retain feedback or improve over time. None of those is a complaint about model quality. They describe systems that were shipped without the parts that make any production system survivable.

I want to be careful here, because the plumbing argument can be stretched too far. Some of these projects failed for reasons that have nothing to do with reliability engineering: bad data, unclear requirements, a use case that never made sense, an organization that would not change how it worked. Those are real, and they are not what this essay is about. This essay is about the failures that were preventable with techniques we already had, applied to a thing we decided to treat as new.

There are five places production agents fail quietly. None of them is exotic.

The first is evaluation. Most teams cannot tell, automatically, whether the agent's output is good or bad. So a quality regression ships, and the first signal is a customer. Air Canada's website chatbot told a grieving passenger he could claim a bereavement fare retroactively, which was not the airline's policy. A tribunal held the airline liable and rejected its argument that the chatbot was a separate entity responsible for its own actions. The damages were small. The precedent was not. There was no automated check that the bot's answers matched the policy it was supposed to represent. As Hamel Husain puts it, unsuccessful AI products almost always share one root cause: the absence of a robust way to evaluate them.

The second is observability. You cannot fix what you cannot see, and most agents run blind. Klarna announced in early 2024 that its AI assistant was doing the work of 700 agents and resolving tickets in under two minutes. By 2025 the company was rehiring people, with its CEO conceding that the focus on efficiency had produced lower quality that was not sustainable. The dashboards that showed resolution rate and handle time were real. They were also an average, and the average hid the distribution. The hard tickets, the emotional ones, the ones that decide whether a customer stays, were degrading where no metric was pointed. Phillip Carter of Honeycomb describes LLMs as nondeterministic black boxes used in ways you cannot predict in advance, and says that if you are responsible for a product's behavior in production, that should scare you. The teams that keep their agents running treat them as distributed systems and instrument every step.

The third is reversibility. The July database deletion is the clean example, but it has an older twin. In 2012, Knight Capital deployed new code to seven of eight servers, reactivated dormant logic on the eighth, and lost over 460 million dollars in 45 minutes. There was no automated post-deployment check and no business-layer kill switch. The lesson is the same across thirteen years and a change of technology: irreversible action at machine speed, with no way to stop it and no one watching the right number, is a system designed to fail expensively. Reversibility is not a feature you add later. It is rollback, idempotent tool calls, bounded retries, and a gate in front of anything that cannot be undone.

The fourth is autonomy boundaries. An agent should be able to do a known, enumerated set of things, and should have a defined way to refuse or escalate when it is out of its depth. A car dealership's chatbot was talked into agreeing to sell a Chevrolet Tahoe for one dollar, with, in the customer's words, no takesies-backsies. Cursor's support agent invented a subscription policy that did not exist to explain a bug, and users canceled over it. Neither failure required a smarter model. Both required a limit. Prompt injection sits at the top of the OWASP Top 10 for LLM applications for the second edition running, which is another way of saying that a system prompt is not a security boundary and was never going to be one.

The fifth is operational drift. An agent that works today will not necessarily work next quarter, because the inputs move, the model updates, and the context shifts underneath it. DPD's chatbot, after a routine system update, was provoked into swearing at a customer and writing a poem about how useless its own company was. New York City's official business chatbot confidently gave advice that was against the law, telling users that landlords could refuse housing vouchers and that businesses could go cash-free. Both had no scheduled re-evaluation, no gate that caught a behavior change before customers did, and, underneath that, no person whose job was to own the thing's reliability over time.

It is worth steel-manning the case against all of this, because two of the objections are good.

The first is that models are improving so fast that the reliability layer will be absorbed into them. There is something to this; each generation hallucinates less and follows instructions better. But reversibility, idempotency, scoped permissions, audit trails, and human checkpoints are properties of the system around the model, not the model. A smarter agent still should not have unscoped write access to your production database. The reliability layer is precisely the part that is not the model.

The second is that evals are theater. This is the sharpest objection, and it is partly right: bad evals create false confidence, which is worse than no confidence at all. A green test suite is a snapshot, and production is a stream. But the answer to bad evals is good evals plus observability, not the absence of both. Domain-specific checks built from real failures, judges calibrated against human review, an eval set that is refreshed from production traces. Evaluation and observability are complementary, and treating them as the same thing is the actual mistake.

If you doubt the pain is real, follow the money. There is now a funded category of companies that exist to sell exactly this plumbing. Braintrust raised at an 800-million-dollar valuation, LangChain at over a billion, with Arize, Langfuse, Galileo, Patronus, and the major observability vendors all building eval and tracing products for agents. Capital is not proof of correctness. But hundreds of millions of dollars moving toward one thesis, that agents in production have to be tested and monitored like real systems, is a strong signal about where the problem actually lives.

The conclusion is not that agents are too dangerous to ship. It is narrower and more useful than that. An agent is a production system that happens to be probabilistic, and it has to be operated like one: owned by a named person, observable in every run, reversible when it goes wrong, bounded in what it can do, and re-evaluated as the world around it changes. The teams in the surviving minority are not the ones with access to a better model. Everyone has roughly the same models. They are the ones operating better.

That is the whole difference, and it is the part you control.

What I Optimize for Now

Thu, 21 May 2026 12:00:00 GMT

What I optimize for has shifted, and the older preferences were defensible at the time.

I optimized for capability, because capability is the easiest thing to demonstrate. I optimized for elegance, because elegance reads well in review. I optimized for speed, because speed is the only metric a project tracks before it ships. Each was reasonable in the room where it was decided. None of them were quite what the system needed a few years in.

What changed was not a framework or a principle. It was rereading code I had written a few months earlier and noticing the cost of choices I had already half-forgotten.

Leave more things concrete

The first thing I would tell my younger self is to leave more things concrete.

The instinct to generalize is strong in engineering culture. It is rewarded in interviews. It is rewarded in code review. It is sometimes the difference between a junior and a senior label. But generalization paid up front draws against a future you have not yet seen.

Most of the abstractions I am embarrassed by were correct generalizations of the cases I knew at the time. They were not careless. They were premature.

A concrete piece of code can be generalized later, when the second and third use cases reveal what they actually share. A generalization written before those cases exist tends to fit the first case and constrain the next two.

I now prefer to copy three times before extracting once. The copies look ugly during review. They survive change better than the abstraction that would have replaced them.

Reversibility over correctness

The decisions that have hurt me most were not bad decisions. They were good decisions that could not be undone when context shifted.

A vendor lock-in chosen for speed. A schema choice that propagated through a hundred call sites before anyone noticed. A platform commitment that made sense for the team at the time and outlived three reorganizations.

Each was reasonable. Each became expensive in a way that had nothing to do with the original analysis.

I now apply a different filter before committing. Not “is this the best decision,” which is rarely answerable, but “if this turns out to be wrong, can the people who inherit it actually undo it.” That second question has changed more of my choices than any architectural principle.

The honest version of this filter is uncomfortable. It treats most of my own decisions as hypotheses I expect to be partly wrong about. It builds for a team that will know things I do not. It accepts that the present moment is rarely a good vantage point for long decisions.

Ambiguity is a defect

One of the largest unforced costs in long-lived systems is deferred clarity.

A team disagrees about what something means. The disagreement is uncomfortable. The conversation is postponed. The system is built. The disagreement settles into the code as ambiguity. A year later, two parts of the system operate on incompatible interpretations of the same word. Five years later, a new engineer is told that nobody really knows why it is this way.

The cost of clarifying early is small. It is one awkward meeting, one written paragraph, one explicit decision. The cost of clarifying late is structural. It is rewrites, migrations, and an accumulating tax on every new contributor.

I now treat unresolved ambiguity as a defect with the same weight as a bug. Not because clarity is virtuous, but because ambiguity compounds quietly and is invisible until it is structural.

The most useful question I ask in design conversations now is not “what should we build.” It is “what would we have to agree on for this to make sense in three years.” If the answer requires a long pause, the design is not ready.

Not a retreat into caution

This is not a retreat into caution.

I still build things. I still ship. I still make choices I will not be able to defend a decade from now. The change is not in tempo. It is in what I am willing to spend the future’s optionality on.

I am more willing to spend it on concrete things that work and can be replaced. I am less willing to spend it on abstractions that promise reuse. I am more willing to spend it on small commitments that can be undone. I am less willing to spend it on architectural bets that lock in a reading of a problem I do not yet fully understand.

The three preferences pull against each other often enough that this remains a posture, not a procedure. The work is in deciding, case by case, which one is paying more than it is asking.

The net effect is fewer impressive moments and more boring weeks. I have come to read that ratio as a leading indicator, with a caveat I have earned the hard way: boring is not the same as healthy. Sometimes uneventful weeks are a system whose quiet failure modes have not yet surfaced. The most durable systems I know look uninteresting in their day-to-day. The prior is not proof.

The posture I want to avoid

There is a posture I want to avoid here.

It would be easy to claim that what I optimize for now is what I should have always optimized for, and that the earlier versions of these preferences were errors. That framing is unfair to the engineer I was.

I could not have arrived here without the years of optimizing for the wrong things. The judgment I have now is the residue of a hundred small surprises, each of which would have read as exaggerated caution if it had been advice.

What I optimize for now is shaped by what I have already done. The same will be true a decade from now. The next version of these preferences will read back on this version as incomplete. That is the expected case.

The system that will exist after me

If I had to compress this into one frame, it would be this.

Earlier in my career I optimized for the system I was building. I now optimize for the system that will exist after me.

Those are not the same system. The first one only has to ship. The second one has to be lived in by people whose names I do not know, against constraints I cannot predict, using context that has eroded.

Optimizing for the second system looks slower in the short run. It looks indistinguishable from over-caution to anyone whose horizon is the next release. It is rarely the thing that gets celebrated.

It is the optimization that compounds in the direction I now choose.

The first eleven weeks of this arc were about the patterns underneath these preferences — time as a constraint, simplicity as a posture, judgment over imported rules, the cost of cleverness, the weight of irreversible choices, the demands of ownership, the regret of decisions that aged badly, the information in systems that surprised me.

Fewer abstractions. Fewer irreversible decisions. Earlier clarity.

That is the trade. I have chosen it knowing what it costs, and I expect to choose it again knowing more.

Systems That Surprised You

Fri, 15 May 2026 12:00:00 GMT

The most useful audits of long-lived software are the ones where you were wrong about what would last.

Not wrong about a feature or a deadline. Wrong about durability — the property that is hardest to evaluate at the moment of choice and the most expensive to misjudge.

Every system I have stayed with long enough has eventually surprised me. Something I expected to break held. Something I expected to last quietly stopped working in a way I did not notice for months. The lesson was rarely about the part. It was about the assumption underneath it.

These surprises are worth more than most retrospectives.

Unexpected durability

There is always a part of the system that should not have survived this long. An old script. A library someone wrote in a week. A schema that predates the team. A queue choice made over a lunch conversation.

On any clean evaluation, those pieces should have been replaced years ago. They were not. They quietly kept working, often through changes that should have broken them.

What those parts share is rarely cleverness. It is narrow scope. They did one thing. They had few callers. They made no assumptions about the rest of the system, and the rest of the system made few assumptions about them.

The components that surprised me by lasting were almost always the ones that refused to grow. Not impressive in isolation. Impressive in retention. They survived because nothing depended on them more than necessary, and because nothing about them invited expansion.

This is not protection from every failure mode. Narrow scope can carry its own buried assumptions — date formats, character encodings, integer widths — that one day get tested by a condition the original author never imagined. A small component is not a safe one. It is one whose failure modes have a smaller surface to find. Both can be true.

The honest reading is still uncomfortable. Most of the engineering I have spent worrying about would have been better spent ensuring the same restraint elsewhere. The durable parts were durable because somebody — sometimes accidentally — kept them small.

Quiet failure modes

The opposite surprise is more common. A system stops working correctly, but not in a way that triggers alarms. A scheduled job has been failing silently for weeks. A cache has been serving stale data since a deploy nobody remembers.

These are not the failures the team rehearses for. The team rehearses for the loud ones — the outage, the spike, the 5xx wall. The quiet failures are the ones the system was never instrumented to notice.

What is exposed in those moments is not a missing alert. It is a missing belief. The team did not believe this particular thing could fail. So no one wrote the check. So no one wired the alarm. The failure mode lived in plain sight for as long as the assumption held.

Every quiet failure mode I have found mapped back to an assumption nobody articulated. The fix was rarely the alarm. It was the conversation that surfaced the assumption.

This is the harder kind of stewardship: not catching failure when it happens, but noticing the categories of failure your monitoring is shaped to ignore.

Assumptions exposed by time

Time is the cheapest test the system has. It runs continuously. It exercises every assumption implicit in the original design, including the ones nobody wrote down.

A system in production for five years has been audited by reality more thoroughly than any review process could manage. The parts still standing have passed tests nobody designed.

What surprises me in those audits is not which assumptions failed. It is which assumptions turned out to have been load-bearing in the first place.

Assumptions about traffic volume that quietly shaped the data model. Assumptions about team size that determined how operations were structured. Assumptions about deployment cadence that fixed the release pipeline. Assumptions about the trustworthiness of upstream services that defined the error paths.

Most were never debated. They were the conditions of the moment. They became architecture without ever being decided.

When a long-lived system surprises you, the surprise is usually about one of these. A condition you treated as background turned out to be foreground.

What surprised me about surprise itself

The first few times a system surprised me, I read each surprise individually. A specific job that should not have lasted. A specific failure that should have been caught.

Eventually a pattern emerged. The surprises clustered around two recurring shapes.

Things that turned out to be more durable than expected almost always had less surface area than I remembered.

Things that turned out to be more fragile than expected almost always rested on an assumption I had stopped seeing.

This is not a satisfying lesson. It would have been more useful if certain technologies, patterns, or disciplines reliably produced durable systems. They did not. What produced durability was usually that something stayed small. What produced fragility was usually that something stayed unexamined.

Both findings resist optimization. You cannot decide to keep a system small from the outside; that decision has to survive years of pressure. You cannot list assumptions you have stopped seeing; by definition they are invisible. The best technique available is to assume some are there and invite people who did not build the system to point at them. They have not yet learned which assumptions to overlook.

What I do with the surprises now

Two adjustments have followed from this, both modest.

I treat any old component still in production as evidence of something worth preserving. Not the technology. The conditions that kept it small. Before replacing it, I try to understand whether the replacement will preserve those conditions or quietly remove them.

I treat anything that has not failed for a long time with mild suspicion. Especially anything that processes data, retries on errors, or handles silent edge cases. The absence of visible failure is not the same as correctness. It often just means the monitoring matches the same assumptions the code does.

Neither adjustment is dramatic. Neither shows up in a roadmap. Both have changed more decisions than I expected.

What the surprises were really about

Systems that surprised me did so on a single axis: my model of them was wrong in a specific, locatable way. The durable parts taught me I had over-valued sophistication. The fragile parts taught me I had under-valued the act of naming what the system was relying on.

There is no clean rule that follows. Long-lived systems do not reward rules.

What they reward is the habit of asking, regularly: which parts of this should not still be working — and why are they. Which parts have not failed recently — and what would it look like if they were failing in ways my instruments do not show.

Surprise is information. It is the system telling you which parts of your understanding have not been updated in a while.

That is worth listening to. Even when it means accepting that the parts of the system you invested the least in may have taught you more than the ones you spent years on.

Decisions That Aged Badly

Thu, 07 May 2026 12:00:00 GMT

They were defensible at the time. Reviewed. Discussed. Approved. Made by capable people with the information they had.

The cost arrived years later. Often after the people involved had moved on. Including, in some cases, me.

That gap is the whole problem.

Wrong decisions and decisions that aged badly

A wrong decision is easy to point to. You can name the missing data, the ignored warning, the sloppy reasoning. You can fairly assign blame. Sometimes you should.

A decision that aged badly is harder to find fault with. The reasoning was correct given what was known. The context simply moved past it.

What was once a reasonable trade became a load-bearing constraint nobody chose.

These two failure modes feel similar in retrospect. They are not the same thing.

Most engineering hindsight collapses them. Either everything in the past was a mistake — which is unfair and useless. Or nothing was — which prevents any learning at all.

The honest middle is harder to hold.

Regret without blame

A decision can be defensible at the time and still age badly. The useful posture toward this category is regret without blame.

Regret, because the system would be better if it had gone differently.

Without blame, because the people who made it were paying attention. They simply could not see the ten years that came after.

The risk in this posture

There is a real risk in this posture.

"Regret without blame" can become evasion. It can become the language a team uses to avoid examining actual misjudgments — analyses that were wrong on their own terms, warnings that were available and ignored.

The discipline is to keep the categories separate. Some decisions were wrong. Some aged badly. Some were both.

Treating every bad outcome as a victim of changed context is its own failure of judgment.

Patterns that recur

A few patterns recur, looking back.

Premature abstractions.

The shape of the problem was not yet clear. Someone wrote a generic interface for the cases they could anticipate. The cases that actually arrived were not those. The abstraction now obscures more than it reveals. But it is depended on widely, so it stays.

Optionality preserved just in case.

A configuration flag. A plugin point. A strategy interface for one strategy. Each was nearly free at the moment it was added. Cumulatively they form the bulk of the system's surface area. Removing them is now a quarter of work.

Tooling chosen for early productivity.

The framework that made the first six months fast. The build system that fit the original architecture. None of these were wrong. All of them eventually became things future maintainers had to work around rather than with.

Coupling between things that did not need to know each other.

The shared database two services briefly used. The library that grew to import from the application. The deploy pipeline that assumed one team's release cadence. Each coupling was a small saving. Each is now a constraint.

Best practices imported without context.

The pattern that worked at a previous company. The architecture style that fit a different problem at a different scale. The decision was anchored in authority rather than fit.

Decisions made under deadline that became permanent.

The temporary table. The hardcoded value. The "we will fix this next sprint." There is no next sprint. The shortcut becomes the shape.

The shared property is reversibility

The shared property of all of these is not effort or skill.

It is reversibility.

Each one reduced the cost of the present by spending the future's optionality. Each made the system easier to write and harder to change.

The asymmetry was invisible because the future maintainers were not in the room.

Reversibility is contextual

Reversibility is itself contextual.

What is reversible for a small team can be effectively permanent for a large one, because the cost of coordination scales faster than the cost of the change.

A two-line config swap in a startup is a multi-quarter migration in a public company.

The honest test is not "could this be undone in principle." It is "is the team that would have to undo it likely to be able to."

The strongest filter

This is why the strongest single filter on long-lived decisions is not "is this correct."

It is "is this reversible by the people who will inherit it."

A reversible decision can be wrong and still survive. The team adjusts as understanding improves.

An irreversible decision must be defended against every change in context that comes later. Usually by people who do not know why it was made.

Most decisions that aged badly were locally optimal and globally irreversible.

That is the shape to watch for.

Commit later and lighter

Looking back, the change I would make is not a particular technical choice.

It is to commit later and lighter. To treat decisions as hypotheses. To assume I will be partly wrong, and to build so that being wrong is survivable.

The decisions that aged best in the systems I have known were rarely the cleverest. They were the smallest commitments that addressed the actual problem.

Smaller surfaces.

Fewer dependencies.

Shorter assumptions.

Things that could be undone in an afternoon.

That is not a guarantee against aging. Nothing is.

It is the best available defense.

The lesson

Regret without blame ends in the same lesson.

The judgment is not "I should have known."

It is "I should have made it cheaper to be wrong."

That posture is the one I try to carry into the next decision.

Stewardship Begins Where Ownership Ends

Wed, 29 Apr 2026 12:00:00 GMT

Most systems are built as if the original builder will always be around to explain them.

That assumption fails early. People change teams. Priorities shift. Memory thins out. The system remains.

This is why ownership is not a sufficient concept for long-lived software. Ownership describes responsibility in the present. Stewardship describes responsibility across time.

A person can own a service for a quarter. They cannot own the future conditions under which it will be changed.

That future belongs to other people. Usually people who did not choose the original trade-offs, do not share the original context, and will meet the system under pressure.

Ownership is local. Stewardship is temporal

Ownership is useful because it makes accountability visible. Someone is on call. Someone approves changes. Someone is expected to know where the sharp edges are.

But ownership has a built-in weakness. It encourages a model of software where knowledge lives inside the current owner.

That works until the owner is unavailable.

Many systems appear healthy only because the same people keep carrying them. The moment they step away, the real condition of the system becomes visible. Documentation is missing. Operational assumptions were never written down. Naming decisions only make sense if you remember a migration from two years ago. A change that looked routine becomes risky because nobody can tell what is load-bearing.

This is the practical difference between ownership and stewardship.

Ownership says: I know how this works.

Stewardship says: someone else should be able to know how this works without me.

That second standard is harder. It requires a builder to spend effort on people who are not present yet. It asks for work whose benefit arrives later, often after the builder has moved on.

It is still part of the job.

Handoff friction is a design signal

Teams often treat painful handoffs as a documentation problem. Sometimes they are. Often they are a system design problem that documentation cannot rescue.

If a new engineer needs three weeks of oral history before they can safely change a service, the issue is not only that the history was undocumented. The system depends on history too heavily.

If a service requires knowing which exception to ignore, which alarm is noisy on purpose, which table can never be backfilled, and which endpoint is "legacy" but still critical, then the system is communicating through folklore.

Folklore scales badly.

It also creates a hidden tax on teams. The current maintainers spend time answering the same orientation questions. New maintainers learn caution instead of clarity. Risk accumulates around every undocumented edge because nobody wants to be the person who discovers, in production, that a strange rule mattered.

Handoff friction should be read as a structural signal. It means knowledge is stored in the wrong place.

Some of that knowledge belongs in documents. Some belongs in runbooks. Some belongs in tests. Some belongs in the code itself, through naming, boundaries, and fewer surprising paths.

The point is not to write more words. The point is to move critical understanding out of private memory.

Documentation is part of the interface

Teams often speak about documentation as if it were secondary work, adjacent to the real system. For long-lived systems, it is part of the system.

An undocumented deploy process is not a complete deploy process. An undocumented recovery path is not an operational capability. A dependency nobody can explain is not just messy. It is a blind spot with maintenance attached.

This is why "the code is the documentation" is rarely true in practice. Code can show what the system does. It usually cannot explain which constraints matter, which decisions were temporary, which failures are acceptable, and which simplifications must not be undone.

Those are design facts. If they exist only in memory, the design is incomplete.

Good documentation does not try to narrate every file. It reduces ambiguity where ambiguity becomes expensive.

Why does this service exist. What does it own. What does it deliberately not own. How is it safely changed. How is it safely operated. What assumptions would surprise a reasonable new maintainer.

These are not administrative details. They are continuity mechanisms.

Stewardship changes what "done" means

A task can be complete and still be poorly stewarded.

The feature works. The ticket is closed. The owner understands the trade-off. Nobody else does.

That is not a finished piece of engineering. It is a working private arrangement.

Stewardship raises the standard meaningfully. It asks whether the system became easier or harder for the next person to carry.

Did the change introduce a new hidden dependency. Did it expand the set of things that must be remembered. Did it create another special case with no visible explanation. Did it leave behind a migration state that only makes sense to the person who performed it.

These are ordinary questions. But they are often skipped because they do not affect the immediate deliverable.

Time is where the bill arrives.

This is also why stewardship often looks unglamorous. Clear naming is stewardship. Removing dead paths is stewardship. Explaining why a constraint exists is stewardship. Writing a small runbook before an incident forces one is stewardship. Declining a clever shortcut because nobody else will trust it later is stewardship.

None of this looks impressive in the week it is done. All of it matters in the year after.

Build for the next honest reader

There is a useful standard for long-lived systems: build for the next honest reader.

Not the ideal maintainer. Not the person who sat in the meeting. Not the original author on a well-rested day.

The next honest reader is competent, busy, and missing context. They are trying to make a safe change without breaking something important. They do not need elegance. They need legibility.

This standard improves both code and communication.

It discourages hidden invariants. It exposes names that only make sense historically. It makes one-off exceptions feel expensive. It pushes knowledge toward artifacts that survive turnover.

Most importantly, it resists a common failure mode in experienced teams: mistaking familiarity for clarity.

A team that has lived with a system for years can stop seeing what is opaque about it. Everything feels obvious because the explanations are carried socially.

The next honest reader is the corrective. If they cannot safely understand the thing, the thing is not yet clear enough.

Systems outlive confidence

One reason stewardship matters is that confidence is temporary.

The certainty present during implementation decays fast. Six months later, even the original author may remember the decision but not its edges. Two years later, a surviving comment or note may be the only thing separating an intentional constraint from a superstition.

Long-lived systems should not depend on durable confidence. They should depend on durable traces.

This is not a call for exhaustive process. It is a call to leave behind enough structure that the system can be changed by someone who did not witness its formation.

That is what stewardship is.

Not permanent ownership. Not total foresight.

A quieter responsibility: to build so the system can survive your absence without becoming dangerous.

What You Refuse to Do Is Architecture

Fri, 24 Apr 2026 12:00:00 GMT

Most architecture conversations focus on what a system can do. The harder, more useful conversation is about what it won't.

A system without explicit constraints is not flexible. It is undecided. And undecided systems force every contributor to re-make the same decisions, in private, every week.

This is how complexity arrives. Not through bad code. Through deferred choice.

Constraints reduce the surface area of disagreement

When a team says "we don't run background jobs in this service," a hundred future debates disappear. When it says "we never call the database directly from controllers," code review becomes shorter and clearer. When it says "we don't accept new dependencies without a documented reason," dependency drift slows.

None of these are restrictions on capability. They are restrictions on ambiguity.

Capability without constraint becomes a request to negotiate every decision twice — once when the code is written, and again when someone tries to understand it.

Optionality is sold as freedom. It behaves as debt.

Every option you preserve is something a teammate must consider, document, test, and eventually maintain.

When the original author moves on — and they will — the option remains. No one can quite say whether it's holding something up or just sitting there. So nobody touches it. So it stays.

A team can lose a year to options nobody ever chose to use.

The cost of optionality is rarely visible at the moment it is added. It compounds in the form of caution, hesitation, and the slow accumulation of "I'm not sure if we can change this."

Explicit constraints calm teams

This is the part most people underestimate.

A large share of engineering energy goes not to writing code but to adjudicating what is acceptable. The smaller the allowed set, the less energy is consumed there.

A clear constraint says: don't think about this. Don't relitigate this. Spend your judgment elsewhere.

Teams that operate inside well-defined bounds tend to ship more, argue less, and produce systems that age predictably. Not because they have better engineers. Because they have fewer open questions per square meter.

The useful constraints are voluntary

The most useful constraints are the ones a team chooses before they are forced to.

A budget for latency. A ceiling on service count. A rule that no service owns more than one database. A list of dependencies that require written justification.

These are not best practices. They are local agreements with future-you.

When the constraint is explicit, the trade-off becomes visible. You can see what you are giving up. You can argue against it on its actual terms.

When the constraint is implicit — "we just don't tend to do that here" — it operates as folklore. Folklore decays with turnover. Documents survive turnover better.

Constraints clarify trade-offs

Without a stated limit, a team optimizes for a moving target. With one, the trade-off is legible: we accept slower iteration here because we want fewer surprises later. We accept higher memory use because we want simpler code paths. We accept fewer features because we want smaller error surfaces.

These are not heroic decisions. They are ordinary ones, made legible.

The legibility is the point.

The cost is paid early. The benefit arrives late.

A system designed under explicit constraints tends to look smaller, slower to start, and disappointing on the day it ships.

It also tends to look reasonable five years later.

That is the trade — and it is the trade most teams are reluctant to take, because the cost of restraint is paid up front, and the benefit accrues to people who may not be in the room when it is felt.

This is also why constraints are often added under duress, after a system has grown beyond comprehension. By then, the constraints feel punitive instead of clarifying. They are perceived as taking something away, rather than preserving something.

It is much cheaper to start with constraints than to retrofit them.

The wrong constraint is also expensive

A constraint chosen before the team understands the problem can ossify a misreading. Writing down "we never do X" too early locks in a decision the team does not yet have the evidence to make.

The defense is modest: prefer constraints that can be relaxed, and treat any rule older than the system itself with suspicion.

A team can relax a rule in an afternoon if it proves wrong. Reintroducing a rule once a hundred files quietly violate it is a quarter of work, sometimes more.

The asymmetry runs one direction. When in doubt: start narrower than feels natural. You can always widen. You will rarely need to.

Constraints are not the opposite of design

They are its load-bearing structure.

The systems that age well are not the most powerful or the most general. They are the ones whose authors were willing, early on, to write down what the system would not be.

That decision — to refuse, in writing, on the record — is the part that carries.

AI in Long-Lived Systems

Wed, 15 Apr 2026 12:00:00 GMT

Every AI integration is a bet that you can monitor something you do not fully control.

The conversation about AI in software systems is mostly about capability. What it can do. What it will do next.

The harder question is what happens when you depend on it. Not for a demo. Not for a prototype. For a system that needs to work reliably for years, maintained by people who did not build it, under conditions that shifted since the original design.

Probabilistic output, deterministic expectations

AI introduces a category of behavior that most engineering practices were not designed for: probabilistic output embedded in deterministic expectations. A function that returns a different result for the same input is not a function in the way most systems assume. When that behavior is buried inside a pipeline — making decisions, filtering data, generating content, routing requests — the system becomes harder to reason about in ways that do not show up in testing.

This is not a reason to avoid AI. It is a reason to treat it differently than other dependencies.

Observability

The first problem is observability. Most systems log what happened. AI components require logging enough context to reconstruct why. A model that rejects a transaction, summarizes a document, or classifies a support ticket is making a judgment. If you cannot inspect that judgment after the fact, you cannot debug it, audit it, or explain it to the applicant who was denied.

Traditional observability assumes deterministic behavior. You trace a request, you see the path it took, and you understand the outcome. With AI, the path includes a decision that may not be reproducible. The same input tomorrow might produce a different output. Logging the input and output is necessary but not sufficient. You need the model version, the prompt template version, the context window — logged alongside each call, not reconstructed after the fact. This tooling tends to arrive late.

Irreversibility

The second problem is irreversibility. AI is increasingly used to make routing, filtering, and approval decisions that are difficult or impossible to undo. Approving or denying applications. Prioritizing work. Filtering what a user sees. Generating communications sent to real people. Each of these has downstream consequences that compound. A wrongly filtered email is not just a missed message — it is a missed message that influenced a decision that influenced a timeline no one can trace back to the filter.

The systems that handle this well treat AI decisions as proposals, not conclusions. A model suggests; a human confirms. A model classifies; a review queue catches edge cases. A model generates; a validation layer checks constraints. This is slower. It is also more durable. The speed advantage of AI is real, but it is not free. The cost is paid in the infrastructure required to keep the system accountable.

Containment

The third problem is containment. AI capabilities are general-purpose by nature, which makes them easy to spread across a system. A team integrates a language model for one use case. Another team sees it working and adopts it for a different use case. The model becomes a shared dependency with no clear owner, no consistent evaluation criteria, and no unified understanding of its failure modes. This is tool sprawl, but with a dependency that changes behavior when the provider updates it.

Containment means treating AI as a bounded component with explicit interfaces. What goes in. What comes out. What the acceptable range of behavior is. What happens when the behavior falls outside that range. These are the same questions you would ask about any critical dependency, but AI makes them harder to answer because the behavior is not specified in code. It is learned, and it shifts.

The pattern that holds

The pattern that holds up is the same one that holds up for most long-lived system decisions: make it observable, make it reversible, make it replaceable.

These constraints feel like they slow things down. They do. That is the point. The speed of integration is not the bottleneck in long-lived systems. The bottleneck is the speed of understanding — how quickly someone unfamiliar with the system can figure out what it does and why.

AI does not make systems fragile by itself. Systems become fragile when AI is embedded without the same discipline applied to every other critical component: clear boundaries, monitoring that captures why and not just what, and the assumption that it will eventually be wrong in a way no one predicted.

The question is not whether to use AI. It is whether you are building the infrastructure to live with it.

Tooling and Accidental Complexity

Fri, 10 Apr 2026 12:00:00 GMT

No one decides to make a system heavy. It happens one reasonable choice at a time.

A team adds a caching layer because latency matters. A monitoring tool because visibility matters. A build plugin because developer experience matters. A message broker because decoupling matters. Each addition solves a real problem. Each is justified on its own terms. The weight is not in any single decision. It is in the aggregate.

Accidental complexity

This is accidental complexity — not complexity born from the problem, but complexity accumulated from the solutions. The system grows heavier than the problem requires, and the weight arrives so gradually that no one notices until a change that should take a day takes a week.

Stack drift

Stack drift is the quiet version of this. The stack you run today is not the stack you chose. It is the stack you chose plus everything that was added to keep it working, plus the things those additions required, plus the compatibility layers that hold the seams together. Nobody designed the full picture. It emerged.

The asymmetry of adding and removing

The asymmetry is what makes this hard. Adding a tool is a local decision. One team evaluates it, one team adopts it, and the integration cost seems small. Removing a tool is a global decision. It requires understanding every system that touches it, every pipeline that depends on it, every configuration that references it. Adding takes a week. Removing takes a quarter. So things accumulate.

Configuration is code

Configuration is part of the weight, and it is the part that gets the least scrutiny. A YAML file that controls deployment behavior is code. An environment variable that changes runtime behavior is code. A feature flag that alters control flow is code. But none of these go through the same review process as code. They accumulate in places that are harder to search, harder to test, and harder to reason about.

Hidden dependencies

Hidden dependencies are the most expensive form of this. A service that calls another service through an intermediary. A build step that depends on a global state set by a previous step. A test suite that passes only because an unrelated process is running. These dependencies are invisible in architecture diagrams. They show up at 2 AM when an on-call engineer is trying to understand why something that has not changed is suddenly broken.

The actual system

The system is less resilient than it appears. The diagram shows five services. The reality is five services, three shared libraries, a custom build plugin, two configuration management tools, an internal CLI that nobody remembers writing, and a Makefile that references a Docker image tagged "latest" from 2024. That is the actual system. The weight is real even when it is not visible.

The consolidation instinct

The instinct when this becomes painful is to consolidate. Replace five tools with one platform. Build an internal developer platform. Standardize everything. This sometimes works. But it often introduces a different kind of weight — the weight of the platform itself, which becomes the new thing that is hard to change. The second system is frequently heavier than the first, built to solve every problem the first system had, and a few it did not.

Audit over consolidation

The better response is not consolidation but audit. What is here? Why was it added? Is it still solving the problem it was introduced for? If not, what would it take to remove it? These questions are simple. Answering them honestly is not.

Carrying cost

Removal requires understanding that adding was not a mistake. The caching layer was correct when it was introduced. The monitoring tool was the right choice at the time. The message broker solved a real coordination problem. The question is not whether it was a good decision then. The question is whether it is still earning its carrying cost now. Every tool has a carrying cost — in maintenance, in cognitive load, in the time it takes a new team member to understand the system. That cost is never on the invoice.

Subtraction as operations

The systems that stay manageable are not the ones that avoid adding tools. They are the ones that treat removal as a normal part of operations, not as a special project. They notice when something stops earning its cost. They budget time for subtraction, not just addition.

Intentionality over lightness

Lightness is not the goal. Intentionality is. A heavy system where every component is justified and understood is better than a light system held together by assumptions. The problem is not weight. It is weight that nobody chose and nobody maintains.

Systems grow heavier than intended because adding is easy and removing is hard. The discipline is not in choosing fewer tools. It is in continuing to ask whether the tools you have are still the right ones.

Abstractions That Age Poorly

Thu, 02 Apr 2026 12:00:00 GMT

Good abstractions compress complexity. Bad abstractions relocate it.

The best ones disappear. You stop noticing them. The worst ones announce themselves on every change, every new requirement exposing another edge case the abstraction was not designed for. The team spends more time working around the abstraction than working through it.

Most abstractions that age poorly were not bad ideas. They were good ideas applied too early, before the problem was understood well enough to know what should be hidden and what should remain visible.

Early abstraction is seductive

You see a pattern forming — two or three things that look similar — and the instinct is to unify them. Extract a base class. Create a shared interface. Build a generic handler. The code becomes shorter. The duplication disappears. It feels like progress.

The cost arrives later. The things that looked similar turn out to be similar only at the surface. As requirements diverge, the shared abstraction accumulates conditionals. It grows parameters. It develops modes. The generic handler now has seven configuration options, four of which exist to accommodate a single consumer. The abstraction that reduced complexity has become the primary source of it.

The cleverness tax

The initial abstraction was elegant. Maintaining it is not. Every modification requires understanding the full surface area of the abstraction, not just the part you are changing. New team members cannot reason about one use case without understanding all of them. The cost is no longer paid once. It is paid on every change.

Local elegance creates this problem more often than carelessness does. A developer writes a beautiful generic solution. It works. It is well-tested. It handles the current use cases with minimal duplication. Within its own scope, it is correct. But zoom out, and the system now has a coupling point that resists change. Two features that should evolve independently are bound together by a shared interface that neither fully fits.

The failure is not in the code. It is in the assumption that the things being unified will continue to evolve together. That assumption is rarely tested at the time the abstraction is introduced. It is tested later, when one use case needs to move in a direction the abstraction does not support.

When the generic bus breaks

I have watched this happen with event systems. A team builds a generic event bus early in a project's life. Every domain event flows through the same pipeline. It works well for the first year. Then one domain needs strict ordering because payment reconciliation depends on event sequence. Another needs transactional boundaries. A third needs fire-and-forget because latency matters more than delivery guarantees. The generic bus, originally a clean solution, becomes a negotiation layer where every team must accommodate every other team's requirements.

The event bus was not a mistake. It was a reasonable choice given what was known at the time.

The mistake was not revisiting that choice when the domains diverged. The abstraction became structurally necessary before anyone noticed, and by the time the cost was visible, replacing it required coordinated effort across multiple teams.

The pattern

Abstractions that age poorly tend to share a few traits. They unify things that are similar now but will diverge later. They hide differences that turn out to matter. They are introduced before the problem space is stable. And they become harder to remove the longer they exist, because more code depends on them.

The alternative is not to avoid abstraction. It is to delay it. Write the duplication. Let the pattern prove itself over three or four instances, not two. Wait until you understand not just what is similar, but why it is similar. The shape of a good abstraction comes from understanding the forces that will act on it, not from the code that currently exists.

Duplication is cheaper than the wrong abstraction

Duplicated code can be changed independently. A bad abstraction forces coordinated change. Duplicated code is obvious. A bad abstraction hides its cost behind a clean interface that no longer reflects reality.

The hardest part is resisting the urge to clean things up too early. Duplication feels like a problem. It looks unfinished. It triggers the same instinct that makes refactoring attractive — the desire for order. But premature order can be more expensive than temporary disorder.

The abstractions that last are the ones introduced after the problem was understood, not before. They are smaller than you would expect. They hide less than you would think necessary. They leave room for the cases that have not appeared yet, not by being generic, but by being narrow enough that they do not get in the way.

Simplicity is not the absence of abstraction. It is the presence of the right ones, introduced at the right time, hiding the right things.

Refactoring as a Signal

Fri, 27 Mar 2026 12:00:00 GMT

The code that needs refactoring most often is rarely the worst code. It is the code sitting on top of a structural assumption that no longer holds.

Refactoring is framed as improvement. You take something that works and make it more modular, more readable, easier to extend. That framing is not wrong. But it hides the more interesting question: why does the code need restructuring in the first place?

The need to refactor is a signal

It tells you something about the gap between what the system was designed for and what it is now being asked to do. That gap is worth understanding before you begin restructuring.

Most refactoring conversations start with the code. The function is too long. The module has too many responsibilities. The naming is inconsistent. These describe the code. They do not explain why it drifted. The deeper question is: what changed? What shift in requirements, team structure, or understanding created the misalignment you are now trying to fix?

When you skip that question, refactoring becomes cosmetic. You improve the shape of the code without addressing the structural mismatch that created the problem. Six months later, the same area needs restructuring again — not because the first refactor was bad, but because it treated the symptom.

The pattern of repeated cleanup

A module gets refactored, stabilizes briefly, then drifts again. Each pass feels productive. Each pass addresses real issues. But the cycle continues because the underlying cause — a shifting domain boundary, an unclear ownership model, a feature that outgrew its original scope — was never confronted.

Repeated cleanup is not a sign of careless engineering. It is a sign that the system's organizational shape no longer matches its purpose. The refactoring is correct at the code level and insufficient at the design level.

Consider a payments module originally designed as a simple pass-through — validate, forward, log. Over two years, it becomes the system's primary orchestration layer. It coordinates retries, manages state, triggers downstream workflows. It gets refactored three times, each time to accommodate a responsibility it was never meant to own. Each refactor improves the code. None questions whether the module should still exist in this form.

Refactoring as avoidance

There is a subtler failure mode. Ambiguity makes refactoring attractive. The team faces a question it cannot cleanly answer — whether a feature belongs in this service at all, how to scope a requirement that keeps shifting, a technical decision with no clear winner. Refactoring offers a productive alternative. You spend a week improving the codebase, and the code is genuinely better afterward. But the hard question remains untouched.

This is not laziness. Refactoring has clear inputs and measurable outputs. The hard question does not. Given a choice between visible progress and uncertain exploration, most teams choose progress. The cost is that the architectural question gets deferred again.

I have seen this play out in systems where the same area gets refactored by successive teams. Each team improves the code according to their understanding. Each refactor is defensible. But the cumulative effect is a module that has been reshaped multiple times without anyone asking whether it should exist in its current form at all.

"This code keeps becoming messy"

The signal worth paying attention to is not "this code is messy." It is "this code keeps becoming messy." The first is a maintenance task. The second is a design problem.

The misalignment has a specific shape. The code is organized around one model of the domain, but actual usage has shifted to another. Features that were once separate now overlap. Boundaries that made sense at one scale create friction at the current one. The refactoring impulse is correct — something is wrong. But the fix is not reorganizing the code within the existing structure. It is questioning the structure itself.

This is harder work. It requires understanding not just the code but the forces acting on it. Why do these two modules keep needing to change together? Why does every new feature touch the same three files? Why does onboarding to this area take twice as long as it should?

These questions do not have clean answers. They require conversations about product direction, team boundaries, and debt that accumulated from decisions that made sense when the team was smaller and the product was simpler.

What a refactor should produce

The most useful thing a refactoring can produce is not cleaner code. It is a clearer understanding of why the code looked the way it did. If you finish a refactor and can articulate what assumption was wrong, you have learned something that will inform the decisions that follow. If you finish a refactor and can only say the code is better now, you have improved one file and learned nothing about the system.

Refactoring is maintenance when it addresses known issues in stable code. It is a signal when it keeps happening in the same places. And it is avoidance when it substitutes for the harder work of confronting what actually needs to change.

The discipline is not in knowing how to refactor. It is in asking what the need to refactor is trying to tell you.

Defaults Nobody Revisits

Wed, 18 Mar 2026 12:00:00 GMT

Best practices are borrowed conclusions. They compress someone else's context into a rule you can follow without thinking. That is their value, and that is their cost.

The rule says: keep functions short. So you split a coherent operation into five pieces, each with a name that explains what it does but not why it exists. The code is clean. It follows the linter. It also resists change, because changing the behavior now means understanding the relationship between fragments that were never designed to be separate.

This is what blind rule-following produces. Not bad code, compliant code. Code that passes review but quietly increases the cost of the next decision.

Best practices originate in specific environments. Extract a service when your deployment boundaries require it. Write unit tests when your domain logic is complex enough to justify the maintenance cost. These are reasonable conclusions, inside their original constraints. Detach them from those constraints, and they become defaults nobody revisits.

The problem is not that best practices are wrong. It is that they are incomplete. They encode the what without the why. And when the why changes, a new team, a different scale, a shifted business model, the what keeps running on inertia. Nobody questions it, because questioning a best practice feels like questioning competence.

This is authority-based decision-making. The practice carries weight not because it fits the current situation, but because it was endorsed somewhere credible. A conference talk. A well-known post. A team lead who used it at their last company. The reasoning gets replaced by the reference.

Senior engineers are not the ones who know more best practices. They are the ones who know when to stop following them. Judgment is not the rejection of rules. It is the ability to evaluate whether a rule still applies in the current context, and to accept the cost if it does not.

There is a specific failure mode I have seen repeatedly. A team adopts clean architecture and applies it uniformly. Every service gets the same layered structure. Simple CRUD endpoints get use-case classes, repository interfaces, domain models, and DTOs, a five-layer ceremony around a single database insert. The architecture is consistent. It is also expensive. New features take longer than they should because every change must pass through layers that exist for structural purity, not because the domain demands them.

The cost is not visible in any single file. It is distributed across the entire system as friction. And friction does not trigger alerts. It shows up as slower velocity, longer onboarding, a growing sense that something is heavier than it needs to be.

Clean code that resists change is not clean. It is rigid. Cleanliness is not a property of the code alone, it includes how the code responds to the next requirement. If every modification requires reshuffling abstractions, the structure is serving itself, not the system.

The alternative is not chaos. It is context-sensitive reasoning. You evaluate a practice against the actual constraints: team size, expected lifetime, rate of change, domain complexity. Sometimes the best practice applies perfectly. Sometimes it applies partially. Sometimes it actively gets in the way.

This requires something harder than knowledge. It requires the willingness to make a judgment call and own the consequences. Best practices offer safety, if the outcome is bad, the decision was defensible. Judgment offers no such cover. You chose, and the result is yours.

That is why organizations default to best practices. They reduce variance. They make decisions reviewable. They create a shared vocabulary. These are real benefits. But they come at a cost: they suppress the reasoning that distinguishes systems that work from systems that last.

Judgment has its own failure mode. Without shared constraints, it becomes preference. I have seen senior engineers override team conventions not because the context demanded it, but because they had the authority to do so. The resulting system is legible only to its author. This is not judgment, it is taste mistaken for reasoning. The difference matters: judgment accounts for the team that inherits the decision. Taste accounts only for the person making it.

Medicine has a useful model here. Clinical guidelines are best practices, standardized treatment protocols backed by research. Experienced clinicians deviate from them routinely, based on patient-specific context. But the field does not treat deviation as rebellion. It treats it as a skill that must be taught, practiced, and documented. The guideline provides the default. Clinical judgment handles the exception. And when a clinician deviates, the reasoning goes in the chart. Software engineering rarely has an equivalent. The decision is made, but the reasoning stays in someone's head, or in a Slack thread that no one will search six months later.

This points to what is actually missing. Not more best practices, and not more individual judgment, but a culture that treats the override as a first-class event. When someone deviates from a convention, the deviation should be visible and the reasoning should be recorded. Not as bureaucracy, as design context for the next person who encounters the same decision point.

The systems that hold up over time are not the ones built with the most discipline. They are the ones where someone understood the constraints well enough to know which rules to follow and which to set aside. That understanding cannot be codified. It is earned through decisions that went wrong and the willingness to examine why.

Judgment is not a replacement for best practices.

It is the thing that makes them useful.

The Carrying Cost of Flexibility

Thu, 12 Mar 2026 12:00:00 GMT

Flexibility is one of the most intuitive design goals in software. Make it configurable. Make it extensible. Build the abstraction so it handles future cases. These sound like good engineering. Often, they are. But flexibility has a carrying cost, and that cost does not announce itself early.

Early in a system's life, flexibility feels free. A generic interface, a plugin architecture, a config-driven behavior, these seem like responsible choices. They signal foresight. They reduce the fear of being wrong. And in some cases, they genuinely pay off.

The problem is that most flexibility is speculative. It solves a problem that has not arrived yet, and may never arrive. Meanwhile, it is already shaping the system, adding indirection, widening interfaces, and distributing logic across layers that could have been one.

There is a specific pattern I keep seeing. A team builds a module that handles one case well. Someone raises the question: what if we need to support a second case later? So the module gets generalized. An interface is extracted. A strategy pattern is introduced. The original behavior is now one implementation of a broader contract.

Six months later, there is still only one implementation. But the abstraction remains, and every new developer reads it as if it matters. They study the interface. They wonder what the other implementations are. They hesitate to change it, because it looks load-bearing. The flexibility that was meant to reduce future cost is now increasing present cost, not through failure, but through indirection that no one needs.

This is not a failure of engineering skill. It is a failure of timing. The abstraction was introduced before the pressure that would justify it. And without that pressure, there is no feedback loop to tell you whether the abstraction is right.

Premature flexibility and premature optimization share the same root. Both try to solve a future problem with present effort, using incomplete information about what the future actually requires.

The difference is that premature optimization is widely understood as a risk. Premature flexibility often is not. It passes code review. It gets praised as forward-thinking. It survives because it looks like the kind of thing a senior engineer would do.

But the cost is real. Every unused extension point is a question mark in the codebase. Every config flag that controls behavior no one toggles is a branch that still has to be understood, tested, and maintained. Every generic interface that only has one implementation is a contract that constrains without earning its keep.

Optionality is not free. It defers the cost of a decision, but it does not eliminate it. Keeping your options open means keeping the cognitive surface area wide. Someone has to hold all those possibilities in their head, or discover them the hard way when they intersect in production.

The systems I trust most are not the most flexible ones. They are the ones where the constraints are visible and the boundaries are tight. Where a module does one thing and the interface reflects that one thing. Where someone made a decision instead of deferring one.

Simplicity is not the absence of thought. It is the result of making hard choices early, choosing what the system will not do, what cases it will not handle, what abstractions it does not need yet. That takes more judgment than generalization, because you have to be willing to be wrong about a specific bet rather than hedge across all of them.

There is a phrase that sounds like a compliment but often signals a problem: "It can handle anything." A system that can handle anything usually handles nothing particularly well. The cost of generality is diffusion. The behavior is spread across configurations, strategies, and runtime switches. Understanding what the system actually does requires tracing through layers that exist for hypothetical cases.

The more useful question is not can this handle future cases, but does this make the current case clear? A system that is clear about what it does now is easier to change later than one that is vague about what it might do eventually.

This is counterintuitive. It feels like rigidity. But rigidity and clarity are not the same thing. A rigid system resists change. A clear system makes change legible, you can see what exists, understand why, and decide what to replace.

Flexibility helps early when it addresses a known, concrete axis of variation. If you know from real usage that the system must support multiple authentication providers, an interface for that is earned. If you are guessing that someday someone might want to swap the database, that interface is speculative, and you will likely guess the abstraction boundary wrong anyway.

The signals are usually visible if you look for them. Over-generalization shows up as interfaces with one implementation, config flags that are always set to the same value, and extension points that no one has extended. Optionality without purpose shows up as code that is harder to read than the problem it solves.

Simplicity as a strategy means accepting that you will sometimes need to change a system that was not designed for that specific change. That is a real cost. But it is a cost you pay once, with full context, when the need is concrete. The alternative, paying a continuous tax on flexibility you never use, is more expensive. You just do not see it on any single line item.

The hardest part of simplicity is that it requires confidence in a decision you cannot fully validate at the time you make it. Flexibility feels safe because it avoids commitment. But avoidance has its own cost, and it compounds quietly.

Build for what you know. Make the current case clear. Let the future problem arrive before you solve it.

The systems that age well are not the ones that anticipated everything. They are the ones that were easy to change when the unanticipated thing finally showed up.

Time As A Design Constraint

Wed, 04 Mar 2026 12:00:00 GMT

Most systems are designed against requirements. Features, deadlines, throughput targets. These are real inputs, but they share an omission: they describe the system at one point in time.

Treating time as a design input changes what you notice, and what you defer.

Software ages whether you plan for it or not. The team that built it rotates. The domain shifts. The libraries update or stop updating. The person who understood the naming convention leaves, and no one asks why things are named that way. They just keep going.

None of this is failure. It is the default behavior of systems under time. The question is whether your design accounts for it.

There is a common pattern in engineering decisions: something works well today, so it ships. The trade-off is deferred, not avoided. You know the naming is inconsistent. You know the module boundary is wrong. You know the config layer is doing too much. But it works, and there is pressure to move forward.

This is not laziness. It is rational under short-term constraints. The problem is that short-term rationality compounds. Each small deferral is reasonable on its own. Together, they form a system that no one fully understands, not because it is complex, but because its structure no longer reflects its intent.

The config layer that parses YAML, validates environment variables, and also handles feature flags, because at one point those were the same concern. They are no longer the same concern, but the module does not know that.

That gap between structure and intent is where maintenance cost lives.

When I say "treat time as a design input," I do not mean plan for every future scenario. That leads to over-abstraction, another cost that compounds. I mean something simpler: when making a decision, ask what this will look like in two years if no one touches it.

Not what if requirements change. Not what if traffic doubles. Just: what happens if this decision sits, untouched, while the context around it keeps shifting?

Some decisions hold up. A well-named module, a clear boundary, a simple contract between services, these age well because they carry their own context. You can read them later without needing the original author to explain.

Others do not. A clever shortcut, a shared mutable structure, an implicit dependency between teams, these become opaque. Not because they were wrong, but because they required context that no longer exists.

There is a phrase I keep returning to: deferred clarity. It shows up everywhere. In code that works but cannot be read. In architecture that functions but cannot be explained. In decisions that were made for good reasons that no one wrote down.

Deferred clarity is not technical debt. Debt implies someone borrowed deliberately. Deferred clarity is quieter, the slow accumulation of decisions where understanding was present but never made explicit. The cost does not show up as a bug. It shows up as slower onboarding, longer review cycles, more meetings to re-establish context that should have been in the code.

Context erosion is the mechanism behind most of this. Every system carries implicit knowledge: why this service exists separately, why that field is nullable, why the deployment order matters.

When the people who hold that knowledge move on, the system does not change, but its readability does. This is not a documentation problem. It is a design problem. Systems that depend on external context to be understood are fragile, they work until someone needs to change them.

The fix is not more documentation. It is designing for reduced context. Naming that explains intent. Boundaries that match real domain splits. Contracts that are explicit rather than implied.

This does not require more effort. It requires effort at a different point, at the moment of decision rather than after the fact.

The systems I am most proud of are not the ones that were technically impressive at launch. They are the ones that someone else changed successfully two years later without calling me.

That is the real test. Not whether the system works on day one, whether it remains legible on day seven hundred.

Treating time as a constraint changes what you optimize for. You favor clarity over cleverness. Explicit contracts over implicit ones. Fewer abstractions rather than more. You become skeptical of flexibility that no one has asked for, because you have seen how unused flexibility becomes unused complexity.

You do not stop making trade-offs. You just weigh them differently. The question shifts from does this solve the problem to will this still make sense when the context around it has changed?

That second question does not have a clean answer.

But asking it changes the decisions you make.

Decisions, more than code, are what age.

When Code Is No Longer Scarce

Tue, 17 Feb 2026 12:00:00 GMT

For most of software history, change was expensive.

Humans wrote the code. Humans hesitated. Humans coordinated.

That friction was not a flaw.

It was the stabilizing force.

Most of our architecture patterns, monorepos, microservices, CI pipelines, code review, ownership models, were built around a quiet assumption:

Change would be limited by human effort.

That assumption is dissolving.

The hidden constraint

Software systems work because change is naturally constrained.

Not by rules.

By people.

A developer considers the blast radius before refactoring. A team weighs whether touching a stable subsystem is worth it. A reviewer pushes back when something feels unnecessary.

Those pauses shape the system.

They create stability not by policy, but by friction.

When change becomes cheap

AI systems can now:

generate entire modules
refactor thousands of files
standardize patterns instantly
propose improvements continuously

When production cost drops toward zero, the limiting factor shifts.

It is no longer writing code.

It is coordinating it. Understanding it. Trusting it.

And those costs do not shrink at the same rate.

The new bottleneck: comprehension

What happens when 30% of your codebase changes in a week?

Not because strategy changed, but because optimization became automatic.

Reviews become procedural. Ownership becomes symbolic. Tests validate motion more than intent.

The system becomes fluid.

But not necessarily clearer.

We are entering a world where the cost of producing code collapses while the cost of understanding it remains stubbornly human.

That asymmetry is the real shift.

The reflex to add more structure

When coordination becomes painful, we add structure:

more metadata
stricter boundaries
deeper dependency graphs
more automation to manage automation

We call this scaling.

Often, it is complexity redistribution.

The system becomes easier for machines to operate on.

Harder for humans to reason about.

At some point, the coordination layer consumes more energy than the product itself.

The system becomes the product.

Abundance is not automatically good

Software has never been hard because we could not produce enough code.

It is hard because:

deciding what to build requires judgment
deciding what not to touch requires restraint
removing code requires confidence

A system optimized for constant motion loses the ability to say:

“This is done.”

Without that boundary, everything becomes provisional.

That is not agility.

It is restlessness.

What survives in a world of abundant change

When change is cheap, the scarce resources become:

attention
context
taste
long-term memory
judgment

These do not scale linearly with automation.

They become more valuable as production accelerates.

Not less.

Protect the real constraint

AI expands what we can generate.

It does not automatically improve what we should generate.

Architecture was never just about managing code.

It was about managing change.

And in a world where change is abundant, the role of restraint becomes structural, not optional.

Protect the real bottleneck: human judgment.

Because once judgment becomes secondary, complexity becomes inevitable.