The Defensibility Problem

There is a gap opening between what automated systems now decide and what organizations can account for. It is not a gap in accuracy. Models have gotten better at the thing they are asked to do. The gap is that almost nothing has been built to hold the reasoning, and so a growing share of consequential decisions arrive with no record capable of surviving a serious question.

This is the problem I work on. What follows is the argument in one place.

Decisions moved before the record did

Over roughly three years, automated systems moved from recommending to deciding across credit, care, access, hiring, safety, and enforcement. The move happened quickly and mostly without ceremony — a threshold got raised, a human review step became optional under load, an agent was given the ability to act rather than suggest.

What did not move was the record. The systems around those decisions were built to move work, not to hold reasoning. They log that something happened. They do not preserve why, on what evidence, at what version, considered against what alternatives.

For a while this cost nothing, because nobody asked. That period is ending. Regulators are asking. Auditors are asking. Plaintiffs' counsel has noticed. And the question they ask is always the same one, which is not was your model good but what did you know, when did you know it, and why did you conclude that.

Three failures, one shape

The essays on this site each take one part of it. Together they describe a single structural failure.

The record cannot be reconstructed. Most systems hold a current view of the world. Evidence is overwritten as it updates, deleted when it is retracted, and stamped with one timestamp doing the work of three. The knowledge state that existed at the moment of the decision is gone. Reviewing that decision later means reasoning from evidence the decider never had, which makes every past call look either obvious or negligent, with no way to tell which.

The output is a number instead of an argument. A confidence score is a summary of a reason, and the summary is what got stored. It cannot be audited, disputed, or defended, and in most deployments it has never been validated against measured outcomes — so it is not even an estimate. Worse, scores aggregate. Correlated signals from a single upstream source get combined as though independent, and the system grows more certain the more it repeats itself.

The judgment layer was bought instead of built. Organizations short of decisions have tried to purchase decisions, and the market has obliged with products that convert findings into differently-sorted findings. This cannot work, for a reason that is structural rather than commercial: the context that turns a finding into a decision — what the system does, what depends on it, what was decided about it before — never leaves the building. A vendor deciding without it is deciding without the facts that determine the answer.

The common shape: in each case something essential was discarded because nothing in the design required keeping it.

Defensibility is a property, not a posture

"Defensible" is currently used as a claim about intent. It should be a testable property of a record.

The test I would apply: given only the decision record and the evidence store, can an independent party who was not present rebuild the knowledge state as of the decision and re-derive the conclusion, without access to anything recorded afterward?

That question has a yes or no answer. It can be run against a system today. Most systems fail it, and most teams do not know they fail it until someone asks, which is the most expensive possible moment to find out.

Meeting it requires four things, none exotic and all difficult to retrofit. Evidence that is append-only, so retraction is a new assertion rather than a deletion. Separate clocks for when a fact was true, when it was recorded, and when the decision issued. Evidence that is addressable and versioned, so you know which version was used. And conclusions that carry their justification — the specific evidence, the inference, the alternatives ruled out — rather than a score standing in for all of it.

Why this is architectural

The common response is to treat this as something to add later, once the model works and the pipeline is stable. It does not work later. A system that has been overwriting state for two years has no past to recover, and the assumption of a single mutable present will have hardened into every component that touches it.

This is why the problem is interesting. It is not a compliance feature or a reporting layer. It is a decision made at the start about whether the system is capable of accounting for itself, and organizations are making that decision right now, mostly by default and mostly wrongly.

What I do about it

Three things. I write about the problem, because most teams have not yet named it and cannot fix what they have not named. I maintain DDR-1, an open specification for what a defensible decision record must contain — requirements, vendor-neutral, free to adopt or argue with. And I build these systems, which is the part that keeps the writing honest.

If you want to know where your own systems stand, the conformance assessment takes about four minutes and tells you which requirements you fail. It is uncomfortable for most people who run it. That is the point.