The Counterfactual Decision Ledger: Evaluate AI Recommendations Without Inventing the Road Not Taken

Method and evidence boundary: This is an authored operating method. Every facility, person, system, decision, action, score, probability, and result in the examples is fictional. The method is not evidence of a product deployment, customer result, legal conclusion, certification, service level, or causal effect. Any operational experiment needs an appropriate owner and review for safety, privacy, fairness, employment, consumer, contractual, and other applicable requirements.

An AI system recommends that a self-storage manager call one rental lead first, inspect one maintenance item sooner, or move one access anomaly to the top of a review queue. The manager takes a different action. A week later, the team looks at the outcome and says the model would have done better.

That conclusion is usually unsupported.

The team observed what happened after the action it took. It did not observe what would have happened, for the same decision in the same circumstances, after the unchosen action. The missing outcome is not hiding in a dashboard. It is the road not taken.

This is the core difficulty of evaluating decision systems. Predictive accuracy asks whether a forecast matched a later observation. Policy evaluation asks what outcomes would occur if one decision rule were used instead of another. The second question is causal. It requires more than recording model scores beside business results.

A counterfactual decision ledger makes that boundary operational. It preserves exactly what was known, which actions were eligible, what the system recommended, what a person saw and chose, what was actually executed, what outcome was observed, and why a row is—or is not—eligible for a particular evaluation method. It turns shadow mode from a demo into an evidence-producing process without pretending that shadow mode proves impact.

Open the full-size accessible diagram.

Governed companion package

Inspect the decision ledger and evaluation gates.

These exact files contain an authored method and fictional or blank-contract examples. They do not establish a product deployment, customer result, causal effect, model validity, source authority, experimental approval, performance level, certification, or independent validation.

Download the source and limitations register ↓Download the counterfactual decision-ledger template ↓Download the evaluation-readiness suite ↓Download the accessible decision-ledger SVG ↓Download the decision-ledger PNG ↓

Start with the one-outcome rule

For a single decision, imagine two potential outcomes:

• the outcome if action A were taken; and
• the outcome if action B were taken.

Only one action is taken, so only one of those outcomes can ordinarily be observed for that decision. This is a foundational idea in the potential-outcomes framework. Hernán and Robins describe causal effects using hypothetical interventions and make clear that, in observed data, the factual treatment and outcome are known while the alternative potential outcome is not (Causal Inference: What If).

In a self-storage operation, the decision might be which maintenance issue to inspect first. If the manager inspects a reported door problem at 10:00 a.m., the operation can observe the subsequent facts for that choice: what the inspection found, how long the issue remained open, whether a temporary control was applied, and when verified closure occurred. It cannot also observe what would have happened to that exact issue had the inspection waited until 3:00 p.m.

The unobserved outcome is not zero. It is not blank because someone forgot to enter it. It is counterfactual.

This yields the first ledger rule:

An observed outcome may be attached only to the action that was actually executed and independently read back. Every unchosen action must remain explicitly marked counterfactual—not observed.

That label prevents a common analytical error: scoring the model by comparing its recommendation with the outcome produced by somebody else's action.

Shadow mode has value, but it does not prove value

Shadow mode runs a candidate decision system without granting it operating authority. The system can receive the same eligible context, generate recommendations, and record them while the existing human or rule-based process remains in control.

This can answer useful questions:

• Does the system receive the required inputs at the decision time?
• Does it identify the same unit, event, tenant, lead, work item, and facility as the operator?
• Are candidate actions within policy and site capabilities?
• Are scores stable enough to inspect and reproduce?
• How often is the recommendation late, missing, ineligible, or internally contradictory?
• How often would the proposed policy disagree with the current policy?
• Are disagreements concentrated at particular facilities, times, event classes, or data-quality states?
• Can reviewers understand the decision packet and identify unsafe or nonsensical proposals?

Those are compatibility, integrity, and disagreement measurements. They are valuable. They are not causal outcome measurements.

A shadow recommendation does not cause the operation's observed outcome when the recommendation was never executed. If the current policy chose a same-day inspection and the shadow system proposed next-day inspection, the same-day result cannot be counted as the shadow system's success. If the shadow recommendation matches the chosen action, the row may be more informative, but matching alone still does not solve selection bias, confounding, changing context, or the absence of an evaluation design.

The NIST AI RMF 1.0 is a voluntary, non-sector-specific framework for managing AI risk across the lifecycle. Its Core supports testing under conditions similar to deployment, documenting methods and uncertainty, monitoring in production, and recording limits to generalizability. Those practices strengthen evaluation. They do not convert unexecuted recommendations into observed effects. NIST's current program page says version 1.0 is being revised, so the exact source version and check date should travel with the method.

Six states must remain separate

Many failed evaluations collapse an entire decision into one row called recommendation. A defensible ledger separates at least six states.

1. Eligible context

The exact facts available before the decision: facility, time, source versions, event identity, candidate unit, data freshness, missing fields, active policy, capacity, and relevant operational constraints.

Context must be frozen by identity or content hash. An after-the-fact database export may contain corrections and fields that were not available to the decision maker. Using those later facts as if they were known at decision time creates leakage.

2. Candidate action set

The actions that were genuinely available at that moment. A model cannot be fairly evaluated for choosing an action that local policy prohibited, the facility could not execute, the tenant had not consented to, or staffing made impossible.

The ledger should record exclusions and their authority. “Not selected” is different from “not eligible.”

3. Recommendation and exposure

The model or rule output, including version, score, rank, score meaning, generation timestamp, and policy filters. The ledger must also record what a person actually saw. A recommendation generated in a backend but never displayed cannot explain a human choice.

Exposure is more than a yes/no flag when formal evaluation is contemplated. If a randomized or stochastic logging policy decides which action is presented or selected, the probability assigned by that policy may be essential. A probability reconstructed later from a different model is not the same evidence.

4. Choice and authority

The action selected by the responsible person or system, the time of selection, the authority used, and a bounded reason code. Free text can add context, but it should not replace structured reasons such as insufficient_identity, site_capability_unavailable, customer_contact_preference, safety_hold, or operator_judgment.

An override is neither automatically good nor automatically bad. It is a decision that needs a reason, authority, and outcome path.

5. Execution and readback

Selection is not execution. The ledger should record whether the action was attempted, accepted by the downstream system, independently observed, partially completed, reversed, or never carried out.

Only the executed action can be linked to a factual outcome. If a work order was queued but never dispatched, the system should not evaluate the recommendation as if the inspection occurred.

6. Outcome and observation window

The outcome definition must exist before analysts inspect results. It needs a unit, numerator, denominator, observation window, censoring rule, source, and direction of preference.

“Resolved quickly” is not an outcome contract. A better definition might be: verified_closure_within_48h, measured from qualified work-item creation to independent closure readback, excluding rows censored by facility closure under a declared rule.

The ledger also records co-interventions: a storm, staffing change, vendor outage, manager call, rate change, marketing campaign, or policy update that may affect the outcome. Recording a co-intervention does not automatically adjust for it. It makes the limitation visible.

The 26-field minimum decision packet

The exact storage format can vary, but the semantic contract should not. I use twenty-six minimum fields grouped into seven blocks.

Decision identity

1. decision_id
2. facility_id
3. decision_type
4. decision_time_utc
5. context_snapshot_id

Candidate and policy context

6. candidate_action_id
7. eligibility_state
8. eligibility_reason
9. policy_id_and_version

Recommendation provenance

10. recommender_id_and_version
11. recommendation_generated_at_utc
12. rank_or_score
13. score_semantics
14. display_state

Logging policy

15. logging_policy_id_and_version
16. action_probability
17. probability_provenance

Human or system disposition

18. chosen_action
19. choice_authority
20. choice_reason_code

Execution truth

21. execution_state
22. execution_receipt_or_readback

Outcome and analysis boundary

23. outcome_definition_id
24. outcome_observation_state
25. censoring_and_co_interventions
26. analysis_eligibility_and_limitations

In practice, a production schema will expand several of these fields. The accompanying template uses one row per candidate action so the chosen and unchosen branches cannot overwrite one another.

The provenance structure has a direct precedent in W3C PROV-O, which models entities, activities, and agents and supports qualified attribution, association, use, and derivation. PROV-O does not define this ledger. It demonstrates why “the model said” is an inadequate provenance statement.

Distinguish display probability from action propensity

One of the most dangerous fields in an evaluation ledger is a number labeled probability without a contract.

It might mean:

• the model's estimated probability of an event;
• a normalized score across candidate actions;
• the chance an action was displayed;
• the chance the logging policy selected an action;
• a calibration estimate;
• a human confidence judgment.

Those quantities are not interchangeable.

Off-policy methods often rely on the probability that the historical logging policy chose an action in a given context. Research on learning from logged bandit feedback treats those propensities and the variance they create as central, not optional metadata (Swaminathan and Joachims, 2015). If the past process was an undocumented human choice, entering 0.5 because there were two visible buttons does not recover the propensity.

The ledger should therefore store both the numeric value and its provenance:

• computed by which policy version;
• before or after eligibility filtering;
• over which candidate set;
• with what randomization mechanism;
• logged at what time;
• whether the probability was actually used to select the action;
• whether the record was altered or imputed later.

When that evidence is absent, the honest value is unknown, and the row is not eligible for an estimator that requires known propensities.

Four evaluation levels

The ledger should assign each analysis to the strongest level its evidence actually supports.

Level 0 — replay integrity

Can the system reproduce the recommendation from the pinned context, model, policy, and candidate set? Can it explain missing or changed output? This tests determinism, version identity, input completeness, and provenance.

It says nothing about operational benefit.

Level 1 — shadow compatibility

How often was a recommendation generated on time, eligible, exposed, understandable, and executable? Where did it disagree with the active policy? What data-quality or site-capability problems appeared?

This level supports descriptive rates with exact denominators. It does not support “would have improved.”

Level 2 — counterfactual or off-policy estimation

Under a suitable design, logged data may support estimates of how a different policy would perform. The technical literature provides several approaches, including direct outcome models, inverse-propensity weighting, and doubly robust estimators. Bottou and colleagues give a broader account of counterfactual reasoning for learning systems and controlled experiments in computational advertising (JMLR, 2013). The domain is not facility operations, and its results should not be transplanted without a valid operational design.

These methods are not spreadsheet formulas to apply after the fact. They depend on conditions such as:

• consistency: the action being analyzed is well defined and corresponds to the intervention represented in the data;
• exchangeability or adequate control of confounding: conditional comparisons are not driven by unmeasured reasons for action selection;
• positivity or overlap: each action the target policy might choose had a nonzero chance of being chosen in relevant contexts;
• accurate treatment of time, interference, censoring, missingness, and changing policies;
• correct propensity or outcome models where the estimator requires them;
• enough effective sample size after weighting;
• an outcome definition that is measured consistently.

Dudík, Langford, and Li address the partial-feedback setting in which rewards are observed only for chosen actions and develop a doubly robust approach combining outcome and past-policy models. The phrase doubly robust is easy to misuse. It does not mean twice as trustworthy, immune to poor overlap, safe under arbitrary hidden confounding, or valid when both component models are wrong.

Any Level 2 estimate needs a named method owner, diagnostics, uncertainty, sensitivity work, and an explicit statement of which assumptions remain unverified. If the team cannot explain why a row is eligible, it should remain at Level 1.

Level 3 — controlled operational experiment

A bounded randomized or otherwise defensible prospective design can create more direct evidence, provided the action is appropriate to experiment on and the design has the necessary operational, ethical, privacy, legal, and safety review.

The experiment must define:

• the decision unit and randomization unit;
• eligibility and exclusion rules;
• treatment versions and contamination controls;
• primary outcome and observation window;
• guardrails and stop conditions;
• power or precision target;
• analysis plan;
• operator override and appeal paths;
• incident response and rollback;
• who may start, pause, expand, and end the experiment.

Not every action belongs in an experiment. Access-state changes, tenant rights, employment decisions, emergency response, life-safety work, and legally sensitive communications need especially conservative treatment. A low-consequence queue-ordering or information-presentation question may be more suitable than an action that changes a person's rights or physical access.

A fictional self-storage example

Consider three fictional maintenance items at Site Alder:

• WORK-101: a unit-door alignment concern reported by a tenant;
• WORK-102: an office light flicker reported by staff; and
• WORK-103: debris observed near a drive aisle.

The current policy gives the manager a governed queue with safety and access-related exclusions already applied. A shadow model ranks WORK-101 first. The manager chooses WORK-103 because a local delivery is scheduled and the aisle must be inspected before the vehicle arrives.

The ledger records all three eligible actions as candidate rows. It records the shadow rank, what the manager saw, the structured reason, and the actual dispatch. The outcome for WORK-103 is observed under the chosen action. The outcome for WORK-101 under “inspect first” remains counterfactual—not observed.

What can the team learn?

• whether the shadow model received the delivery schedule in its context;
• whether that schedule should have affected eligibility or ranking;
• whether the manager's reason code identifies a recurring missing signal;
• whether the recommendation was early enough to use;
• whether the model's candidate set matched the governed queue;
• how often similar disagreements occur.

What can it not honestly claim from this row?

• that the model would have reduced closure time;
• that the manager made the wrong choice;
• that ranking WORK-101 first would have improved tenant experience;
• that the shadow system saved labor or prevented a loss.

The row is operationally useful precisely because it refuses to fabricate those conclusions.

Define the outcome before looking

An outcome selected after results are visible can make almost any policy look favorable. The outcome contract should be versioned before the evaluation window begins.

For a maintenance-prioritization use case, the contract might include:

• decision unit: qualified work item at first eligible queue entry;
• primary outcome: verified closure within 48 hours;
• safety guardrail: no missed emergency-class escalation;
• quality guardrail: reopen within seven days;
• start event: governed work-item creation;
• end event: independent closure readback;
• censoring: facility inaccessible due to declared emergency closure;
• exclusion: duplicate or identity-unresolved work item;
• direction: higher verified-closure rate is preferred only when guardrails remain within bounds;
• evaluation window and analysis version.

This prevents the team from quietly switching from closure quality to response time, or from excluding inconvenient rows after inspecting outcomes.

The NIST AI RMF Playbook's Measure suggestions encourage documenting assumptions, comparing predeployment and production behavior, tracking drift and feedback loops, and comparing AI with simpler or human baselines. Those are sound habits. The Playbook is voluntary, not a certification checklist, and NIST says it will be updated after the AI RMF revision.

The feedback-loop trap

Once people see recommendations, the system can change the data it later learns from.

A manager may investigate an item because it was ranked highly, add notes that would not otherwise exist, or change how work is coded. Staff may learn which reason codes preserve their discretion. A model may suppress a candidate action that then disappears from downstream records. These are not merely model-drift problems. They are policy-induced changes to observation.

The ledger should record:

• whether the recommendation was hidden, displayed, selected, or merely generated;
• when the operator first saw it;
• whether the display changed the information-gathering process;
• whether the model or policy was retrained during the window;
• whether outcome capture depends on the chosen action;
• whether decisions at one facility affect capacity or outcomes at another;
• whether the same tenant, unit, work item, or staff member appears across decisions.

Ignoring these relationships can understate uncertainty and overstate sample size. Ten thousand rows are not ten thousand independent decisions when they come from repeated interactions with the same small set of facilities and people.

Human review needs its own measurement

Human-in-the-loop is not an evaluation method. A person can add context and authority, but can also introduce inconsistent thresholds, automation bias, fatigue, delay, or selective documentation.

Measure the human-system configuration:

• exposure rate and display latency;
• agreement, override, and abstention with exact denominators;
• reason-code completeness;
• time to disposition;
• distribution by facility and decision class;
• escalation and appeal rate;
• post-choice execution mismatch;
• whether reviewers had the required evidence;
• whether disagreement review is independent of model developers;
• whether a recommendation changes later labeling.

Do not interpret agreement as correctness. A high agreement rate may mean the model learned the old policy, reviewers defer to it, or both are responding to the same incomplete data. A low agreement rate may reveal missing context, poor recommendations, different objectives, or an unclear policy. The ledger makes the disagreement investigable without assigning credit prematurely.

The architecture

A durable counterfactual ledger has six layers.

1. Decision clock and identity — assigns a stable decision ID and freezes the eligible context at the decision time.
2. Candidate-set and policy resolver — records what actions were available and why others were excluded.
3. Recommendation observer — captures model version, output, timing, score semantics, and display state without granting authority.
4. Choice and execution reconciler — joins human or system selection to downstream acceptance and independent readback.
5. Outcome observer — applies a versioned definition and window, preserving censoring and co-interventions.
6. Evaluation gate — assigns replay, descriptive, off-policy, or experiment eligibility and refuses analyses whose assumptions or provenance are missing.

Each layer should be append-oriented. A correction creates a new attributed assertion rather than silently rewriting what was supposedly known at decision time. The ledger should distinguish event time, observation time, decision time, execution time, and outcome-window closure.

This architecture does not require a particular database or AI platform. It requires stable identity, provenance, versioning, immutability or equivalent history, access controls, and a governed correction path.

Acceptance tests before analysis

Before a dashboard reports policy value, run a conformance suite against the ledger.

At minimum, refuse or downgrade analysis when:

• decision or candidate identity is missing;
• the context snapshot was captured after the choice without a trustworthy historical source;
• the candidate set is incomplete;
• score meaning is undocumented;
• generation and display are conflated;
• chosen action and executed action differ;
• no independent readback exists for a consequential action;
• an observed outcome appears on an unchosen branch;
• the outcome definition changed after inspection;
• censoring or co-interventions are blank where required;
• the logging policy or action propensity is reconstructed without provenance;
• target-policy actions have insufficient overlap;
• extreme weights collapse the effective sample size;
• a doubly robust result is presented without both component-model diagnostics;
• repeated units or cross-facility interference are ignored;
• the production policy changed during the evaluation without segmentation;
• a Level 1 shadow analysis is labeled causal;
• uncertainty, sensitivity, and generalizability limits are absent.

The accompanying suite contains beginner, intermediate, advanced, and architect cases. It deliberately tests false conclusions, not only missing fields.

What AI may do inside this method

AI can help:

• propose candidate reason-code groupings for human review;
• detect impossible timestamps or state transitions;
• compare context schemas across facilities;
• identify missing provenance;
• summarize disagreement clusters without assigning causal credit;
• draft test cases from a governed outcome contract;
• flag possible overlap and weight-instability problems for a qualified analyst;
• explain why a row was downgraded from Level 2 to Level 1.

These are proposed uses, not deployment or performance claims.

AI should not invent a propensity, infer an unobserved outcome, approve its own experiment, erase an override, redefine the outcome after seeing results, or label an association causal. A language model's confidence is not an action probability under the historical logging policy.

A 30-day operator exercise

An operator can learn from this method without running an experiment.

Week 1 — choose one bounded decision

Select a low-consequence, repeatable decision such as the order in which already-qualified maintenance items enter a human review queue. Write the decision unit, candidate actions, eligibility rules, authority, and exclusions.

Week 2 — freeze context and outcome contracts

Define the context snapshot and source versions. Choose one primary outcome and guardrails, with numerator, denominator, start, stop, window, censoring, and readback rules. Do not activate a new policy.

Week 3 — run hidden and visible shadow checks

First capture recommendations without showing them to operators to test input and output integrity. If authorized, run a separate visible-shadow phase and record exposure. Keep the phases distinct because visibility can change behavior.

Week 4 — audit evidence eligibility

Run the conformance suite. Report replay integrity, coverage, latency, eligibility, disagreement, missing context, and reason-code completeness. Mark counterfactual outcomes as unobserved. Do not publish an estimated causal effect unless a qualified method owner has approved the design, assumptions, diagnostics, and uncertainty.

The most valuable result may be a refusal: the historical process did not record propensities, candidate-set overlap is unknown, and outcome capture depends on the chosen action; therefore this dataset supports Level 1 descriptive analysis only.

That is not a failed project. It is a truthful finding that tells the operator what evidence must be designed into the next phase.

The standard to hold

Applied AI authority will not come from placing a model score beside an operational KPI and calling the difference impact. It will come from knowing which question the evidence can answer.

A counterfactual decision ledger gives a self-storage operator that discipline. It preserves the recommendation without granting it credit, preserves human judgment without romanticizing it, preserves the factual outcome without assigning it to an unchosen action, and preserves the assumptions required for more advanced evaluation.

The standard is simple to state and demanding to follow:

Record the decision that existed, the action that happened, and the outcome that was observed. Treat every other path as a question until an appropriate design makes it evidence.

That is how shadow mode becomes an operating instrument instead of a retrospective story machine.