Causal Inference for Shared Compute

In shared compute, the "treatment" is often a policy rather than a single static action. A router, rate limit, cache policy, or capacity allocation rule is dynamic, and chooses an action based on the request-level workload and the live system state.

In the router example, premium-fleet assignment is not simply "treated." The same request may be routed to premium capacity when the queue is shallow and sent to a cheaper fleet when the system is congested. The estimand must attach to the allocation rule, not merely to the serving path that happened to execute.

System State Is Both Confounder and Outcome

Shared inference systems make the timing of state measurement central. At time t, system state is an input to the current routing, admission, or fallback decision. At time t + 1, that same class of state variables can be downstream of earlier policy decisions.

This loop is what makes shared compute different from a standard product A/B test. Queue depth before a routing decision can be a confounder. Queue depth after the policy has run can be part of the policy's effect. When an experiment changes the queue, cache, or fleet state that later requests experience, the unit-level treatment label is no longer the whole story.

Fat-Tailed Costs and Capacity Risk

For AI infrastructure and inference products, a plain difference-in-means readout can be fragile when the metric is tokens, cost, or GPU-hours per user. Both of those outcomes can be dominated by a small number of very large users, long-context workflows, or high-concurrency windows.

Tokens per request plausibly follow something like a truncated power law. The mean is finite because the context window bounds every request, but the variance is large and likely growing as use cases become more heterogeneous and context windows expand with each model release. If that shape is roughly right, the distribution is estimable: the mean exists, it just converges slowly, and a handful of long-context or agentic workflows can dominate any short experiment.

True resource cost per request is harder. It depends on live system state: batch concurrency, KV-cache pressure, prefill/decode balance, GPU efficiency, and temporal demand. The effective capacity boundary moves in a way the context-window limit does not, so the conditional mean of resource cost may not settle inside the experiment window.

KV-cache memory scales with sequence length and batch size, so unrelated long contexts arriving concurrently can push a fleet past its memory boundary even when every individual request looks ordinary. The tail event is not only "one expensive user." It can be an overcommitted fleet, cache-pressure spirals, elevated fallback rates, or a crash. A rare peak-concurrency window can erase the margin earned on the other requests served in that window.

A useful discipline is to ask what you would measure if the experiment were perfect. If every request could be randomly routed to a performant or standard cluster with no interference, no capacity limits, and full observability, what would that experiment tell you? That line of questioning exposes the causal target and the measurement constraints, so the design can approximate the ideal experiment rather than merely summarize the logs.

1. Name the action pattern. Is the decision to ship broadly, target a segment, change a routing rule, reserve capacity, keep a holdout, pause, roll back, or instrument before acting?
2. Choose the comparison. The counterfactual might be the old policy, untreated users, a different cluster, a prior model checkpoint, a threshold-adjacent group, or a different time block.
3. Frame the answer. Broad rollout, treated-population impact, targeting, and policy value answer different operational questions.
4. Use unit economics as a guardrail. A policy can improve task success while burning too much compute. Pair retention or task completion with latency, errors, GPU-hours, and cost per successful request.
5. End with a rule. Ship, ramp, target, hold out, pause, roll back, or instrument more. The readout should state the recommendation and the tradeoff that would change it.

Decision Pattern Cheat Sheet

Experiments still start with the same question: what is the unit that receives the policy, and what other traffic changes because that unit was treated? If the policy does not change shared state, a user or request RCT may be enough. If it changes queue, cache, or fleet state, the experiment has to randomize a larger unit or time block.

Observational work should not oversell the estimator. The central question is whether the historical comparison has the right pre-decision variables, enough overlap, and a credible assignment story. Unlike an experiment, confounding is not solved by design. It is the problem.

Observational Estimators

Bad Controls in Shared Compute Logs

Quasi-Experimental Designs

These are strongest when the business already created quasi-random variation: a phased rollout, a threshold, a beta invite, or one region/fleet/provider changing before the rest. Pick the design based on the assignment story, then state the assumption that could fail.

Offline policy evaluation is useful for routers, schedulers, ranking policies, and allocation rules, but only if the logs contain enough examples of the actions the new policy wants to take. Check support before estimating anything.

Logged propensities can support short-horizon contextual routing evaluation, especially when the action affects the current request and the relevant state was logged. They are less sufficient for full dynamic-policy evaluation when actions alter future queue and cache state. In that setting, IPS or doubly robust estimates can help screen policies, but live validation is still needed before broad deployment.

1. Check support. Did the old policy try the same actions for similar state often enough?
2. Check propensities. If action probabilities were logged, IPS and doubly robust methods become more credible.
3. Inspect weights. If a few records drive the answer, the offline estimate is too fragile for a broad rollout.
4. Validate live. Even a good OPE result should usually become a limited ramp or controlled exploration bucket before full deployment.

Heterogeneity should change the allocation rule, not merely decorate the analysis. If the result does not change who gets capacity, which workload is routed differently, or which customer segment is worth serving differently, it is mostly decoration.

1. Start with the average effect. That shows whether there is a real lever worth segmenting.
2. Pre-specify operational segments. Good segment definitions include workload length, customer tier, model family, region, latency sensitivity, and peak versus off-peak demand.
3. Use CATE only if targeting is the decision. The segment has to be stable, interpretable, and simple enough to operate.
4. Convert lift into net value. A high treatment effect can still be a bad allocation if it uses expensive capacity for low-margin traffic.
5. Keep some exploration alive. If the router only exploits the current best segment, future evaluation loses support for alternatives.

Short-run metrics are useful, but they should not be treated as business outcomes by default. First state what part of the mechanism they measure, then explain why that mechanism should translate into retention, expansion, or LTV.

Surrogate Ladder

1. System metric: TTFT, latency, cache hit rate, error rate, retry rate, or cost per successful request.
2. Session outcome: task completion, abandonment, re-run rate, or successful workflow completion.
3. User behavior: return rate, deeper usage, broader adoption, or more high-value workflows.
4. Business outcome: retention, expansion, renewal, LTV, or margin.

The ladder can break. A faster request is not automatically higher retention, and more usage can still hurt margin if the workload is expensive or low value. A metric that predicts retention is not necessarily a valid surrogate for retention. A valid surrogate claim requires evidence that movement in the short-run metric captures the causal path by which the policy changes the long-run outcome.

If the long-run outcome takes too long to observe, the recommendation should be conditional: ramp only if the mechanism metric improves, the cost and reliability guardrails hold, and early behavior moves in the direction past experiments suggest. For policies expected to affect retention or expansion, keep the smallest durable holdout that can answer the long-run question. If no holdout is possible, label the surrogate risk explicitly rather than pretending the short-run metric proves the whole case.

The check is not there to show diligence for its own sake. It should change the recommendation: ship, ramp, pause, roll back, narrow the target population, or instrument more.

Common Decision Patterns

This table belongs at the end because the preceding sections explain why these fields matter. Decision-time state, intended action, executed action, and system guardrails are not generic telemetry. They are the minimum evidence needed to distinguish the policy effect from the state that caused the policy to act.

Methodological references: potential outcomes and bias, randomized experiments, graphical causal models, Causal Inference for the Brave and True chapters on propensity scores, doubly robust estimation, DiD, synthetic control, RDD, IV, and heterogeneous treatment effects, plus Why fat tailed costs emerge at scale.