Mathematical and Statistical Soundness¶

updatesupport is mathematically sound for a specific and explicit class of audits:

finite retained refinement cells, fixed public projections, target functionals fixed after workflow compilation, and explicit admissible distribution classes.

In that setting the reported ambiguity interval is a well-defined partial-identification or sensitivity interval. It is not a confidence interval, not a causal identification result, and not a generic nonlinear functional optimizer.

For product positioning and lineage, see Positioning and lineage.

Two Boundaries Up Front¶

First, “hidden” means not publicly reported at the chosen reporting level. It does not mean unobserved, latent, unknowable, or missing from the analyst’s data. In the usual tabular workflow, hidden cells are retained finer cells that the analyst has observed or constructed but has chosen not to publish in the coarse public report. A better mnemonic is:

public = reported buckets
hidden = retained finer refinement inside those buckets

Second, every ambiguity number is relative to a declared refinement and a declared admissible class Q. It is not an absolute bound on all ways the world could differ, all omitted variables, or all possible unmodeled populations. Changing the retained refinement columns, sparse-cell rule, target values, public projection, or Q changes the mathematical problem and can change the bound.

The most important technical boundary is the aggregate target. The primary tabular contract compiles to a fixed linear plug-in target:

psi(q) = sum_d h(d) q(d)

where d ranges over retained finer cells, q(d) is retained-cell mass, and h(d) is the supplied target value for retained cell d.

RatioTarget adds an explicit fixed linear-fractional contract for supported solver backends. MomentTransformTarget adds fixed transforms of linear moments, with current solver support for affine transforms. ProcedureTarget adds a workflow contract for representation-dependent reporting procedures: the procedure is compiled to a column or row metric for each representation, and the finite problem then solves the compiled fixed target.

UncertainLinearTarget keeps the same fixed linear point-estimate target and adds retained-cell estimator standard errors. Those standard errors do not change the base transport optimization. They are used in the report layer to widen the hidden-composition interval with endpoint-specific and conservative estimator-uncertainty margins.

If the aggregate target is nonlinear in q or depends on the transported distribution itself, it is not automatically covered by the current soundness guarantee. Such a target needs an explicit reformulation or a dedicated target-functional backend.

Core Object¶

The current mathematical object is finite.

D is a finite retained refinement state space. These are the cells not shown in the public report, not necessarily unobserved cells.
pi: D -> O maps retained cells to public report buckets.
h: D -> R is a fixed retained-cell target value.
q is a probability distribution over D.
Q is the admissible class of retained-cell distributions.
psi(q) = sum_d h(d) q(d) is the aggregate being audited.

The public law is the pushforward pi#q. For a fixed observed public law p, the primary interval is:

lower = inf { psi(q) : q in Q, pi#q = p }
upper = sup { psi(q) : q in Q, pi#q = p }
ambiguity = upper - lower

This is the hidden-composition ambiguity conditional on:

the retained finite refinement support,
the supplied cell-level target values,
the public projection,
the chosen admissible set Q.

Nothing in this definition says that D is the true complete state space. It is the state space the audit has chosen to retain and stress. If substantively important refinements are absent from D, the interval does not protect against them.

Target Contract¶

The default target contract is:

retained cell d -> fixed scalar h(d)
aggregate      -> q-weighted average of h(d)

This covers many common review metrics when expressed at the right unit:

rates,
means,
average scores,
expected loss rates,
default rates,
loss-given-default averages,
average treatment effects or uplift scores supplied by an external estimator.

The phrase “fixed scalar” matters. Once from_dataframe(...) compiles one finite problem, updatesupport treats the retained-cell target values as inputs. It does not update them when q changes. If a ProcedureTarget is supplied, procedure-aware workflows compile a fresh target for each public representation or scenario before constructing that finite problem.

What Is Not Covered Automatically¶

Many useful targets are not fixed linear functionals in their native form. Those targets are not invalid, but they need explicit handling.

Ratio Targets¶

Some ratios can be made linear by choosing the right weight.

For example, expected loss rate can be written as an exposure-weighted mean:

EL rate = sum_i exposure_i * pd_i * lgd_i / sum_i exposure_i

If exposure defines the row or cell weight, the compiled target is the exposure-weighted mean of pd * lgd, which is linear in the normalized exposure distribution.

But an arbitrary ratio:

T(q) = numerator(q) / denominator(q)

is not the same thing unless the denominator is fixed by the chosen measure or the ratio has been deliberately reformulated. If the denominator can move under the hidden-composition stress test, the current linear interval is not a bound for the native ratio without additional math.

Moment Transforms¶

Targets like:

T(q) = g(E_q[a], E_q[b], E_q[c])

are nonlinear unless g is affine. Examples include squared means, calibrated indices, nonlinear risk scores, and transformed moments.

The current library can audit a supplied scalar cell value such as h(d) = g_d, but that is different from optimizing the global nonlinear functional g(E_q[a], E_q[b], E_q[c]).

MomentTransformTarget makes this boundary explicit. If g is affine and the target declares affine_coefficients, then:

g(E_q[m_1], ..., E_q[m_k])
  = c + sum_j a_j E_q[m_j]
  = sum_d (c + sum_j a_j m_j(d)) q(d)

so the target is a fixed linear target in equivalent retained-cell form. Those affine moment transforms support adequacy checks, interval solving, and public-fiber decomposition.

Non-affine moment transforms are supported only when the declared mathematical shape justifies the requested operation:

if g is convex and a DCP-valid cvxpy_transform is supplied, minimization is a convex problem and the lower endpoint can be exact;
if g is concave and a DCP-valid cvxpy_transform is supplied, maximization is a convex-compatible problem and the upper endpoint can be exact;
if g is monotone in every bounded moment, the library can produce a conservative two-sided interval by optimizing each moment separately and applying g to the monotone box corners.

Those monotone box endpoints may not be jointly attainable. The resulting interval is therefore valid as a conservative bound, not necessarily a sharp identified interval. Non-affine moment transforms also do not claim public adequacy or additive public-fiber decomposition support, because retained-cell point ranges of g(m(d)) are not, in general, the same object as variation in g(E_q[m]).

Distributional Targets¶

Quantiles, tail probabilities at endogenous thresholds, Gini coefficients, AUC, KS statistics, top-decile shares, and approval-rate-at-threshold metrics are not fixed linear averages in general.

Some distributional targets can be bounded with specialized linear, convex, or mixed-integer formulations. The current public API does not claim those bounds.

Representation-Dependent Targets¶

The trickiest case is a target that changes when the public representation changes. Examples:

the analyst refits a model after adding a public bucket,
subgroup estimates are shrinkage-adjusted differently under a finer public table,
bins are recalibrated after refinement,
a public-cell estimator uses within-cell sample size or regularization,
a causal estimator recomputes effects after changing reported covariates.

Then the object is not one fixed psi(q). It is a reporting procedure:

public representation -> compiled target values -> reported aggregate

ProcedureTarget supports this as a workflow-level object. The compiler receives a ProcedureTargetContext containing the public columns, hidden columns, weights, sparse-cell threshold, and Q preset, then returns a concrete column name or RowMetric. Each compiled finite problem is still a fixed-target transport problem. Comparisons across refinements, sensitivity scenarios, or frontier candidates are therefore procedure-comparison sensitivity analyses, not transport intervals for one unchanged target.

Composition-Dependent Structural Targets¶

If the target value itself depends on the transported retained-cell distribution:

h = h(d, q)

then the aggregate is a nonlinear or equilibrium functional T(q), not sum_d h(d) q(d). Examples include interference, capacity effects, market equilibrium, feedback policies, or model outputs that change when portfolio composition changes.

The current adequacy, fiber-range, witness, and refinement formulas do not apply directly to that setting.

Public Adequacy¶

For the current fixed linear target, a support or reporting partition S is adequate when:

S#q1 = S#q2  implies  psi(q1) = psi(q2)

for every admissible q1, q2 in Q.

Public adequacy is the special case where S is the public partition induced by pi. If public adequacy is false, then at least one admissible hidden composition shift preserves the public representation while changing the linear aggregate target.

For a general nonlinear target T(q), the analogous condition would be:

S#q1 = S#q2  implies  T(q1) = T(q2)

That is a different mathematical problem. The current least-support and fiber-range shortcuts are not valid for arbitrary T.

Saturated Reweighting¶

The saturated preset fixes the observed public law and allows arbitrary reweighting among retained cells inside each public fiber.

For the current fixed linear target, let:

range(o) = max_{d: pi(d)=o} h(d) - min_{d: pi(d)=o} h(d)

Then the observed-law saturated ambiguity is exactly:

sum_o p(o) range(o)

The lower endpoint puts each public bucket’s mass on the lowest-target retained cell in that bucket. The upper endpoint puts the same public mass on the highest-target retained cell. Because public buckets are disjoint and the public law is fixed, the bucket-level contributions add. This is why the report can decompose saturated ambiguity into public-fiber contributions:

contribution(o) = p(o) range(o)
ambiguity = sum_o contribution(o)

For saturated environments with an explicitly fixed public law, public fibers with zero admissible mass are irrelevant to adequacy. Target variation in a zero-mass public fiber cannot move the aggregate because no admissible distribution can place mass there.

Linear and Convex Backends¶

The finite linear backend uses scipy.optimize.linprog.

The simplex constraints are implicit: q(d) >= 0 and sum_d q(d) = 1.
Public-law constraints are linear equalities.
User-supplied bounds and linear constraints are added directly.
The objective is the fixed linear target sum_d h(d) q(d).
Linear objectives over polytopes attain extrema at feasible optima, so the reported lower and upper values are the correct LP interval endpoints when the LP solves successfully.

The CVXPY backend currently generalizes the admissible set Q, not the target functional.

The target objective remains linear in q.
Built-in and custom CVXPY constraints define a convex feasible set.
CVXPY solves the lower and upper optimization problems separately.
Dual diagnostics are KKT-style local sensitivity diagnostics for the solved problem; they are not causal explanations or global feature importance.

This distinction matters: CVXPY support does not mean that arbitrary nonlinear aggregate targets are currently supported. It means convex constraints on the admissible retained-cell distribution are supported for the same fixed linear target.

Parameterized CVXPY reuses the same symbolic problem while changing parameters such as a radius. Tests compare parameterized and standard CVXPY backends on the built-in divergence presets.

Meaning of the Q Presets¶

Each Q preset defines an explicit admissible retained-cell composition stress test.

preset	admissible set
`observed`	only the observed retained-cell distribution
`saturated`	fixed public law, arbitrary retained-cell reweighting inside public fibers
`bounded_shift(r)`	fixed public law and `(1-r) q0(d) <= q(d) <= (1+r) q0(d)`
`tv_budget(r)`	fixed public law and `0.5 *
`chi_square_budget(r)`	fixed public law and `sum_d (q(d)-q0(d))^2 / q0(d) <= r`
`kl_budget(r)`	fixed public law and `sum_d q(d) log(q(d)/q0(d)) <= r`
`wasserstein(cost, r)`	fixed public law and transport cost from `q0` to `q` no larger than `r`

Here q0 is the observed retained-cell distribution. The presets do not estimate which stress test is true. They make the stress test explicit and then solve it exactly or numerically under that definition.

Tabular Statistical Semantics¶

from_dataframe(...) compiles rows into the finite linear target contract.

Hidden states are retained finer cells, meaning cells not shown in the public report.
The retained-cell target h(d) is the weighted empirical mean of the target within retained cell d.
The observed retained-cell distribution q0 is the normalized retained cell weight.
The observed public law is the pushforward of q0.

This is statistically sound as a plug-in finite empirical analysis: the optimization treats the compiled cell means and weights as the inputs being audited.

It deliberately does not claim that those inputs are known without error. If cell means are estimated, modeled, survey-weighted, or causal-effect estimates, their standard errors, bootstrap intervals, survey-design uncertainty, or model uncertainty should be accounted for explicitly.

When retained-cell estimator standard errors are supplied through target_standard_error=... or effect_standard_error=..., updatesupport constructs an UncertainLinearTarget. The base interval remains:

lower = inf { sum_d mu(d) q(d) : q in Q, pi#q = p }
upper = sup { sum_d mu(d) q(d) : q in Q, pi#q = p }

where mu(d) is the retained-cell point estimate. The report then adds:

se(q) = sqrt(sum_d (se(d) q(d))^2)

assuming independent retained-cell target-estimation errors after compilation. If lower and upper witness distributions are available, the report evaluates se(q) at those endpoint witnesses and widens each endpoint by confidence_multiplier * se(q).

The report also emits a conservative fixed-public-law outer interval using:

se_conservative = sqrt(sum_o (p(o) max_{d: pi(d)=o} se(d))^2)

This bound is deliberately wider when the retained cell that maximizes estimator standard error is not the same retained cell that maximizes or minimizes the point estimate. It is not an exact joint nonconvex solve over both retained-cell composition and target-estimation error. It is a transparent reporting adjustment that keeps statistical uncertainty and hidden-composition ambiguity visible as separate quantities.

When the selected Q backend is CVXPY-compatible, the report may also include an SOCP confidence-core diagnostic:

core_lower = sup { mu(q) - z se(q) : q in Q, pi#q = p }
core_upper = inf { mu(q) + z se(q) : q in Q, pi#q = p }

The lower-core problem is a concave maximization and the upper-core problem is a convex minimization, so both are disciplined convex/SOCP-compatible. This diagnostic computes the intersection of all composition-specific estimator-adjusted confidence bands. It is not an outer partial-identification interval. A nonempty core means every admissible retained-cell composition shares a common estimator-adjusted value range. An empty core means retained-cell composition shift can separate the estimator-adjusted bands.

The min_cell_weight option changes the retained finite support. That can make reports less sensitive to one-off sparse cells, but it also changes the estimand to the retained support. Data diagnostics report dropped cells, dropped mass, missing category encoding, singleton public fibers, and constant-target fibers so that this choice is visible.

Causal Inference Semantics¶

In causal workflows, updatesupport audits reported effects after a causal estimator has produced row-level, subgroup-level, or retained-cell-level effect values.

The library does not:

identify causal graphs,
choose adjustment sets,
estimate treatment effects,
prove ignorability,
correct interference, positivity, or measurement problems.

It does:

take supplied effect estimates as fixed h(d) values,
hold the public reporting distribution fixed,
stress retained-cell composition under the selected Q,
report how much the aggregate fixed-effect summary could move,
keep causal estimate, statistical uncertainty, hidden-composition ambiguity, and refinement recommendations separate.

If the causal estimator is refit under each representation, or if the effect function changes with the transported distribution, the current fixed-target interpretation no longer applies. That workflow should be documented as a procedure-level sensitivity analysis.

Ratio Targets¶

RatioTarget(numerator, denominator) represents the fixed linear-fractional functional:

psi(q) = sum_d n(d) q(d) / sum_d w(d) q(d)

The current ratio slice is intentionally narrow and explicit:

denominator values must be strictly positive on retained cells;
FiniteEnvironments can evaluate ratios directly on enumerated admissible distributions;
PublicFiberSaturated solves fixed-public-law ratio extrema exactly with a Charnes-Cooper linear-program transform;
CvxpyEnvironments solves local and fixed-public-law ratio extrema with CVXPY disciplined quasiconvex programming (solve(qcp=True));
if the denominator is constant over the state space, the ratio reduces to a fixed linear target and can use the existing linear LP/CVXPY backends;
variable-denominator ratios under the finite-linear LP backend, parameterized CVXPY backend, or pairwise same-support-law global CVXPY problems without a fixed public law still raise UnsupportedTargetError.

For a fixed public law p, the saturated ratio interval solves:

lower = inf { (n . q) / (w . q) : q in Q_sat, pi#q = p }
upper = sup { (n . q) / (w . q) : q in Q_sat, pi#q = p }

The Charnes-Cooper transform sets y = q / (w . q) and tau = 1 / (w . q). The ratio objective becomes linear:

n . y

and the constraints become:

w . y = 1
sum_{d in pi^{-1}(o)} y_d = p_o tau
y >= 0, tau >= 0

This is a linear program, so the fixed-public saturated ratio interval is an exact partial-identification interval for the declared finite problem.

Ratio targets do not have the same additive public-fiber decomposition as linear targets. Reports therefore treat public-fiber tables for ratio targets as point-range diagnostics, not as additive shares of total ambiguity.

What Future Nonlinear Support Would Need¶

A sound nonlinear extension should make the target contract explicit, for example:

LinearTarget(h)
UncertainLinearTarget(mu, se)  # linear point target with report-level SE widening
RatioTarget(numerator, denominator)  # supported for finite/saturated Q
MomentTransformTarget(moments, transform)  # affine, one-sided CVXPY, monotone boxes
CvxpyTarget(objective_builder)

Each target type would need its own adequacy condition, interval solver, witness construction, report language, and tests. Affine moment transforms are already reduced to fixed linear retained-cell target values; convex/concave moment transforms expose only the DCP-compatible endpoint; monotone moment transforms can expose conservative box bounds. Some remaining nonlinear targets are convex or linear-fractional and can be solved cleanly. Some require mixed-integer optimization or only admit conservative bounds. Representation-dependent procedures are currently treated as scenario comparisons through ProcedureTarget, not as a single transported nonlinear functional.

Until a broader target-functional layer exists, unsupported nonlinear targets should either be:

reduced explicitly to a fixed linear plug-in target,
audited through externally computed scalar retained-cell values with the limitation stated,
kept out of the core soundness claim.

The current API includes guardrails for this boundary. UnsupportedTarget can be used as an explicit marker for quantile, distributional, or other nonlinear target objects that are not supported. Passing one into FiniteProblem or from_dataframe(...) raises UnsupportedTargetError instead of silently treating it as a linear target. Passing an uncompiled ProcedureTarget directly to FiniteProblem also raises; pass it to from_dataframe(...) or a procedure-aware workflow instead.

Implementation Checks¶

The test suite exercises the current mathematical contract directly:

saturated closed-form fiber-range formulas,
saturated ratio-target intervals via Charnes-Cooper LP,
affine moment-transform targets and capability guardrails for nonlinear moment transforms,
uncertain linear targets, including retained-cell standard-error aggregation estimator-uncertainty-aware report intervals, and SOCP confidence-core diagnostics,
procedure-target compilation and recompilation across tabular, report, and frontier workflows,
fixed public-law saturated witnesses and zero-mass public fibers,
finite-environment witness construction,
linear-program local and global transport intervals,
the line-segment no-least-support example,
CVXPY DQCP local/fixed-public-law ratio intervals,
CVXPY custom convex constraints,
TV, chi-square, KL, and Wasserstein budget presets,
parameterized CVXPY equivalence to standard CVXPY on the preset grid,
tabular compiler validation and diagnostics,
report/refinement/frontier structured outputs.

The code also validates distribution normalization, nonnegative weights, finite targets, public-law normalization, valid partition refinements, and nonnegative divergence radii/costs.

What Would Make a Claim Unsound¶

The reported interval can be misleading if:

a nonlinear target is silently interpreted as a fixed linear target,
a representation-dependent estimator is refit but described as one unchanged transported target rather than a ProcedureTarget workflow,
the retained refinement space omits important relevant cells,
retained-cell target values are noisy but treated as exact without separate uncertainty reporting,
the chosen Q preset is not a plausible stress test for the application,
min_cell_weight drops substantively important cells,
public buckets encode post-treatment or otherwise inappropriate variables in a causal workflow,
greedy or beam frontier search is read as exhaustive.

These are modeling and interpretation risks, not undocumented behavior in the optimizer. The library is designed to make the finite support, target, public projection, Q preset, diagnostics, and limitations inspectable.

Bottom Line¶

updatesupport is sound for the question it currently asks:

Conditional on the finite retained support, fixed retained-cell target values, selected public representation, and explicit admissible retained-cell composition class, how much can the fixed linear aggregate move while the public distribution is held fixed?

That is a well-defined finite optimization problem. The core library solves it by closed form, linear programming, or convex optimization depending on Q, and reports the result separately from statistical and causal uncertainty.

If the aggregate target is nonlinear or representation-dependent, the project should say so explicitly and either reformulate it as a fixed supported target, use ProcedureTarget for procedure-comparison workflows, or add a dedicated target-functional solver mode.

Mathematical and Statistical Soundness¶

Two Boundaries Up Front¶

Core Object¶

Target Contract¶

What Is Not Covered Automatically¶

Ratio Targets¶

Moment Transforms¶

Distributional Targets¶

Representation-Dependent Targets¶

Composition-Dependent Structural Targets¶

Public Adequacy¶

Saturated Reweighting¶

Linear and Convex Backends¶

Meaning of the Q Presets¶

Tabular Statistical Semantics¶

Causal Inference Semantics¶

Refinements, Sensitivity, and Frontiers¶

Ratio Targets¶

What Future Nonlinear Support Would Need¶

Implementation Checks¶

What Would Make a Claim Unsound¶

Bottom Line¶