Conformal And MAPIE Integration

Conformal prediction and updatesupport answer different questions:

  • conformal tools such as MAPIE quantify row-level prediction uncertainty,

  • updatesupport audits whether an aggregate uncertainty, coverage, risk, or policy claim is stable under hidden subgroup recomposition.

The handoff is:

model predictions -> conformal intervals or sets -> audit-ready row columns
    -> updatesupport conformal stability report

updatesupport does not depend on MAPIE in core. Use MAPIE, another conformal library, or your own conformal procedure upstream, then pass its arrays or columns to the generic conformal adapters.

One-Call Stability Report

After adapting the conformal outputs, call .reporting_stability(...) to audit the conformal-derived targets that are present:

report = adapted.reporting_stability(
    public=["region", "segment", "channel"],
    hidden=[
        "region",
        "segment",
        "channel",
        "cohort",
        "source_quality",
        "rep_team",
    ],
    weight="account_weight",
    candidate_refinements=["cohort", "source_quality", "rep_team"],
    ambiguity_limits={
        "interval_width": 0.03,
        "crosses_threshold": 0.02,
        "miscovered": 0.01,
    },
)

print(report.to_markdown())

For conformal regression, the report discovers available targets such as mean prediction, lower and upper bounds, interval width, empirical coverage, miscoverage, and threshold crossing. For conformal classification, it discovers prediction-set size, ambiguous-set rate, coverage, miscoverage, and positive label containment when those columns are present.

The report is an orchestration layer over ordinary ClaimAudit and PublicReportDesign outputs. Use the lower-level claim workflow when you want one very specific decision rule, repair budget, or target definition.

The repository includes a no-download worked example:

uv run python examples/conformal_reporting_stability.py \
  --output data/conformal_reporting_stability.md

There is also a conformal classification prediction-set example:

uv run python examples/conformal_classification_stability.py \
  --output data/conformal_classification_stability.md

Regression Intervals

For regression, conformal methods usually return a point prediction and a lower and upper prediction interval:

adapted = us.adapt_conformal_regression(
    df_audit,
    prediction=y_pred,
    lower=y_lower,
    upper=y_upper,
    y_true=y_observed,
    threshold=50000.0,
)

The adapter returns a ConformalAdapterResult whose rows include:

  • y_pred,

  • y_lower,

  • y_upper,

  • interval_width,

  • covered and miscovered, when y_true is supplied,

  • crosses_threshold, when threshold is supplied.

If a conformal library returns one interval tensor, pass it as interval=.... The adapter accepts common shapes such as (n, 2) and MAPIE-style (n, 2, n_levels):

adapted = us.adapt_conformal_regression(
    df_audit,
    prediction=y_pred,
    interval=y_intervals,
    interval_index=0,
    y_true=y_observed,
)

For a single target, you can still design the public report directly:

claim = us.claim(
    "Intervals rarely cross the automation threshold",
    public=["region", "segment", "channel"],
    hidden=[
        "region",
        "segment",
        "channel",
        "cohort",
        "source_quality",
        "rep_team",
    ],
    target="crosses_threshold",
    weight="account_weight",
    candidate_refinements=["cohort", "source_quality", "rep_team"],
    decision=us.threshold_decision("<=", 0.20),
    ambiguity_limit=0.03,
)

design = adapted.design(claim)
print(design.to_markdown())

This separates the conformal prediction interval from the hidden-composition stability interval. A model can have useful conformal intervals while the reported automation claim is still unstable under hidden mix shift.

Classification Prediction Sets

For classification, conformal methods often return prediction sets:

adapted = us.adapt_conformal_classification(
    df_audit,
    prediction=y_pred,
    prediction_sets=prediction_sets,
    y_true=y_observed,
    positive_label="approve",
)

The adapter adds:

  • prediction_set,

  • prediction_set_size,

  • ambiguous_set,

  • covered and miscovered, when y_true is supplied,

  • contains_positive_label, when positive_label is supplied.

If prediction sets are encoded as class-membership masks, pass the class order:

adapted = us.adapt_conformal_classification(
    df_audit,
    classes=["approve", "review", "reject"],
    prediction_sets=prediction_set_masks,
    y_true=y_observed,
    positive_label="approve",
)

A natural policy claim is:

claim = us.claim(
    "Prediction sets stay small enough for automation",
    public=["product", "region"],
    hidden=["product", "region", "language", "difficulty", "source_system"],
    target="prediction_set_size",
    weight="case_weight",
    candidate_refinements=["language", "difficulty", "source_system"],
    decision=us.threshold_decision("<=", 2.0),
)

design = adapted.design(claim)

Useful Targets

Good conformal targets for updatesupport are aggregate quantities that a model reviewer or operator would report:

  • mean prediction,

  • mean lower or upper conformal bound,

  • mean interval width,

  • interval-width tail rate,

  • coverage or miscoverage rate,

  • prediction-set size,

  • ambiguous-set rate,

  • threshold-crossing rate,

  • abstention or manual-review rate,

  • risk-control failure indicator.

Each target answers a different review question. For example, coverage asks whether a nominal uncertainty claim is stable; threshold crossing asks whether a decision policy remains stable; prediction-set size asks whether automation burden remains stable.

MAPIE Usage Pattern

The MAPIE-specific code should stay upstream:

# Example shape only; exact MAPIE class names vary by MAPIE version.
mapie.fit(X_train, y_train)
y_pred, y_intervals = mapie.predict_interval(X_audit)

adapted = us.adapt_conformal_regression(
    df_audit,
    prediction=y_pred,
    interval=y_intervals,
    y_true=y_audit,
)

updatesupport only needs the resulting arrays. That avoids coupling core to MAPIE’s API surface while preserving the useful product combination:

conformal prediction tells you how uncertain the model is; updatesupport tells you whether the uncertainty report is stable across the population you care about.