Talts et al. (2018)

The big idea¶

Every Bayesian analysis runs an inference algorithm (MCMC, HMC, variational, INLA) whose correctness is rarely checked — “we always get some result from a given algorithm, [but] we have no idea how good it might be without some form of validation.” Simulation-Based Calibration (SBC) is a generic procedure that validates whether an algorithm samples the posterior correctly, for any model you can simulate from, using only forward draws — no analytic posterior required.

The whole method rests on one exact self-consistency identity. If you draw a ground truth from the prior and data from it, the posterior averaged over all such datasets returns the prior (the data-averaged posterior, Eq. 1):

Any discrepancy between the data-averaged posterior and the prior signals a broken posterior computation or a mis-implemented model. SBC turns this identity into a sharp, easy-to-read test. It is the corrected, sampling-friendly successor to Cook, Gelman & Rubin (2006), which used continuous CDF values that suffer discretization artifacts and need continuity corrections; SBC replaces those with discrete rank statistics that are exactly, not asymptotically, uniform.

Core results¶

The rank statistic (§4.1). Draw a prior sample θ̃ ~ π(θ), data ỹ ~ π(y|θ̃), and L posterior draws {θ_1,…,θ_L} ~ π(θ|ỹ). For any one-dimensional function f(θ), the rank of the prior draw among the posterior draws,

is uniformly distributed over the integers {0,…,L} when the posterior samples are genuine and independent (Theorem 1, proof in their Appendix B). This is exact for any joint distribution π(y,θ) — the lognormal-style normalizations and model details all cancel.

The SBC histogram + uniform band (Algorithm 1). Repeat for N simulated datasets, binning the N rank statistics into a histogram over the L+1 possible values. Under correct calibration the histogram is flat. Each histogram carries a gray band showing 99% of the variation expected from uniformity: the band runs from the 0.005 to the 0.995 percentile of Binomial(N, (L+1)^{-1}) (or Binomial(N, 1/B) for B merged bins), so on average only ~1 bin in 100 strays outside it. For variance reduction, merge neighbouring bins to keep N/B ≈ 20; choosing L+1 a power of two (e.g. L = 1023) makes the re-binning clean. Uniformity can also be quantified with a χ² test of the binned counts against the expected N/B per bin.

Deviation shapes are diagnostic (§4.2). How the histogram departs from flat tells you how the posterior is wrong:

• ∩-shaped (excess in the middle) → the computed posterior is over-dispersed (too wide) relative to truth.

• ∪-shaped (spikes at both ends, ranks pushed to 0 and L) → under-dispersed (too narrow) — or, for MCMC, autocorrelated posterior draws (Figure 4).

• Asymmetric / sloped → the posterior is biased in the opposite direction (ranks biased low ⇔ posterior biased high).

Autocorrelation and thinning (§5.1, Algorithm 2). Theorem 1 assumes independent posterior draws. MCMC chains are correlated, which clusters draws relative to the prior sample and biases ranks toward the extremes — the same ∪-spikes as under-dispersion. The fix is to thin the chain to ≈independent states: keep every T-th draw so the effective sample size N_eff[f] ≤ N; antithetic chains (e.g. dynamic HMC with N_eff > N) should be thinned by 2 first. They suggest sizing the thinning from the minimum N_eff over a set of test quantiles (e.g. 19 equispaced quantiles of f(θ)).

Honest scope (§4). SBC validates only the computational/inference aspect “under the assumed model”: a green SBC means the algorithm samples the posterior correctly for the model as written. It is not a guarantee that the posterior covers the truth for any single dataset (that needs sensitivity analysis) or that the model is rich enough to match reality (that needs posterior predictive checks). SBC is “similar to checking the coverage of a credible interval under the assumed model.”

Use in progenax¶

The experimental gravoturb differentiable-inference layer uses SBC as workstream ① of its trustworthiness arc (see Differentiable inference — natal cloud parameters from cluster substructure), upgrading the α-recovery result from single-θ injection–recovery (AC16) to calibration across the whole prior:

• The rank statistic (2) is compute_sbc_rank_histogram (added to jaxstroviz analysis/inference.py); the SBC driver (inference/sbc.py) runs the Algorithm-1 loop (draw θ* ~ BM19Prior → simulate the mock → run NUTS → rank θ*), and AC18 asserts per-parameter rank uniformity via a χ² test (p > 0.05).

• The uniform band is plot_sbc_rank_histogram (jaxstroviz plots/inference.py): the Binomial(N, 1/B) 0.5%–99.5% shaded band.

• Thinning (§5.1) is the SBC driver’s n_thin — posterior draws are thinned toward independence before ranking, so the ∪-spikes in AC18 reflect real miscalibration, not HMC autocorrelation.

• Honest scope is load-bearing here: the SBC mock is drawn from the same BM19 model the likelihood fits, so a green AC18 certifies the inference engine is calibrated under BM19 — not that real molecular clouds follow BM19. That external/misspecification test is a separate workstream (③).

Notes¶

• SBC is the corrected successor to Cook et al. (2006) (continuous CDF values → discretization artifacts); it shares the self-consistency identity (1) with the Geweke (2004) Gibbs-sampler validator but avoids the auxiliary sampler’s convergence problems.

• The procedure is embarrassingly parallel across the N simulated datasets — the cost is N full posterior fits — which is why the gravoturb SBC acceptance test is slow-marked and run at a reduced grid/N in the test wrapper, full N in the acceptance driver main().

• HMC convergence diagnostics (R̂, bulk/tail-ESS, divergences, BFMI) are the complementary per-fit check (AC19); SBC is the across-the-prior calibration check. Both are needed.