Differentiability gradient audit

progenax’s value proposition is that a differentiable forward model turns cluster inference into information geometry: because every initial condition is jax.grad-able, the Fisher matrix is computable and forecasts what a survey can and cannot measure Cartwright & Whitworth, 2004 (see the science-demos throughline).

That guarantee has a sharp failure mode. The Fisher information is $F = J^\mathsf{T} J$ with $J = \partial(\text{statistic})/\partial(\text{params})$. A silently zeroed column of $J$ makes that parameter look infinitely well-constrained, or makes $F$ rank-deficient — a confident wrong uncertainty, with no error raised. A NaN at least crashes loudly; a zeroed gradient does not. Gradient integrity is Fisher integrity.

This page is the living status of a systematic audit that, for every public sampling and physics entry point, measures the autodiff gradient against a central finite difference and classifies it. Test gate: tests/validation/test_grad_audit.py (parametrised over tests/validation/grad_audit/registry.py); numbers regenerated by scripts/audit_gradients.py → validation/data/grad_audit_results.json (this page cites that file — every ratio below is a real measured number).

How a case is classified¶

Each registry case declares an expect class and the engine computes the status — it is never hand-set:

Tolerances are per class: 10^-5 for closed-form paths (Plummer inverse-CDF, IMF ppf/cdf, Kepler), 10^-3 for ODE/quadrature paths (King/Michie/EFF/multicomponent), and $\sim10^{-2}$ for the linear-interpolation $r_t$ derivative. The central FD uses a relative step $h = h_{\rm rel}\max(|\theta|,1)$, $h_{\rm rel}=10^{-4}$. A confirmed-but-unfixed hazard is encoded as xfail(strict=True) so that fixing it later forces the marker’s removal (a self-cleaning ratchet).

Status — spatial profiles & velocity DFs (params → IC)¶

Generic and edge/boundary parameters; every ratio measured this run.

Status — initial mass functions (params → IC and → summary)¶

Status — differentiable diagnostics (params → summary)¶

Status — multi-component clusters, binaries & rotation (params → IC)¶

The Tier-2 layer: the two multi-component cluster engines, the binary orbital-mechanics layer, and the rotation kinematic overlays. Every ratio measured this run; all clean, no new hazards.

The two linear overlays ($\omega$, $v_{\rm peak}$) are machine-exact ($|\text{ratio}-1|\sim10^{-13}$); the $R_{\rm peak}$ case is genuinely nonlinear (it enters both the $(R/R_{\rm peak})$ prefactor and the $e^{1-R/R_{\rm peak}}$ rotation curve) and so carries the real teeth, FD-consistent to $|\text{ratio}-1|=5.5\times10^{-8}$.

Tier-2 findings (all suspects cleared with a measured number)¶

• The H2 $\psi=0$ boundary is benign (Engine A). The lowered-isothermal mass-CDF draw places stars by mass, and the truncation-shell carries $\sim$zero mass, so no sampled star ever reaches the jnp.where(psi>0, ...) support mask or the max(\psi,0) sampling clamp. Measured at the most extended $W_0=3$ edge (the boundary most stretched, $r_t=4.26$): the minimum sampled $\psi(r)$ is 0.153 (max sampled radius $3.66 < r_t$), with 0/400 stars hitting the clamp and 0/400 categorical-assignment flips at $h$. The $W_0=3$ edge is FD-consistent ($|\text{ratio}-1| = 2.6\times10^{-4}$). The shared tidal radius $r_t$ is itself differentiable in $W_0$ through from_imf (Tier-1 unclamped-$\psi$ fix, ratio 1.000000).

• The discrete is_binary / is_circular branches do not block the $q$/$e$ gradient (MoeCompanions). The $(P, q, e)$ draw is a smooth grid-CDF inverse; the only discrete selections are the is_binary Heaviside mask and the is_circular ($e\to 0$) gate. Measured at the baseline: 0/400 is_binary flips and 0/400 is_circular flips at $h$, so the discrete selection injects zero finite-difference noise and the mask-independent $\langle e\rangle$ channel flows cleanly (ratio 1.000000).

• The single-star sanitization is NaN-safe (a Fisher-integrity win). The mixed-is_binary resolve_binary_components case (25 binaries + 25 singles) activates the $a_{\rm safe}/e_{\rm safe}/m_{2,\rm safe}$ sanitization path. jacfwd of the resolved positions w.r.t. the per-element $a$ is NaN-free, with $\mathrm{d}/\mathrm{d}a = 0$ on every single slot (a finite zero, not a NaN) and a live non-zero gradient on every binary slot — directly verifying the sanitization cannot leak a NaN into a Fisher matrix.

• Engine A/B $r_t$ and the OM anisotropy are differentiable. The Engine B density-defined Eddington/OM build (Poisson quadrature + Abel/Eddington inversion, no ODE) is FD-consistent in the prescribed-density scale leaves ($r_h$, $\gamma$) and in the Osipkov–Merritt anisotropy radius $r_a$ (whose gradient lives in the velocities). The rotation overlays are exact: the solid-body $\omega$ gradient matches the closed form $\langle|\hat{n}\times\mathbf{r}|\rangle$ to $<10^{-12}$, and the nonlinear differential-$R_{\rm peak}$ derivative is FD-consistent to $5.5\times10^{-8}$.

Status — binned-kinematic Fisher path (params → summary)¶

The Tier-3 layer: the binned summary statistics a Fisher matrix $F = J^\mathsf{T}J$ is actually built from, with $J = \partial(\text{binned summary})/\partial\theta$. The radial bin edges are frozen (the observer fixes the bins; gradients flow through the model that fills them). The dispersion $\sigma(r)$ and anisotropy $\beta(r)$ bin values move smoothly with the parameters, so their gradients are genuinely live; the $N(r)$ count is the illustration that the Fisher gradient lives in the model $p_k(\theta)$, not the frozen data.

The $\sigma(r)$ case carries the mutation-check teeth: wrapping the sampled $(\text{pos}, \text{vel})$ in jax.lax.stop_gradient before the binner collapses AD to 0 exactly while the finite difference stays live ($\sim -3.35$), proving the case differentiates the real params → summary channel and not a constant (test_grad_audit.py::test_binned_sigma_mutation_has_teeth).

The $N(r)$ split is the arc’s headline principle made concrete. A Poisson number-density likelihood writes the expected per-shell count $\mu_k(\theta) = N_{\rm total}\,p_k(\theta)$, where $p_k = F(r_{k+1}) - F(r_k)$ and $F(r) = M(<r)/M = r^3/(r^2+a^2)^{3/2}$ is the new public PlummerProfile.enclosed_mass_fraction CDF. The gradient flows through $\mu_k$ — i.e. through the model $p_k$ — and is closed-form machine-exact (ratio 1.0). The observed binned count $N_k$, by contrast, is a sum of indicator functions on frozen edges: its autodiff gradient is identically 0 (a finite, correct-by-design answer), while a finite difference registers a nonzero discrete step as stars cross the frozen edges. That FD step is the data-side bin-crossing the audit treats as frozen and out-of-scope; AD $=0$ is the right answer, so the case is pinned known_blocked (it passes because AD is finite — not known_zero, which would mis-flag the nonzero FD as a hazard). Frozen data is out of scope; the Fisher gradient lives in the model.

Tier 4 — the registry as the release gradient-gate¶

Tiers 0–3 built the harness and a 57-case coverage map. Tier 4 makes the registry the single source of truth for gradient correctness — the release gradient-gate — and consolidates the scattered ad-hoc gradient tests the codebase had accumulated.

A read-only inventory catalogued every released-core test whose assertion is about a gradient (autodiff finiteness, AD-vs-FD consistency, gradient sign/value, differentiability-through-a-path). Each was classified migrate / keep / delete:

• ~24 duplicate AD-vs-FD assertions were migrated into the registry. These re-derived a finite-difference comparison the registry already owns (or now owns); the duplicate assertion was removed and the scattered test carries a one-line pointer comment to the registry Case that is the source of truth.

• ~15 finite-only isfinite(grad) smoke tests were deleted. This is audit finding T6: an assert jnp.isfinite(jax.grad(f)(x)) with no finite-difference comparison gives false confidence — a silently zeroed gradient passes it. The silent-zero is exactly the Fisher-corrupting failure mode this arc exists to catch, so a test that cannot see it is worse than no test. The registry’s $|\mathrm{AD}|>\varepsilon$ guard, paired with the FD comparison, has the teeth those smoke tests lacked.

• ~62 tests were kept — genuine non-gradient physics, or unique gradient properties the registry should not absorb: the find_alpha custom-VJP / implicit-function-theorem regression, the $\alpha=1$ pins (forward-value smoothness plus, post-S4, AD $=$ FD at exactly $\alpha=1$ — properties the registry rows do not assert), the Engine-B $\beta(r)$ anisotropy anchor, the energy-kernel double-where softening=0 guards, and the uncovered-channel grad-correctness pins (binary period/eccentricity distributions, mass-ratio samplers, the environment-IMF $\alpha_3$ relations). Each kept test gained a registry-pointer comment.

The migration surfaced 9 new entry-point × parameter Cases the duplicates exercised but the registry did not yet own — added in Tier 4 and visible in the tables above tagged (Tier 4): KingVelocityDF.sample_velocities$(r_c)$, EFFProfile.sample_positions$(a)$, EFFVelocityDF.sample_velocities$(a)$, PowerLawIMF.ppf$(m_{\min})$, the three Kepler Jacobian columns (KeplerElements.to_state in $a$ and $M_0$, and KeplerElements.from_state), the Michie density observable ($r_c$, $W_0$), and the standalone H6 boundary probe. (The Michie density observable’s two parameter channels count as one Case here, matching its single dual-param table row — so the gate’s full row count is 66, of which these are the nine additions.) The H6 case was promoted: it had been a folded edge on the Chabrier $m_c$ Case (which never actually drove $u$ to the clamp); it is now a standalone Case with the real $u = 10^{-12}$ boundary draw, measured FD-consistent at ratio 0.999669.

A safety interlock guarded the consolidation: a fast per-file gate after each edit and a full released-core gate per logical batch (new-cases, migrates, deletes). The interlock earned its keep — the migrate batch surfaced two real coverage gaps the duplicates had quietly covered (the Michie density-$W_0$ channel and the EFF positions-$a$ channel), which were then added as Cases rather than lost. Tier 4 grew the gate from 57 to 66 cases and found 0 new hazards.

Tier 5 — the self-policing release gate¶

Tiers 0–4 measured gradient integrity and made the registry the gate; nothing yet enforced it. A future commit could reintroduce a silent-zero, drift the registry away from its committed numbers, or add a new public sampler with no gradient case — and CI would not notice. Tier 5 makes the gate permanent, self-policing, and complete: a manifest pins what the registry must cover, three CI layers fail loudly on any drift, and the D4 completion closes the last deferred Fisher-coverage holes.

The manifest — the gradient-coverage source of truth¶

tests/validation/grad_audit/manifest.py holds three hand-curated, independently-frozen structures (deliberately not derived from the registry or __all__ at runtime — a derived manifest could not catch a deleted case or a new symbol):

• MUST_AUDIT — the (id, param) coverage units the registry must keep covering, each with a one-line rationale. The ratchet checks the live registry against this literal, so a deleted case reds CI. (Several MUST_AUDIT ids are internal/submodule entry points legitimately absent from progenax.__all__ — the frozen-edge binners, q_approx, lambda_msr_approx, Schechter, IMFParams, the Plummer+OM velocity DF — and are ratcheted by a direct (id, param) set check.)

• SYMBOL_CATEGORY — a complete map from every progenax.__all__ symbol to either AUDITED (must resolve to ≥1 registry case) or a typed EXEMPT_* reason (PROTOCOL, CONTAINER, ANALYTICAL_IC, NON_FISHER_DIAGNOSTIC, HELPER, COVERED_ELSEWHERE). The Layer-3 invariant is a keys-equality (set(SYMBOL_CATEGORY) == set(progenax.__all__)), not a frozen count, so it tracks the live public-API size (reported on the Test Dashboard); a newly-added public sampler is absent from the map, so the keys-equality check fails and the symbol cannot ship ungated.

• PARAM_ALLOWLIST — the carry-forward known-limitations, each tied to a registry known_blocked/expect and a reason: the binned-count data side (binned_number_density, AD $=0$ correct-by-design) and the du-monotonicity data-side property. (The $\alpha=1$ IMF branch points formerly carried here were fixed by audit S4 and now audit as consistent.)

The three CI enforcement layers¶

A dedicated gradient-gate CI job runs all three layers as pytest tests, so local pytest tests/validation enforces them identically to CI (one engine, no CI-only path):

1. Run-gate (Layer 1). test_grad_audit.py asserts every registry case is clean/known-limitation, and scripts/audit_gradients.py exits 0 iff 0 hazards. A reintroduced silent-zero reds this layer.

2. Staleness comparator (Layer 2). test_json_fresh.py regenerates the audit into a temp path and semantic-diffs it against the committed validation/data/grad_audit_results.json — exact on the discrete/structural projection $(\text{id}, \text{direction}, \text{param}, \theta, \text{expect}, \text{tol}, \text{finite}, \text{status})$, rtol-tolerant on the floats $(\text{AD}, \text{FD}, \text{ratio})$ (rtol measured cross-arch). The registry and its committed measured numbers can never silently drift apart.

3. Coverage ratchet + __all__ cross-check (Layer 3). test_manifest_coverage.py asserts every MUST_AUDIT (id, param) is in the live registry and that set(SYMBOL_CATEGORY) == set(progenax.__all__) with every AUDITED symbol resolving to ≥1 case. A deleted case, a removed coverage unit, or a new/ungated public entry point reds CI.

Plainly: CI fails on a reintroduced silent-zero, a registry↔JSON drift, or a manifest / __all__ entry point without a case.

D4 — closing the deferred Fisher-coverage holes¶

Tier 5 also completes the gate. Eleven cases close the holes the earlier tiers had deferred, each landed measured-first (≥3 seeds, ±3σ band, no weakened tolerance), growing the gate 66 → 77 with 0 new hazards:

• Engine-A velocity-scale ratio w_j (from_components) and equipartition OM anisotropy radius r_a (from_mass_segregation) — finite-only smoke before, now FD-consistent.

• build_binary_cluster end-to-end — the full IMF→companion→spatial assembly gradient in $r_h$, machine-exact (the assembly is $r_h$-linear, AD/FD ratio 1.0 to $\sim10^{-13}$).

• Binary orbital-distribution params — three period samplers, four eccentricity samplers, and the IndependentCompanions thermal e_max through the full companion assembly.

The closed-form anchors landed exactly: LogNormalPeriod gives $\mathrm{d}\langle\log_{10}P\rangle/\mathrm{d}\mu = 1$; the Uniform/LogUniform channels recover $\langle u\rangle = \tfrac12$ (0.4939); and ThermalEccentricity recovers $\langle\sqrt{u}\rangle = \tfrac23$ (0.6620).

The gate is currently enforced via local pytest; the dedicated gradient-gate CI job is wired into the tests aggregator and fires when CI runs.

Intentional non-differentiable sites¶

These are documented, deliberate limitations — not bugs. The audit pins them so an unannounced change fails loudly.

• IMF singularity at $\alpha = 1$ — FIXED (audit S4), no longer a limitation. The historical exp_safe double-where guard removed the $\int m^{-\alpha}\,\mathrm{d}m$ singularity but selected an $\alpha$-independent $\log$ branch at exactly $\alpha=1$, so AD there was finite yet silently wrong (ppf: AD $=0$ vs FD $=-1.384\times10^4$). The segment kernels now use the expm1/log1p-stable forms (progenax.numerics.power_integral_stable / power_ppf_stable) — one smooth expression in $e = 1-\alpha$ — so the gradient is FD-exact at exactly $\alpha=1$ (and at $\alpha=2$ for mean_mass’s numerator). Audited consistent; pinned in tests/unit/imf/test_imf_gradients.py::TestAlphaOneGradients and the registry edges.

Changelog of fixes¶

What this proves, and what it does not¶

The audit proves that across the released spatial/kinematic/IMF samplers and the packaged differentiable diagnostics, the parameter gradients used to build a Fisher matrix are finite and finite-difference-consistent — or are explicitly pinned limitations. It is a gradient coverage map: the entries that say “clean” are the ones a forecasting study can trust, and the two fixed rows are entries that would previously have produced a confidently wrong Fisher element. The audit does not cover non-public internals. (The formerly-pinned known_blocked $\alpha=1$ points were eliminated by the audit-S4 expm1-stable kernels and now audit consistent — see the fixed-site note above.)