Validation methodology

progenax adopts a three-tier test architecture that separates mechanical correctness (unit), end-to-end behaviour (integration), and physical correctness (validation). Each tier has its own purpose, frequency, and tolerance convention.

The three tiers¶

Unit tests run on every push and must complete in $< 5$ minutes total. Integration tests run on PR merge and must complete in $< 30$ minutes. Validation tests run nightly (or on PR merge to main) and may run $\sim 4$ hours.

The validation tier runs only on pre-merge to main — running it on every push would slow developer iteration. The trade-off: a developer pushing to a feature branch sees only unit + fast-integration results, so a validation-tier regression surfaces at the merge step. The mechanics of the gate that enforces this — the FAST/FULL split and the four-part release gate — are at Testing architecture.

Tolerance conventions¶

Tests express these via pytest.approx(expected, rel=tol) or pytest.approx(expected, abs=tol). A regression that exceeds the declared tolerance blocks merging.

Anchor patterns¶

The single most important methodology lesson from progenax’s validation history is anchor on the defining condition, not the derived constant:

Bad anchors test internal constants that can be silently mis-derived; good anchors test the physical condition the constant should satisfy. The PP20 ζ(p) transcription bug shipped past internal tests because those tests asserted the buggy values; it would have been caught immediately by anchoring on $\zeta(1.5) = \sqrt{2}$.

Three failure modes the anchor pattern has caught¶

The defining-condition anchor is not abstract — it has empirically caught three distinct classes of regression in progenax’s history:

1. Closed-form transcription bugs — the 2026-04-28 PP20 ζ(p) bug (PP20 ζ(p) transcription fix) was caught by anchoring on the analytic value $\zeta(1.5) = \sqrt{2}$; the buggy formula gave $\infty$. Without the anchor the bug would have shipped unnoticed, because every other test was self-consistent with the buggy formula.

2. Inversion bugs — the Plummer half-mass-radius bug (see the historical note in Plummer profile), where $a = r_h/\sqrt{2^{2/3}-1}$ was used instead of $a = r_h\sqrt{2^{2/3}-1}$, was caught by test_half_mass_radius checking $M(<r_h) = M/2$ directly rather than asserting $a =$ some constant.

3. Differentiability regressions — accidentally introducing jnp.where (instead of a sigmoid) in a critical path is caught by the per-builder gradient-finiteness tests (Differentiability rules).

Each is now anchored by a regression test on the defining condition, not the derived constant.

Adding a new validation test¶

The four-step pattern:

1. Identify the defining condition — what is the physical or mathematical statement that the function being tested is supposed to satisfy? (E.g. virial theorem $2T + V = 0$.)

2. Compute the expected value — analytical (closed form) or published (with citation).

3. Choose a tolerance based on the convention table above.

4. Write the test with pytest.approx and a descriptive docstring referencing the source.

Example:

def test_plummer_virial_at_equilibrium():
    """Plummer with matched DF must satisfy 2T + V = 0
    (virial theorem) to within Poisson noise.

    Reference: any standard textbook; the closed-form Plummer DF
    in [](../10-theory/velocity-dfs/plummer-dfs.md) is the source of
    truth.
    """
    # Q=None -> the faithful UNSCALED Plummer equilibrium (no virial rescale), so
    # 2T + V = 0 is a genuine test of the matched DF, not a value imposed by the
    # builder. (The default Q=0.5 would force 2T + V = 0 by construction.)
    ic = build_plummer_cluster(n=10_000, r_h=1.0, key=jax.random.PRNGKey(0), Q=None)
    T = compute_kinetic_energy(ic.velocities, ic.masses)
    V = compute_potential_energy(ic.positions, ic.masses, G=STELLAR.G)
    assert (2 * T + V) / abs(V) == pytest.approx(0.0, abs=5e-3)

The docstring cites the source of truth; the assertion uses the defining condition; the tolerance follows the finite-$N$ statistical convention.

When validation tests fail¶

A failing validation test is a physics claim that is no longer true. Three responses, in priority order:

1. Investigate the change. Most validation failures are signals of real regressions — buggy code, broken refactor, unintended parameter change. Find the root cause.

2. If the test was wrong, fix the test (with a clear commit message explaining the prior error). progenax’s PP20 ζ(p) transcription bug is the canonical example.

3. If the physics has changed, update the test and document the change in the dev log.

What is not OK: weakening a tolerance to make the test pass without explaining why. The test exists because the original physics was correct to that tolerance; if the new code produces a value $5\times$ further away, the new code is suspect.

Reproducibility¶

Every validation test uses a fixed PRNG key and fixed numerical tolerances. Re-running on the same hardware reproduces results bit-for-bit. Re-running on different hardware (CPU vs GPU) reproduces results to float-precision tolerance.

The fixed PRNG key is deliberate: it makes regressions reproducible and bisectable. progenax does not run tests with random seeds in CI because the resulting flakiness obscures real regressions.

References¶

The three-tier methodology is standard practice in scientific software; the specific tolerance conventions follow common practice in the JAX ecosystem (NumPyro, BlackJAX). The “anchor on defining condition” lesson came from the 2026-04-28 PP20 ζ(p) bug fix (PP20 ζ(p) transcription fix), which the validation suite caught only after the analytic anchor was added.