OED for robustness — when binaries lie to your mass estimate (Stage 4)

The three OED demos before this one each end with the optimiser rediscovering a piece of physics: anisotropy wants the outskirts, concentration wants the core, dynamical mass wants an interior depth. Each is a clean result — and each is one a referee could wave away as “Fisher-OED recovers the obvious.” This page is different. Its headline is not a recovery; it is a bias. We hand a careful, binary-blind observer the optimal survey design for weighing a cluster, let the real cluster harbour the binaries that every real cluster has, and watch the analysis over-estimate the mass by 184% while reporting a 4.5% error bar — a measurement that is confidently, expensively wrong. Then we show the design that protects against it.

Why this arc exists: a bias a referee cannot dismiss¶

The sibling OED demos are strong engineering — one differentiable Fisher backbone, re-pointed at three science targets — but each headline is either already-known physics or contingent on an arbitrary RV/PM error split that flips with the budget. None is a result a referee couldn’t read as “of course the optimiser put the proper motions where the signal is.”

This demo monetises a capability nobody else has wired to OED: progenax carries a faithful, differentiable Moe & Di Stefano (2017) binary-population engine, so we can ask the question that actually bites in real cluster mass measurements — what does an unmodelled population systematic do to an “optimal” design, and what design defends against it? The answer is a parameter bias larger than its own forecast uncertainty, which is the one kind of result a Fisher-OED demo cannot be accused of staging. It is the canonical binary $M/L$ systematic of dwarf-galaxy and young-cluster dynamics, turned into a design problem.

The physical question: why binaries masquerade as mass¶

You weigh a star cluster with the virial theorem: a hotter cluster — larger line-of-sight velocity dispersion $\sigma_{\rm los}$ — is a heavier one, $M \propto \sigma^2 R / G$. That logic has a silent assumption: that every velocity you measure is a single star orbiting the cluster. It is not. A large fraction of massive stars live in binaries, and in a single-epoch radial-velocity survey you cannot resolve the binary — you measure the star’s orbital velocity around its companion folded into its cluster velocity. Those orbital motions are fast (tens of km/s for short-period massive binaries), so they inflate the measured dispersion above the true cluster value.

The inflation is not noise — noise averages out. It is a systematic offset in the second moment. If the cluster’s true dispersion is $\sigma_{\rm cluster}$ and a fraction $f_{\rm bin}$ of the stars carry a binary velocity drawn from a population with variance $V_{\rm bin}$, the observed dispersion-squared is

a flat pedestal $f_{\rm bin}V_{\rm bin}$ added on top of the real cluster heat at every radius. Feed $\sigma_{\rm obs}$ to a virial analysis that does not know about the pedestal, and the analysis reads the extra dispersion as extra mass. For our young-massive-cluster fiducial the pedestal is $\sqrt{f_{\rm bin}V_{\rm bin}}=6.9$ km/s sitting under a central cluster dispersion of only 9.0 km/s — and the cluster term falls to 0.4 km/s in the outskirts, while the pedestal does not budge. That is the whole story: in the cold outskirts the binary pedestal dominates the signal.

The intuition the optimiser is about to fall for — and then defeat¶

Hold one fact in mind: the cluster mass $M$ scales the dispersion amplitude and shows up at every radius; the binary fraction $f_{\rm bin}$ adds a flat offset. The only thing that tells them apart is radial leverage — the cluster dispersion has a shape $\sigma(R)$ that falls from core to edge, while the binary pedestal is flat. A design that ignores binaries chases the loudest signal into the cold outskirts and parks its budget exactly where the pedestal is largest relative to the cluster — and so confuses pedestal for mass most badly. A design that models binaries instead spends its stars on the core↔outskirts contrast that breaks the $M\leftrightarrow f_{\rm bin}$ degeneracy. Same instrument, same budget, opposite outcome.

Inputs and assumptions¶

The design optimises a fixed budget of $N_{\rm total}$ radial-velocity measurements allocated across $K=12$ log-spaced projected-radius bins — RV only, single-epoch. The science target is the dynamical mass $M$; the EFF shape parameters $(\gamma, a)$ are photometrically pinned nuisances, and the anisotropy radius $r_a$ and binary fraction $f_{\rm bin}$ are kinematic nuisances the radial allocation must disentangle. Everything is computed pre-data — the cross-model Monte Carlo afterwards is a calibration gate, not part of the design loop.

The Fisher is built in the dimensionless $\ln\theta$ metric Important, so $\sigma(\ln M)$ is a fractional precision. The pinned scales are the load-bearing ones: $\sigma_{\rm cluster,central}=8.98$ km/s and $\sigma_{\rm bin}=9.73$ km/s, a contamination ratio $\sigma_{\rm bin}/\sigma_{\rm cluster}=1.08$ at the centre — rising to $\sim24$ in the outskirts, because the cluster term collapses while the pedestal stays flat. That the binaries rival the cluster heat at this operating point is pinned before any inference, exactly so that H1 below has a chance to bite.

The modelling choice: EFF-OM, RV-only, and a held-fixed cluster¶

Two facts about the forward model set up the whole demo, and stating them plainly is the honest way to read the result.

1. EFF density is analytic — no ODE, no OOM. The Elson–Fall–Freeman profile has a closed-form density $\rho \propto (1+r^2/a^2)^{-\gamma/2}$, so project_dispersion does a plain quadrature with no differential-equation solver in the tape. This is what makes the cross-model Monte Carlo runnable — the King/Michie ODE solvers in the concentration demo forced its Michie calibration to OOM at $\sim$28 GB. The EFF slope $\gamma$ doubles as a concentration knob, so the bias vector spans $(M, r_a, \gamma, a, f_{\rm bin})$ with no ODE anywhere. We parameterise by the EFF scale radius $a$ directly rather than a derived half-mass radius, because EFF has no closed-form $r_h(a,\gamma,r_t)$ and inventing one would inject a spurious unpinned inversion.

2. The cluster forward model is the held-fixed CONTROL. This is a misspecification study, and a misspecification study is only clean if it changes one thing. So the EFF-OM cluster model — density shape, OM anisotropy law, RV projection — is identical between the design, the truth, and both fits. The only difference between the “disaster” and the “fix” is whether the analysis models the binaries. Every number on this page is the cost of that one omission, with the cluster physics deliberately frozen so it cannot be blamed.

The method, specialised: c-optimality with a misspecified vs marginalized Fisher¶

This demo pours the additive backbone Equation into a c-optimal-for-$M$ radial allocation, exactly as the sibling designs do. One jacrev of the cluster term at the truth gives the per-bin Jacobian $J=\partial\sigma_{\rm los}/\partial\ln\boldsymbol\theta$ (reverse-mode by policy); the design enters only through the per-bin counts. The criterion minimises the $M$ variance with the nuisances profiled out:

The whole demo is the contrast between two Fishers built on this one equation:

• Binary-blind ($\boldsymbol\theta=(M,r_a,\gamma,a)$, no $f_{\rm bin}$): the observer who does not know binaries exist. Its $4\times4$ Fisher omits the $f_{\rm bin}$ column entirely.

• Binary-aware / marginalized ($\boldsymbol\theta=(M,r_a,\gamma,a,f_{\rm bin})$): adds the $\partial\sigma_{\rm los}/\partial\ln f_{\rm bin} = f_{\rm bin}V_{\rm bin}/(2\sigma_{\rm obs})$ column (the pluggable Fisher block) and profiles $f_{\rm bin}$ out as a free nuisance.

Because the Fisher is additive and design-linear, the expensive projection is differentiated once; the $f_{\rm bin}$ grid for the maximin criterion below only updates the $\sigma_{\rm obs}(f_{\rm bin})$ denominator — no re-jacrev.

The headline (H1): a confident, expensive, wrong mass¶

The pre-registered hypothesis (H1, locked before the Monte Carlo ran) was: fitting the binary-blind model to binary-contaminated RV data biases $\hat M$ high, by more than the design’s own forecast $\sigma(M)$ — false confidence. The rival H0 was that the binary-blind fit would absorb the flat pedestal into its nuisances and stay unbiased. H1 is confirmed, decisively.

The binary-blind c-optimal-for-$M$ design, fit with the binary-blind model on mock catalogues that do contain Moe binaries, recovers

The estimate is nearly triple the truth, and the design’s own error bar — the precision it promised — is 41 times smaller than the error it actually made. This is the worst failure mode in inference: not a wide error bar (which warns you), but a tight one around the wrong answer.

The control is what makes it airtight. Run the same design and fit on mocks with $f_{\rm bin}=0$ — where the fit model exactly matches the generative model — and the bias is $-0.3\%$, well inside the forecast. The entire $+184\%$ is the binaries; nothing else moved.

The fix (measure-and-marginalize): model the binaries, recover them, lose the bias¶

The remedy is not to throw away the outskirts; it is to model the pedestal. Add the $f_{\rm bin}$ parameter to the fit (and to the design Fisher), let the data determine it, and the bias collapses:

• Bias: $+184\% \to +5\%$. The binary-aware fit on the same contaminated mocks recovers $\hat M$ within its honest forecast — the cross-model residual bias is consistent with zero ($+5\%$, inside $2\sigma$ of the marginalized forecast).

• It recovers the binary fraction: $\hat f_{\rm bin}=0.50\pm0.08$. Not from a prior — from radial leverage alone. The flat pedestal and the falling cluster shape are separable, and the design’s core↔outskirts contrast measures both.

• Honest precision: $\sigma(M)/M=6.9\%$, up from the blind design’s claimed $4.5\%$. Marginalizing over an unknown $f_{\rm bin}$ genuinely costs information — and saying so is the point. The blind $4.5\%$ was a fiction; the aware $6.9\%$ is the real cost of not knowing the binary fraction in advance.

That recovery of $f_{\rm bin}$ comes with a caveat sharp enough that it gets its own section below — the mock is generated at the truth $f_{\rm bin}$ and the prior is centred there, so the unbiased recovery alone proves only self-consistency. What proves $f_{\rm bin}$ is genuinely identifiable is the prior-insensitivity test, not this number.

H2 — the binary-aware design is also tighter¶

The fix is not only a better fit; it is a better design. Comparing the two designs under the same binary-aware (marginalized) fit, the binary-aware allocation reaches a target $\sigma(M)$ with fewer stars:

confirming the pre-registered H2 threshold ($\geq1.3\times$). This margin is thin at the fiducial ratio $\sigma_{\rm bin}/\sigma_{\rm cluster}=1.08$ on purpose — and the sweep shows it grows to $\sim6\times$ as binaries come to dominate. H3 is confirmed too: the binary-aware allocation is not a monotone rescaling of the blind one — it reshuffles, pulling budget toward the radii that constrain $f_{\rm bin}$ to break the $M\leftrightarrow f_{\rm bin}$ degeneracy (the blind design spread over $\{1.5, 7.6, 17\}$ pc; the aware design drops 7.6 pc and concentrates 2.26 pc while keeping 17 pc).

The honest test of identifiability: prior-insensitivity, not truth-centred recovery¶

This is the caveat to read before quoting the $\hat f_{\rm bin}=0.50\pm0.08$ recovery as proof that binaries are measurable. The cross-model mock is generated at the truth $f_{\rm bin}=0.5$, and the binary-aware fit’s prior is centred on that same value. So an unbiased recovery, by itself, demonstrates only self-consistency — the fit returns the value it was pointed at. That is necessary but not sufficient.

What actually proves $f_{\rm bin}$ is identifiable from the data is a prior-insensitivity test: loosen the $f_{\rm bin}$ prior by $100\times$ and the recovered $\hat f_{\rm bin}$ moves by only $0.4\%$. The posterior is set by the radial leverage in the data, not by the prior — which is the genuine evidence that the core↔outskirts contrast measures the binary fraction. We state this plainly because it is the difference between a real measurement and a tautology, and the truth-centred recovery number alone cannot tell them apart.

Robustness, honestly: maximin is a near-null¶

The measure-and-marginalize fix assumes you trust the Moe & Di Stefano $f_{\rm bin}$ enough to model it. What if you do not — what if you want a design that is good across a whole range of possible binary fractions you refuse to commit to? That is maximin (robust) design: minimise the worst-case $\sigma(M)$ over $f_{\rm bin}\in[0, 0.7]$.

The honest answer here is a near-null, and reporting it as such is part of the integrity of the arc. Because $\sigma(M)$ is monotone in $f_{\rm bin}$ (more binaries $\Rightarrow$ a larger pedestal $\Rightarrow$ less mass information), the worst case always sits at the upper endpoint $f_{\rm bin}=0.7$ — and the marginalize design, already built to handle a free $f_{\rm bin}$, is already nearly maximin-optimal. The maximin design gives up only $+0.16\%$ of precision at the truth to gain $+0.24\%$ at the worst case: a $\sim0.2\%$ hedge. The lesson is not “maximin wins” — it is that measure-and-marginalize is the right tool here, and the expensive worst-case machinery buys almost nothing on top of it. A null reported faithfully is a finding.

The sweep: the bias is catastrophic where binaries dominate¶

The fiducial operating point ($\sigma_{\rm bin}/\sigma_{\rm cluster}=1.08$) is one slice through a continuum. Sweeping the system mass — which sets $\sigma_{\rm cluster}$ while $\sigma_{\rm bin}$ stays fixed — traces the bias across contamination regimes, and it is the regime-of-validity map that turns a single number into a story.

The binary-blind-fit bias runs from $+26\%$ (massive, hot clusters where binaries are a minor perturbation) to $+850\%$ (low-mass, cold clusters where the pedestal swamps the cluster heat). Across the entire sweep the binary-aware fix holds the residual bias at $\approx0$. And the H2 precision-gain grows from $\sim1.3\times$ at the fiducial point to $\sim6\times$ where binaries dominate — so the thin operating-point margin is the floor, not the typical case. The colder the cluster, the more the binary-blind design lies, and the more the binary-aware design is worth.

Validating a pre-data calculation: two gates¶

A Fisher forecast is a promise — and this demo’s promise is the unusual one that a design will be biased. Both halves are checked.

Gate 1 — the gradients are correct (AD-vs-FD)¶

The load-bearing new derivatives are the $f_{\rm bin}$ Fisher block and the binary $\sigma^2$-inflation term. We verify them against finite differences to $\mathrm{rel}<10^{-3}$ (Richardson where a fixed-step FD is truncation-limited), per gradient-validation. The analytic $f_{\rm bin}$ column $f_{\rm bin}V_{\rm bin}/(2\sigma_{\rm obs})$ matches AD to $\sim10^{-16}$.

Gate 2 — the bias is real (cross-model Monte Carlo)¶

The bias claim is gated by a cross-model Monte Carlo: generate WITH Moe binaries, fit WITHOUT. The mock is forward-model-consistent — the cluster mock is the fit model, so the $f_{\rm bin}=0$ baseline is unbiased by construction and any bias is attributable to binaries alone. Each draw places the design’s per-bin star counts directly, draws each star’s cluster velocity from $\mathcal{N}\!\left(0,\sigma_{\rm los}^2(R)\right)$ with $\sigma_{\rm los}$ from project_dispersion at the truth, adds a flux-weighted Moe blend $\Delta$ (from a build-once $K_{\rm orb}$ pool rescaled to $\mathrm{Var}=V_{\rm bin}$) to the per-star $\mathrm{Bernoulli}(f_{\rm bin})$ fraction, adds per-star $\varepsilon_{\rm RV}$, forms the per-bin $\hat\sigma$, and fits $\hat M$ by MAP in the $\ln\theta$ Gauss–Newton metric (honest realized-$\hat\sigma$ weighting; unpopulated bins dropped, not floored). The H1 accept rule was fixed in advance: accept iff the mean $\mathrm{bias}(\hat M)/M > 2\,\sigma_{\rm forecast}/M$ and the $2\,\text{SEM}$ band does not straddle that threshold. It is met by a wide margin (bias/forecast $=41\times$; SEM 0.06). The binary-aware fit’s accept rule — $|\mathrm{bias}| < 2\,\sigma_{M,\rm marg}$ — is met by the $+5\%$ residual.

The pre-registration (locked before the Monte Carlo)¶

To keep the bias result honest, the hypotheses and accept/reject rules were locked before the Monte Carlo ran, via research-workflow:discriminating-experiment-design. A reject on any of them was defined in advance as a reportable finding, not a failure — and an H1 reject would have descoped the whole arc.

Quantitative results¶

All numbers are from the gated CLI’s run-record (EFF-OM YMC operating point, $N_{\rm total}=5000$, $K=12$ bins, cross-model MC 48 draws):

What the optimum means: science implications¶

1. “Optimal” is only optimal under a correct model. The whole demo is a warning about the silent assumption inside every OED (and every analysis): the design that minimises your forecast variance under the wrong model can maximise your real error. A tighter forecast is not a better measurement — it is only a better measurement if the model is right. The binary-blind design’s $4.5\%$ was the most dangerous kind of number.

2. Population systematics are a design problem, not just an analysis problem. You cannot fix binaries purely after the fact if your survey spent its budget in the contaminated outskirts. The binary-aware design spends differently because it knows the pedestal exists — it earns the $f_{\rm bin}$ constraint from radial leverage at the telescope, not from a prior at the keyboard.

3. progenax can do this because it is differentiable end-to-end. Wiring a faithful Moe & Di Stefano binary population into a Fisher design requires $\partial(\text{information})/\partial(\text{observing strategy})$ through the binary kernel. That is the throughline of the whole OED section: the same additive backbone that targets anisotropy, mass, and concentration here defends a measurement against a model error — it extends OED from “what to measure” to “what to measure robustly.”

Current scope and planned extensions¶

How to run¶

# the cheap design + forecast + mechanism figure (no MC; CI-safe, ~1 min)
env -u VIRTUAL_ENV uv run --no-sync python scripts/demo_oed_binary.py

# the full result: cross-model MC (the H1 bias + headline figure) + maximin + sweep
PROGENAX_RUN_OED_BINARY=1 env -u VIRTUAL_ENV uv run --no-sync \
    python scripts/demo_oed_binary.py --run-mc --maximin --sweep \
    --outdir docs/website/60-science-demos/optimal-design/figures

The cheap default path computes the binary-blind design, its forecast $\sigma(M)/M$, and the mechanism figure, and exits 0. The env-gated cross-model Monte Carlo (the H1 bias, the false-confidence figure) is @slow and out of CI — it is the gate test_H1_naive_design_is_biased_beyond_forecast (and the binary-aware test_fix_binary_aware_fit_is_unbiased) in tests/unit/test_demo_oed_binary.py, run with PROGENAX_RUN_OED_BINARY=1. The figures and run-record land alongside this page.

References¶

The shared Fisher / Cramér–Rao / projection theory and its references are on the OED formalism page. The line-of-sight projection into $\sigma_{\rm los}$ is Binney & Mamon (1982); the Osipkov–Merritt anisotropy law is Merritt (1985); the EFF density profile is documented on the EFF profile theory page. The binary population — the $P$–$q$–$e$ engine behind $V_{\rm bin}$ — follows Moe & Di Stefano (2017), with ZAMS flux weights from Tout et al. (1996); the single-epoch binary-inflated dispersion machinery is the binary dynamical-mass demo (B12). The companion OED designs are anisotropy (where the proper motions belong), dynamical mass (how deep to survey), and concentration (where concentration information lives) — this page extends them to model robustness.