0wr asked which harbor A/Bs were decided by a narrow margin before 39.4.
Measuring harbor's variance makes the margin-by-margin triage moot.
Harbor's paired seed-to-seed sd is 6.19 fails (24 paired ON/OFF runs,
budget 2500), so n=3 resolves nothing finer than 15.4 fails. Every recorded
harbor margin is below that: 13.9 share_edge_cap 3.7, 20 qpk 8.3, 23 f1d
mixed, 37.1 tiering 6.3. So it is not that SOME harbor results were narrow
-- no harbor A/B run at three seeds could resolve the margin it reported,
independently of what 39.4 did to the programme. Of the 220 possible
3-seed subsets of the 24 runs, 56 (25%) show a clean 3/3 sweep for ON.
Re-measured 20's harbor arm, the one backing a live default:
N=3 2W/1L/0T +2.67 p=0.560
N=12 8W/3L/1T +3.50 p=0.076
N=24 13W/10L/1T +1.21 p=0.502 CI [-2.46,+4.88]
Null. The published "harbor: ON wins 3/3, 80.3 -> 72.0" was a lucky draw --
even seeds 1-3 measured here give 2W/1L, not a sweep.
So 20's claim that the qpk verdict "holds at both example scales tested" is
withdrawn and annotated in place. collapse_insearch's default rests on
programme-house alone (38.19, N=60, +0.57, p=0.017). It is not refuted on
harbor -- direction positive but indistinguishable from zero -- but harbor
must not be cited as corroboration.
Harness generalised (PROG/BUDGET/WORKERS) and results kept.
Filed homemaker-py-... : A/B harnesses should report the minimum detectable
difference for the N they run, so an underpowered verdict is visible when
it is made rather than years later.
Closes homemaker-py-0wr.
Lint at parity (46).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
