Three times in this log a verdict rested on a sample that could not have
produced it: 38.19 (programme-house claimed at N=20, resolves at N=60),
38.21 (harbor at n=3 resolves nothing finer than ~15 fails, yet every
recorded margin is smaller), 39.5/38.20 (a 10-fail cpsat margin inside a
+-23-fail noise band). Each was found years later.
experiments/ab_report.py makes it visible when the verdict is made:
minimum detectable difference = t_crit(0.975, N-1) * sd / sqrt(N)
A margin below the MDD is not a weak result but an absent one -- the
experiment could not have distinguished it from zero however it came out.
The report flags that, refuses to endorse a winner, and states the N needed.
Validated against both datasets measured this session, reproducing the
hand-computed figures exactly:
programme-house N=60 +0.567 p=0.017 MDD 0.462 verdict supported
programme-house N=20 +0.700 p=0.085 MDD 0.805 UNDERPOWERED, N~=26
harbor N=24 +1.208 p=0.502 MDD 3.669 UNDERPOWERED, N~=202
Harbor needing ~200 seeds means it cannot answer the collapse_insearch
question at any N this project would realistically run.
Fixed a defect in my own first version: with all-ties (sd=0) the MDD
collapses to zero and the naive abs(mean) < mdd reported "margin exceeds
the MDD -- verdict supported" for a margin of 0.000, with t=nan. A reporter
that endorses a zero margin is worse than none. Degenerate cases are now
explicit and distinguish all-ties from a constant non-zero difference.
Separate correction found while validating: 38.19's published p-values for
N=20 and N=40 were 0.069 and 0.045, from a normal approximation. The exact
paired t-test gives 0.085 and 0.052 -- so N=40 did NOT reach significance
either; it took N=60. The approximation was anti-conservative, making
results look more significant than they are, the same direction of error
this thread is about. Corrected at all four citing sites, and the "needs
N >= 40" guidance raised to N >= 60.
Closes homemaker-py-tco.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
