homemaker-layout/experiments/results
Claude e9e7060fb4
Re-validate collapse_insearch's default under the current objective
38.18 confirmed the 1ph default-flip was sound for its own era, but that
measurement predates three changes to the objective it was measured
against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and
38.12's missing-space cascade -- and collapse_insearch runs collapse_global
inside every eval, valued against exactly the factors those touched. A
default carried on a superseded measurement is an assumption, not a result.

Re-ran the 1ph protocol as published on the current codebase:

                        N    OFF    ON     W/L/T     diff    t     p
  published 1ph         20   7.95   7.10   11/6/3    +0.85   2.38  0.028
  current objective     20   7.85   7.15   10/7/3    +0.70   1.82  0.069
  current objective     40   7.60   7.03   21/14/5   +0.57   2.01  0.045
  current objective     60   7.58   7.02   29/19/12  +0.57   2.45  0.017

Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired
t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero;
Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because
fail counts are small integers and normality is not obvious.

Two caveats. The effect is about a third smaller than published (+0.57 vs
+0.85) -- partly regression from a lucky N=20 draw, partly plausible real
erosion, since several fails collapse_global used to clear have been
redefined out of existence or made harder.

More usefully: the published N=20 can no longer detect its own effect. At
exactly that sample size the current answer is p ~= 0.069, a null by the
conventional threshold. Had I stopped at N=20 the honest report would have
been "the 1ph verdict no longer reproduces" and the default would have
looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3
pattern this log warns about, now biting the flagship result itself: any
future re-validation of this default needs N >= 40.

20 annotated in place so a reader of the original claim sees the current
figure. Harness takes a seed range now (APPEND=1 to extend a sweep).

Closes homemaker-py-ioe.

Lint at parity (46); tests 384 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:57:45 +00:00
..
ab_cpsat_assign_20k_harbor_maple.log homemaker-py-2g7.5: full acceptance-criteria A/B (harbor+maple, 3 seeds, 20k budget) 2026-08-05 00:01:00 +01:00
ab_ssz_power.csv ssz: 61% of reported crinkliness fails are not defects; two corrections 2026-08-26 18:07:01 +00:00
ab_ssz_search.csv ssz: record the A/B result honestly -- not a pass at n=3 2026-08-26 17:57:11 +00:00
coldstart_baseline.tsv Checkpoint long searches; the cold-start runs were lost to a reclaimed box 2026-08-29 05:43:58 +00:00
d86_1ph_iiofix.tsv Re-verify 1ph: the iio bug could never have touched it 2026-08-29 13:21:29 +00:00
d86_1ph_preiio.tsv Re-verify 1ph: the iio bug could never have touched it 2026-08-29 13:21:29 +00:00
ioe_1ph_current_objective.tsv Re-validate collapse_insearch's default under the current objective 2026-08-29 13:57:45 +00:00