bruno/homemaker-layout
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
homemaker-layout/experiments/results/ioe_1ph_current_objective.tsv

121 lines
2.2 KiB
Text
Raw Normal View History

Re-validate collapse_insearch's default under the current objective 38.18 confirmed the 1ph default-flip was sound for its own era, but that measurement predates three changes to the objective it was measured against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and 38.12's missing-space cascade -- and collapse_insearch runs collapse_global inside every eval, valued against exactly the factors those touched. A default carried on a superseded measurement is an assumption, not a result. Re-ran the 1ph protocol as published on the current codebase: N OFF ON W/L/T diff t p published 1ph 20 7.95 7.10 11/6/3 +0.85 2.38 0.028 current objective 20 7.85 7.15 10/7/3 +0.70 1.82 0.069 current objective 40 7.60 7.03 21/14/5 +0.57 2.01 0.045 current objective 60 7.58 7.02 29/19/12 +0.57 2.45 0.017 Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero; Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because fail counts are small integers and normality is not obvious. Two caveats. The effect is about a third smaller than published (+0.57 vs +0.85) -- partly regression from a lucky N=20 draw, partly plausible real erosion, since several fails collapse_global used to clear have been redefined out of existence or made harder. More usefully: the published N=20 can no longer detect its own effect. At exactly that sample size the current answer is p ~= 0.069, a null by the conventional threshold. Had I stopped at N=20 the honest report would have been "the 1ph verdict no longer reproduces" and the default would have looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3 pattern this log warns about, now biting the flagship result itself: any future re-validation of this default needs N >= 40. 20 annotated in place so a reader of the original claim sees the current figure. Harness takes a seed range now (APPEND=1 to extend a sweep). Closes homemaker-py-ioe. Lint at parity (46); tests 384 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:57:45 +00:00
current 1 on 5 9
current 1 off 8 6
current 2 on 5 8
current 2 off 8 5
current 3 on 9 7
current 3 off 8 6
current 4 on 9 8
current 4 off 10 6
current 5 on 9 8
current 5 off 8 6
current 6 on 8 8
current 6 off 11 6
current 7 on 8 7
current 7 off 9 6
current 8 on 8 8
current 8 off 7 7
current 9 on 6 7
current 9 off 9 6
current 10 on 4 8
current 10 off 7 6
current 11 on 5 7
current 11 off 5 6
current 12 on 7 9
current 12 off 6 6
current 13 on 6 8
current 13 off 8 6
current 14 on 8 8
current 14 off 7 6
current 15 on 6 7
current 15 off 8 6
current 16 on 8 7
current 16 off 8 6
current 17 on 9 8
current 17 off 7 6
current 18 on 7 7
current 18 off 8 6
current 19 on 9 7
current 19 off 8 5
current 20 on 7 8
current 20 off 7 6
current 21 on 5 8
current 21 off 5 7
current 22 on 8 8
current 22 off 9 6
current 23 on 8 8
current 23 off 9 6
current 24 on 7 8
current 24 off 8 6
current 25 on 7 9
current 25 off 9 6
current 26 on 8 8
current 26 off 10 6
current 27 on 9 8
current 27 off 10 6
current 28 on 5 8
current 28 off 8 6
current 29 on 6 8
current 29 off 8 6
current 30 on 5 8
current 30 off 7 6
current 31 on 10 8
current 31 off 9 6
current 32 on 10 8
current 32 off 7 6
current 33 on 3 8
current 33 off 7 6
current 34 on 6 8
current 34 off 8 6
current 35 on 8 8
current 35 off 5 6
current 36 on 8 8
current 36 off 7 6
current 37 on 5 7
current 37 off 4 5
current 38 on 5 8
current 38 off 5 6
current 39 on 8 8
current 39 off 7 7
current 40 on 7 7
current 40 off 5 6
current 41 on 10 8
current 41 off 7 6
current 42 on 8 7
current 42 off 8 5
current 43 on 7 7
current 43 off 8 6
current 44 on 7 8
current 44 off 7 6
current 45 on 6 7
current 45 off 5 5
current 46 on 8 7
current 46 off 7 6
current 47 on 6 9
current 47 off 9 6
current 48 on 8 8
current 48 off 10 6
current 49 on 8 8
current 49 off 10 6
current 50 on 7 7
current 50 off 7 6
current 51 on 7 8
current 51 off 7 6
current 52 on 7 8
current 52 off 9 6
current 53 on 10 8
current 53 off 8 6
current 54 on 5 7
current 54 off 9 6
current 55 on 7 8
current 55 off 9 6
current 56 on 7 7
current 56 off 7 6
current 57 on 2 8
current 57 off 5 6
current 58 on 7 7
current 58 off 7 6
current 59 on 8 7
current 59 off 7 6
current 60 on 5 8
current 60 off 5 6
Reference in a new issue Copy permalink