homemaker-layout/experiments/results/ioe_1ph_current_objective.tsv at 426203b154dbf967c74029f3e0f16faf4a745c45

Re-validate collapse_insearch's default under the current objective

38.18 confirmed the 1ph default-flip was sound for its own era, but that
measurement predates three changes to the objective it was measured
against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and
38.12's missing-space cascade -- and collapse_insearch runs collapse_global
inside every eval, valued against exactly the factors those touched. A
default carried on a superseded measurement is an assumption, not a result.

Re-ran the 1ph protocol as published on the current codebase:

                        N    OFF    ON     W/L/T     diff    t     p
  published 1ph         20   7.95   7.10   11/6/3    +0.85   2.38  0.028
  current objective     20   7.85   7.15   10/7/3    +0.70   1.82  0.069
  current objective     40   7.60   7.03   21/14/5   +0.57   2.01  0.045
  current objective     60   7.58   7.02   29/19/12  +0.57   2.45  0.017

Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired
t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero;
Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because
fail counts are small integers and normality is not obvious.

Two caveats. The effect is about a third smaller than published (+0.57 vs
+0.85) -- partly regression from a lucky N=20 draw, partly plausible real
erosion, since several fails collapse_global used to clear have been
redefined out of existence or made harder.

More usefully: the published N=20 can no longer detect its own effect. At
exactly that sample size the current answer is p ~= 0.069, a null by the
conventional threshold. Had I stopped at N=20 the honest report would have
been "the 1ph verdict no longer reproduces" and the default would have
looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3
pattern this log warns about, now biting the flagship result itself: any
future re-validation of this default needs N >= 40.

20 annotated in place so a reader of the original claim sees the current
figure. Harness takes a seed range now (APPEND=1 to extend a sweep).

Closes homemaker-py-ioe.

Lint at parity (46); tests 384 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB

2026-08-29 13:57:45 +00:00

2.2 KiB

Raw Blame History

1	current	1	on	5	9
2	current	1	off	8	6
3	current	2	on	5	8
4	current	2	off	8	5
5	current	3	on	9	7
6	current	3	off	8	6
7	current	4	on	9	8
8	current	4	off	10	6
9	current	5	on	9	8
10	current	5	off	8	6
11	current	6	on	8	8
12	current	6	off	11	6
13	current	7	on	8	7
14	current	7	off	9	6
15	current	8	on	8	8
16	current	8	off	7	7
17	current	9	on	6	7
18	current	9	off	9	6
19	current	10	on	4	8
20	current	10	off	7	6
21	current	11	on	5	7
22	current	11	off	5	6
23	current	12	on	7	9
24	current	12	off	6	6
25	current	13	on	6	8
26	current	13	off	8	6
27	current	14	on	8	8
28	current	14	off	7	6
29	current	15	on	6	7
30	current	15	off	8	6
31	current	16	on	8	7
32	current	16	off	8	6
33	current	17	on	9	8
34	current	17	off	7	6
35	current	18	on	7	7
36	current	18	off	8	6
37	current	19	on	9	7
38	current	19	off	8	5
39	current	20	on	7	8
40	current	20	off	7	6
41	current	21	on	5	8
42	current	21	off	5	7
43	current	22	on	8	8
44	current	22	off	9	6
45	current	23	on	8	8
46	current	23	off	9	6
47	current	24	on	7	8
48	current	24	off	8	6
49	current	25	on	7	9
50	current	25	off	9	6
51	current	26	on	8	8
52	current	26	off	10	6
53	current	27	on	9	8
54	current	27	off	10	6
55	current	28	on	5	8
56	current	28	off	8	6
57	current	29	on	6	8
58	current	29	off	8	6
59	current	30	on	5	8
60	current	30	off	7	6
61	current	31	on	10	8
62	current	31	off	9	6
63	current	32	on	10	8
64	current	32	off	7	6
65	current	33	on	3	8
66	current	33	off	7	6
67	current	34	on	6	8
68	current	34	off	8	6
69	current	35	on	8	8
70	current	35	off	5	6
71	current	36	on	8	8
72	current	36	off	7	6
73	current	37	on	5	7
74	current	37	off	4	5
75	current	38	on	5	8
76	current	38	off	5	6
77	current	39	on	8	8
78	current	39	off	7	7
79	current	40	on	7	7
80	current	40	off	5	6
81	current	41	on	10	8
82	current	41	off	7	6
83	current	42	on	8	7
84	current	42	off	8	5
85	current	43	on	7	7
86	current	43	off	8	6
87	current	44	on	7	8
88	current	44	off	7	6
89	current	45	on	6	7
90	current	45	off	5	5
91	current	46	on	8	7
92	current	46	off	7	6
93	current	47	on	6	9
94	current	47	off	9	6
95	current	48	on	8	8
96	current	48	off	10	6
97	current	49	on	8	8
98	current	49	off	10	6
99	current	50	on	7	7
100	current	50	off	7	6
101	current	51	on	7	8
102	current	51	off	7	6
103	current	52	on	7	8
104	current	52	off	9	6
105	current	53	on	10	8
106	current	53	off	8	6
107	current	54	on	5	7
108	current	54	off	9	6
109	current	55	on	7	8
110	current	55	off	9	6
111	current	56	on	7	7
112	current	56	off	7	6
113	current	57	on	2	8
114	current	57	off	5	6
115	current	58	on	7	7
116	current	58	off	7	6
117	current	59	on	8	7
118	current	59	off	7	6
119	current	60	on	5	8
120	current	60	off	5	6

2.2 KiB Raw Blame History

2.2 KiB

Raw Blame History