Re-verify 1ph: the iio bug could never have touched it
d86 asked for the rigorous version of 35's spot-check -- check out the
codebase at the 1ph commit, backport the iio stale-leaf-share fix, re-run
the historical seed sets.
One constraint had to be worked around: this repository's history begins
2026-07-30, six days AFTER the 1ph commit (2026-07-24), so that checkout
does not exist here. Closest reachable stand-in is 391f510 (2026-07-30), a
genuine ancestor of the iio fix and therefore pre-iio, carrying that era's
examples/ and objective. Measured there, not at the true 1ph commit, and
that is a real limitation of the reproduction.
Protocol as published: programme-house, budget 3000, 4 workers, seeds 1-20,
ON vs OFF, both arms finished with --collapse. Run twice over one worktree,
as-is then with the 22-line iio hunk applied.
OFF ON W/L/T mean diff t(df=19)
published 1ph 7.95 7.10 11/6/3 +0.85 2.38
pre-iio 391f510 8.05 7.10 11/6/3 +0.95 2.59
same + iio fix 8.05 7.10 11/6/3 +0.95 2.59
The published verdict reproduces, and the iio fix changes nothing: 0 of 40
(seed, arm) cells differ, per-seed counts equal cell by cell.
It could not have been otherwise. The bug needs a leaf carrying a STALE
share/share_type, and leaf-sharing only stamps a share where a code has
count > 1 -- programme-house declares count: 1 for all six codes. Over 8
constructed seeds at that commit, programme-house had share_type set on 0
of 56 leaves; harbor-house on 24 of 128. _collapse_value reads
leaf.share_type, so the bug is structurally unreachable on the 1ph
protocol. That is why 35 saw 2/3 harbor seeds diverge by 5-8 fails while
programme-house at N=20 moves not one cell: harbor has codes at counts 10,
6 and 5.
20's retroactive caveat is discharged for the 1ph section and stays live
for harbor-house/qpk, where shares exist and divergence was measured.
Kept experiments/rerun_1ph_protocol.sh and both result TSVs.
Closes homemaker-py-d86.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:21:29 +00:00
|
|
|
#!/usr/bin/env bash
|
|
|
|
|
# Re-run the 1ph protocol (DESIGN.md §20): programme-house init.dom, budget 3000,
|
Re-validate collapse_insearch's default under the current objective
38.18 confirmed the 1ph default-flip was sound for its own era, but that
measurement predates three changes to the objective it was measured
against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and
38.12's missing-space cascade -- and collapse_insearch runs collapse_global
inside every eval, valued against exactly the factors those touched. A
default carried on a superseded measurement is an assumption, not a result.
Re-ran the 1ph protocol as published on the current codebase:
N OFF ON W/L/T diff t p
published 1ph 20 7.95 7.10 11/6/3 +0.85 2.38 0.028
current objective 20 7.85 7.15 10/7/3 +0.70 1.82 0.069
current objective 40 7.60 7.03 21/14/5 +0.57 2.01 0.045
current objective 60 7.58 7.02 29/19/12 +0.57 2.45 0.017
Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired
t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero;
Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because
fail counts are small integers and normality is not obvious.
Two caveats. The effect is about a third smaller than published (+0.57 vs
+0.85) -- partly regression from a lucky N=20 draw, partly plausible real
erosion, since several fails collapse_global used to clear have been
redefined out of existence or made harder.
More usefully: the published N=20 can no longer detect its own effect. At
exactly that sample size the current answer is p ~= 0.069, a null by the
conventional threshold. Had I stopped at N=20 the honest report would have
been "the 1ph verdict no longer reproduces" and the default would have
looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3
pattern this log warns about, now biting the flagship result itself: any
future re-validation of this default needs N >= 40.
20 annotated in place so a reader of the original claim sees the current
figure. Harness takes a seed range now (APPEND=1 to extend a sweep).
Closes homemaker-py-ioe.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:57:45 +00:00
|
|
|
# 4 workers, ON vs OFF, BOTH arms finished with --collapse.
|
Harbor A/Bs at n=3 could never resolve their own margins
0wr asked which harbor A/Bs were decided by a narrow margin before 39.4.
Measuring harbor's variance makes the margin-by-margin triage moot.
Harbor's paired seed-to-seed sd is 6.19 fails (24 paired ON/OFF runs,
budget 2500), so n=3 resolves nothing finer than 15.4 fails. Every recorded
harbor margin is below that: 13.9 share_edge_cap 3.7, 20 qpk 8.3, 23 f1d
mixed, 37.1 tiering 6.3. So it is not that SOME harbor results were narrow
-- no harbor A/B run at three seeds could resolve the margin it reported,
independently of what 39.4 did to the programme. Of the 220 possible
3-seed subsets of the 24 runs, 56 (25%) show a clean 3/3 sweep for ON.
Re-measured 20's harbor arm, the one backing a live default:
N=3 2W/1L/0T +2.67 p=0.560
N=12 8W/3L/1T +3.50 p=0.076
N=24 13W/10L/1T +1.21 p=0.502 CI [-2.46,+4.88]
Null. The published "harbor: ON wins 3/3, 80.3 -> 72.0" was a lucky draw --
even seeds 1-3 measured here give 2W/1L, not a sweep.
So 20's claim that the qpk verdict "holds at both example scales tested" is
withdrawn and annotated in place. collapse_insearch's default rests on
programme-house alone (38.19, N=60, +0.57, p=0.017). It is not refuted on
harbor -- direction positive but indistinguishable from zero -- but harbor
must not be cited as corroboration.
Harness generalised (PROG/BUDGET/WORKERS) and results kept.
Filed homemaker-py-... : A/B harnesses should report the minimum detectable
difference for the N they run, so an underpowered verdict is visible when
it is made rather than years later.
Closes homemaker-py-0wr.
Lint at parity (46).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 19:34:30 +00:00
|
|
|
# Usage: rerun_1ph_protocol.sh <worktree> <tag> <out.tsv> [lo] [hi]
|
|
|
|
|
# env: PROG (default programme-house), BUDGET (3000), WORKERS (4), APPEND=1
|
Re-verify 1ph: the iio bug could never have touched it
d86 asked for the rigorous version of 35's spot-check -- check out the
codebase at the 1ph commit, backport the iio stale-leaf-share fix, re-run
the historical seed sets.
One constraint had to be worked around: this repository's history begins
2026-07-30, six days AFTER the 1ph commit (2026-07-24), so that checkout
does not exist here. Closest reachable stand-in is 391f510 (2026-07-30), a
genuine ancestor of the iio fix and therefore pre-iio, carrying that era's
examples/ and objective. Measured there, not at the true 1ph commit, and
that is a real limitation of the reproduction.
Protocol as published: programme-house, budget 3000, 4 workers, seeds 1-20,
ON vs OFF, both arms finished with --collapse. Run twice over one worktree,
as-is then with the 22-line iio hunk applied.
OFF ON W/L/T mean diff t(df=19)
published 1ph 7.95 7.10 11/6/3 +0.85 2.38
pre-iio 391f510 8.05 7.10 11/6/3 +0.95 2.59
same + iio fix 8.05 7.10 11/6/3 +0.95 2.59
The published verdict reproduces, and the iio fix changes nothing: 0 of 40
(seed, arm) cells differ, per-seed counts equal cell by cell.
It could not have been otherwise. The bug needs a leaf carrying a STALE
share/share_type, and leaf-sharing only stamps a share where a code has
count > 1 -- programme-house declares count: 1 for all six codes. Over 8
constructed seeds at that commit, programme-house had share_type set on 0
of 56 leaves; harbor-house on 24 of 128. _collapse_value reads
leaf.share_type, so the bug is structurally unreachable on the 1ph
protocol. That is why 35 saw 2/3 harbor seeds diverge by 5-8 fails while
programme-house at N=20 moves not one cell: harbor has codes at counts 10,
6 and 5.
20's retroactive caveat is discharged for the 1ph section and stays live
for harbor-house/qpk, where shares exist and divergence was measured.
Kept experiments/rerun_1ph_protocol.sh and both result TSVs.
Closes homemaker-py-d86.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:21:29 +00:00
|
|
|
set -u
|
Re-validate collapse_insearch's default under the current objective
38.18 confirmed the 1ph default-flip was sound for its own era, but that
measurement predates three changes to the objective it was measured
against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and
38.12's missing-space cascade -- and collapse_insearch runs collapse_global
inside every eval, valued against exactly the factors those touched. A
default carried on a superseded measurement is an assumption, not a result.
Re-ran the 1ph protocol as published on the current codebase:
N OFF ON W/L/T diff t p
published 1ph 20 7.95 7.10 11/6/3 +0.85 2.38 0.028
current objective 20 7.85 7.15 10/7/3 +0.70 1.82 0.069
current objective 40 7.60 7.03 21/14/5 +0.57 2.01 0.045
current objective 60 7.58 7.02 29/19/12 +0.57 2.45 0.017
Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired
t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero;
Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because
fail counts are small integers and normality is not obvious.
Two caveats. The effect is about a third smaller than published (+0.57 vs
+0.85) -- partly regression from a lucky N=20 draw, partly plausible real
erosion, since several fails collapse_global used to clear have been
redefined out of existence or made harder.
More usefully: the published N=20 can no longer detect its own effect. At
exactly that sample size the current answer is p ~= 0.069, a null by the
conventional threshold. Had I stopped at N=20 the honest report would have
been "the 1ph verdict no longer reproduces" and the default would have
looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3
pattern this log warns about, now biting the flagship result itself: any
future re-validation of this default needs N >= 40.
20 annotated in place so a reader of the original claim sees the current
figure. Harness takes a seed range now (APPEND=1 to extend a sweep).
Closes homemaker-py-ioe.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:57:45 +00:00
|
|
|
W="$1"; TAG="$2"; OUT="$3"; LO="${4:-1}"; HI="${5:-20}"
|
Harbor A/Bs at n=3 could never resolve their own margins
0wr asked which harbor A/Bs were decided by a narrow margin before 39.4.
Measuring harbor's variance makes the margin-by-margin triage moot.
Harbor's paired seed-to-seed sd is 6.19 fails (24 paired ON/OFF runs,
budget 2500), so n=3 resolves nothing finer than 15.4 fails. Every recorded
harbor margin is below that: 13.9 share_edge_cap 3.7, 20 qpk 8.3, 23 f1d
mixed, 37.1 tiering 6.3. So it is not that SOME harbor results were narrow
-- no harbor A/B run at three seeds could resolve the margin it reported,
independently of what 39.4 did to the programme. Of the 220 possible
3-seed subsets of the 24 runs, 56 (25%) show a clean 3/3 sweep for ON.
Re-measured 20's harbor arm, the one backing a live default:
N=3 2W/1L/0T +2.67 p=0.560
N=12 8W/3L/1T +3.50 p=0.076
N=24 13W/10L/1T +1.21 p=0.502 CI [-2.46,+4.88]
Null. The published "harbor: ON wins 3/3, 80.3 -> 72.0" was a lucky draw --
even seeds 1-3 measured here give 2W/1L, not a sweep.
So 20's claim that the qpk verdict "holds at both example scales tested" is
withdrawn and annotated in place. collapse_insearch's default rests on
programme-house alone (38.19, N=60, +0.57, p=0.017). It is not refuted on
harbor -- direction positive but indistinguishable from zero -- but harbor
must not be cited as corroboration.
Harness generalised (PROG/BUDGET/WORKERS) and results kept.
Filed homemaker-py-... : A/B harnesses should report the minimum detectable
difference for the N they run, so an underpowered verdict is visible when
it is made rather than years later.
Closes homemaker-py-0wr.
Lint at parity (46).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 19:34:30 +00:00
|
|
|
PROG="${PROG:-programme-house}"; BUDGET="${BUDGET:-3000}"; WORKERS="${WORKERS:-4}"
|
|
|
|
|
cd "$W/examples/$PROG"
|
Re-validate collapse_insearch's default under the current objective
38.18 confirmed the 1ph default-flip was sound for its own era, but that
measurement predates three changes to the objective it was measured
against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and
38.12's missing-space cascade -- and collapse_insearch runs collapse_global
inside every eval, valued against exactly the factors those touched. A
default carried on a superseded measurement is an assumption, not a result.
Re-ran the 1ph protocol as published on the current codebase:
N OFF ON W/L/T diff t p
published 1ph 20 7.95 7.10 11/6/3 +0.85 2.38 0.028
current objective 20 7.85 7.15 10/7/3 +0.70 1.82 0.069
current objective 40 7.60 7.03 21/14/5 +0.57 2.01 0.045
current objective 60 7.58 7.02 29/19/12 +0.57 2.45 0.017
Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired
t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero;
Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because
fail counts are small integers and normality is not obvious.
Two caveats. The effect is about a third smaller than published (+0.57 vs
+0.85) -- partly regression from a lucky N=20 draw, partly plausible real
erosion, since several fails collapse_global used to clear have been
redefined out of existence or made harder.
More usefully: the published N=20 can no longer detect its own effect. At
exactly that sample size the current answer is p ~= 0.069, a null by the
conventional threshold. Had I stopped at N=20 the honest report would have
been "the 1ph verdict no longer reproduces" and the default would have
looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3
pattern this log warns about, now biting the flagship result itself: any
future re-validation of this default needs N >= 40.
20 annotated in place so a reader of the original claim sees the current
figure. Harness takes a seed range now (APPEND=1 to extend a sweep).
Closes homemaker-py-ioe.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:57:45 +00:00
|
|
|
[ "${APPEND:-0}" = "1" ] || : > "$OUT"
|
|
|
|
|
for seed in $(seq "$LO" "$HI"); do
|
Re-verify 1ph: the iio bug could never have touched it
d86 asked for the rigorous version of 35's spot-check -- check out the
codebase at the 1ph commit, backport the iio stale-leaf-share fix, re-run
the historical seed sets.
One constraint had to be worked around: this repository's history begins
2026-07-30, six days AFTER the 1ph commit (2026-07-24), so that checkout
does not exist here. Closest reachable stand-in is 391f510 (2026-07-30), a
genuine ancestor of the iio fix and therefore pre-iio, carrying that era's
examples/ and objective. Measured there, not at the true 1ph commit, and
that is a real limitation of the reproduction.
Protocol as published: programme-house, budget 3000, 4 workers, seeds 1-20,
ON vs OFF, both arms finished with --collapse. Run twice over one worktree,
as-is then with the 22-line iio hunk applied.
OFF ON W/L/T mean diff t(df=19)
published 1ph 7.95 7.10 11/6/3 +0.85 2.38
pre-iio 391f510 8.05 7.10 11/6/3 +0.95 2.59
same + iio fix 8.05 7.10 11/6/3 +0.95 2.59
The published verdict reproduces, and the iio fix changes nothing: 0 of 40
(seed, arm) cells differ, per-seed counts equal cell by cell.
It could not have been otherwise. The bug needs a leaf carrying a STALE
share/share_type, and leaf-sharing only stamps a share where a code has
count > 1 -- programme-house declares count: 1 for all six codes. Over 8
constructed seeds at that commit, programme-house had share_type set on 0
of 56 leaves; harbor-house on 24 of 128. _collapse_value reads
leaf.share_type, so the bug is structurally unreachable on the 1ph
protocol. That is why 35 saw 2/3 harbor seeds diverge by 5-8 fails while
programme-house at N=20 moves not one cell: harbor has codes at counts 10,
6 and 5.
20's retroactive caveat is discharged for the 1ph section and stays live
for harbor-house/qpk, where shares exist and divergence was measured.
Kept experiments/rerun_1ph_protocol.sh and both result TSVs.
Closes homemaker-py-d86.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:21:29 +00:00
|
|
|
for arm in on off; do
|
|
|
|
|
flag=""; [ "$arm" = "off" ] && flag="--no-collapse-insearch"
|
|
|
|
|
t0=$(date +%s)
|
|
|
|
|
PYTHONPATH="$W/src" timeout 600 python -m homemaker_layout.evolve init.dom \
|
Harbor A/Bs at n=3 could never resolve their own margins
0wr asked which harbor A/Bs were decided by a narrow margin before 39.4.
Measuring harbor's variance makes the margin-by-margin triage moot.
Harbor's paired seed-to-seed sd is 6.19 fails (24 paired ON/OFF runs,
budget 2500), so n=3 resolves nothing finer than 15.4 fails. Every recorded
harbor margin is below that: 13.9 share_edge_cap 3.7, 20 qpk 8.3, 23 f1d
mixed, 37.1 tiering 6.3. So it is not that SOME harbor results were narrow
-- no harbor A/B run at three seeds could resolve the margin it reported,
independently of what 39.4 did to the programme. Of the 220 possible
3-seed subsets of the 24 runs, 56 (25%) show a clean 3/3 sweep for ON.
Re-measured 20's harbor arm, the one backing a live default:
N=3 2W/1L/0T +2.67 p=0.560
N=12 8W/3L/1T +3.50 p=0.076
N=24 13W/10L/1T +1.21 p=0.502 CI [-2.46,+4.88]
Null. The published "harbor: ON wins 3/3, 80.3 -> 72.0" was a lucky draw --
even seeds 1-3 measured here give 2W/1L, not a sweep.
So 20's claim that the qpk verdict "holds at both example scales tested" is
withdrawn and annotated in place. collapse_insearch's default rests on
programme-house alone (38.19, N=60, +0.57, p=0.017). It is not refuted on
harbor -- direction positive but indistinguishable from zero -- but harbor
must not be cited as corroboration.
Harness generalised (PROG/BUDGET/WORKERS) and results kept.
Filed homemaker-py-... : A/B harnesses should report the minimum detectable
difference for the N they run, so an underpowered verdict is visible when
it is made rather than years later.
Closes homemaker-py-0wr.
Lint at parity (46).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 19:34:30 +00:00
|
|
|
--budget "$BUDGET" --seed "$seed" --workers "$WORKERS" --collapse $flag \
|
Re-verify 1ph: the iio bug could never have touched it
d86 asked for the rigorous version of 35's spot-check -- check out the
codebase at the 1ph commit, backport the iio stale-leaf-share fix, re-run
the historical seed sets.
One constraint had to be worked around: this repository's history begins
2026-07-30, six days AFTER the 1ph commit (2026-07-24), so that checkout
does not exist here. Closest reachable stand-in is 391f510 (2026-07-30), a
genuine ancestor of the iio fix and therefore pre-iio, carrying that era's
examples/ and objective. Measured there, not at the true 1ph commit, and
that is a real limitation of the reproduction.
Protocol as published: programme-house, budget 3000, 4 workers, seeds 1-20,
ON vs OFF, both arms finished with --collapse. Run twice over one worktree,
as-is then with the 22-line iio hunk applied.
OFF ON W/L/T mean diff t(df=19)
published 1ph 7.95 7.10 11/6/3 +0.85 2.38
pre-iio 391f510 8.05 7.10 11/6/3 +0.95 2.59
same + iio fix 8.05 7.10 11/6/3 +0.95 2.59
The published verdict reproduces, and the iio fix changes nothing: 0 of 40
(seed, arm) cells differ, per-seed counts equal cell by cell.
It could not have been otherwise. The bug needs a leaf carrying a STALE
share/share_type, and leaf-sharing only stamps a share where a code has
count > 1 -- programme-house declares count: 1 for all six codes. Over 8
constructed seeds at that commit, programme-house had share_type set on 0
of 56 leaves; harbor-house on 24 of 128. _collapse_value reads
leaf.share_type, so the bug is structurally unreachable on the 1ph
protocol. That is why 35 saw 2/3 harbor seeds diverge by 5-8 fails while
programme-house at N=20 moves not one cell: harbor has codes at counts 10,
6 and 5.
20's retroactive caveat is discharged for the 1ph section and stays live
for harbor-house/qpk, where shares exist and divergence was measured.
Kept experiments/rerun_1ph_protocol.sh and both result TSVs.
Closes homemaker-py-d86.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:21:29 +00:00
|
|
|
--output "$OUT.$arm.dom" > "$OUT.$arm.log" 2>&1
|
|
|
|
|
t1=$(date +%s)
|
|
|
|
|
n=$(PYTHONPATH="$W/src" python - "$OUT.$arm.dom" <<'PY'
|
|
|
|
|
import copy, sys
|
|
|
|
|
from homemaker_layout import dom, fitness
|
|
|
|
|
conf, cost = fitness.load_config(".")
|
|
|
|
|
_, f = fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(dom.load(sys.argv[1])))
|
|
|
|
|
print(len(f))
|
|
|
|
|
PY
|
|
|
|
|
)
|
|
|
|
|
echo -e "$TAG\t$seed\t$arm\t$n\t$((t1-t0))" >> "$OUT"
|
|
|
|
|
done
|
|
|
|
|
done
|