homemaker-layout/experiments/rerun_1ph_protocol.sh

27 lines
945 B
Bash
Raw Normal View History

Re-verify 1ph: the iio bug could never have touched it d86 asked for the rigorous version of 35's spot-check -- check out the codebase at the 1ph commit, backport the iio stale-leaf-share fix, re-run the historical seed sets. One constraint had to be worked around: this repository's history begins 2026-07-30, six days AFTER the 1ph commit (2026-07-24), so that checkout does not exist here. Closest reachable stand-in is 391f510 (2026-07-30), a genuine ancestor of the iio fix and therefore pre-iio, carrying that era's examples/ and objective. Measured there, not at the true 1ph commit, and that is a real limitation of the reproduction. Protocol as published: programme-house, budget 3000, 4 workers, seeds 1-20, ON vs OFF, both arms finished with --collapse. Run twice over one worktree, as-is then with the 22-line iio hunk applied. OFF ON W/L/T mean diff t(df=19) published 1ph 7.95 7.10 11/6/3 +0.85 2.38 pre-iio 391f510 8.05 7.10 11/6/3 +0.95 2.59 same + iio fix 8.05 7.10 11/6/3 +0.95 2.59 The published verdict reproduces, and the iio fix changes nothing: 0 of 40 (seed, arm) cells differ, per-seed counts equal cell by cell. It could not have been otherwise. The bug needs a leaf carrying a STALE share/share_type, and leaf-sharing only stamps a share where a code has count > 1 -- programme-house declares count: 1 for all six codes. Over 8 constructed seeds at that commit, programme-house had share_type set on 0 of 56 leaves; harbor-house on 24 of 128. _collapse_value reads leaf.share_type, so the bug is structurally unreachable on the 1ph protocol. That is why 35 saw 2/3 harbor seeds diverge by 5-8 fails while programme-house at N=20 moves not one cell: harbor has codes at counts 10, 6 and 5. 20's retroactive caveat is discharged for the 1ph section and stays live for harbor-house/qpk, where shares exist and divergence was measured. Kept experiments/rerun_1ph_protocol.sh and both result TSVs. Closes homemaker-py-d86. Lint at parity (46); tests 384 passed, 0 failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
2026-08-29 13:21:29 +00:00
#!/usr/bin/env bash
# Re-run the 1ph protocol (DESIGN.md §20): programme-house init.dom, budget 3000,
# 4 workers, seeds 1-20, ON vs OFF, BOTH arms finished with --collapse.
set -u
W="$1"; TAG="$2"; OUT="$3"
cd "$W/examples/programme-house"
: > "$OUT"
for seed in $(seq 1 20); do
for arm in on off; do
flag=""; [ "$arm" = "off" ] && flag="--no-collapse-insearch"
t0=$(date +%s)
PYTHONPATH="$W/src" timeout 600 python -m homemaker_layout.evolve init.dom \
--budget 3000 --seed "$seed" --workers 4 --collapse $flag \
--output "$OUT.$arm.dom" > "$OUT.$arm.log" 2>&1
t1=$(date +%s)
n=$(PYTHONPATH="$W/src" python - "$OUT.$arm.dom" <<'PY'
import copy, sys
from homemaker_layout import dom, fitness
conf, cost = fitness.load_config(".")
_, f = fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(dom.load(sys.argv[1])))
print(len(f))
PY
)
echo -e "$TAG\t$seed\t$arm\t$n\t$((t1-t0))" >> "$OUT"
done
done