caf480aa32
0wr asked which harbor A/Bs were decided by a narrow margin before 39.4. Measuring harbor's variance makes the margin-by-margin triage moot. Harbor's paired seed-to-seed sd is 6.19 fails (24 paired ON/OFF runs, budget 2500), so n=3 resolves nothing finer than 15.4 fails. Every recorded harbor margin is below that: 13.9 share_edge_cap 3.7, 20 qpk 8.3, 23 f1d mixed, 37.1 tiering 6.3. So it is not that SOME harbor results were narrow -- no harbor A/B run at three seeds could resolve the margin it reported, independently of what 39.4 did to the programme. Of the 220 possible 3-seed subsets of the 24 runs, 56 (25%) show a clean 3/3 sweep for ON. Re-measured 20's harbor arm, the one backing a live default: N=3 2W/1L/0T +2.67 p=0.560 N=12 8W/3L/1T +3.50 p=0.076 N=24 13W/10L/1T +1.21 p=0.502 CI [-2.46,+4.88] Null. The published "harbor: ON wins 3/3, 80.3 -> 72.0" was a lucky draw -- even seeds 1-3 measured here give 2W/1L, not a sweep. So 20's claim that the qpk verdict "holds at both example scales tested" is withdrawn and annotated in place. collapse_insearch's default rests on programme-house alone (38.19, N=60, +0.57, p=0.017). It is not refuted on harbor -- direction positive but indistinguishable from zero -- but harbor must not be cited as corroboration. Harness generalised (PROG/BUDGET/WORKERS) and results kept. Filed homemaker-py-... : A/B harnesses should report the minimum detectable difference for the N they run, so an underpowered verdict is visible when it is made rather than years later. Closes homemaker-py-0wr. Lint at parity (46). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
29 lines
1.2 KiB
Bash
Executable file
29 lines
1.2 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Re-run the 1ph protocol (DESIGN.md §20): programme-house init.dom, budget 3000,
|
|
# 4 workers, ON vs OFF, BOTH arms finished with --collapse.
|
|
# Usage: rerun_1ph_protocol.sh <worktree> <tag> <out.tsv> [lo] [hi]
|
|
# env: PROG (default programme-house), BUDGET (3000), WORKERS (4), APPEND=1
|
|
set -u
|
|
W="$1"; TAG="$2"; OUT="$3"; LO="${4:-1}"; HI="${5:-20}"
|
|
PROG="${PROG:-programme-house}"; BUDGET="${BUDGET:-3000}"; WORKERS="${WORKERS:-4}"
|
|
cd "$W/examples/$PROG"
|
|
[ "${APPEND:-0}" = "1" ] || : > "$OUT"
|
|
for seed in $(seq "$LO" "$HI"); do
|
|
for arm in on off; do
|
|
flag=""; [ "$arm" = "off" ] && flag="--no-collapse-insearch"
|
|
t0=$(date +%s)
|
|
PYTHONPATH="$W/src" timeout 600 python -m homemaker_layout.evolve init.dom \
|
|
--budget "$BUDGET" --seed "$seed" --workers "$WORKERS" --collapse $flag \
|
|
--output "$OUT.$arm.dom" > "$OUT.$arm.log" 2>&1
|
|
t1=$(date +%s)
|
|
n=$(PYTHONPATH="$W/src" python - "$OUT.$arm.dom" <<'PY'
|
|
import copy, sys
|
|
from homemaker_layout import dom, fitness
|
|
conf, cost = fitness.load_config(".")
|
|
_, f = fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(dom.load(sys.argv[1])))
|
|
print(len(f))
|
|
PY
|
|
)
|
|
echo -e "$TAG\t$seed\t$arm\t$n\t$((t1-t0))" >> "$OUT"
|
|
done
|
|
done
|