e9e7060fb4
38.18 confirmed the 1ph default-flip was sound for its own era, but that
measurement predates three changes to the objective it was measured
against -- 39.4's namespace fix, 38.10/38.11's per-space crinkliness, and
38.12's missing-space cascade -- and collapse_insearch runs collapse_global
inside every eval, valued against exactly the factors those touched. A
default carried on a superseded measurement is an assumption, not a result.
Re-ran the 1ph protocol as published on the current codebase:
N OFF ON W/L/T diff t p
published 1ph 20 7.95 7.10 11/6/3 +0.85 2.38 0.028
current objective 20 7.85 7.15 10/7/3 +0.70 1.82 0.069
current objective 40 7.60 7.03 21/14/5 +0.57 2.01 0.045
current objective 60 7.58 7.02 29/19/12 +0.57 2.45 0.017
Verdict: the default STANDS. At N=60, mean diff +0.567 fails/seed, paired
t=2.454 (df=59), p=0.0171 exact, 95% CI [+0.105, +1.029] excluding zero;
Wilcoxon signed-rank cross-check agrees (p=0.0138), which matters because
fail counts are small integers and normality is not obvious.
Two caveats. The effect is about a third smaller than published (+0.57 vs
+0.85) -- partly regression from a lucky N=20 draw, partly plausible real
erosion, since several fails collapse_global used to clear have been
redefined out of existence or made harder.
More usefully: the published N=20 can no longer detect its own effect. At
exactly that sample size the current answer is p ~= 0.069, a null by the
conventional threshold. Had I stopped at N=20 the honest report would have
been "the 1ph verdict no longer reproduces" and the default would have
looked unjustified. It took N=60 to resolve. That is the 8sh/1ph/qi6/lj3
pattern this log warns about, now biting the flagship result itself: any
future re-validation of this default needs N >= 40.
20 annotated in place so a reader of the original claim sees the current
figure. Harness takes a seed range now (APPEND=1 to extend a sweep).
Closes homemaker-py-ioe.
Lint at parity (46); tests 384 passed, 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
27 lines
1.1 KiB
Bash
Executable file
27 lines
1.1 KiB
Bash
Executable file
#!/usr/bin/env bash
|
|
# Re-run the 1ph protocol (DESIGN.md §20): programme-house init.dom, budget 3000,
|
|
# 4 workers, ON vs OFF, BOTH arms finished with --collapse.
|
|
# Usage: rerun_1ph_protocol.sh <worktree> <tag> <out.tsv> [lo] [hi] (APPEND=1 to add)
|
|
set -u
|
|
W="$1"; TAG="$2"; OUT="$3"; LO="${4:-1}"; HI="${5:-20}"
|
|
cd "$W/examples/programme-house"
|
|
[ "${APPEND:-0}" = "1" ] || : > "$OUT"
|
|
for seed in $(seq "$LO" "$HI"); do
|
|
for arm in on off; do
|
|
flag=""; [ "$arm" = "off" ] && flag="--no-collapse-insearch"
|
|
t0=$(date +%s)
|
|
PYTHONPATH="$W/src" timeout 600 python -m homemaker_layout.evolve init.dom \
|
|
--budget 3000 --seed "$seed" --workers 4 --collapse $flag \
|
|
--output "$OUT.$arm.dom" > "$OUT.$arm.log" 2>&1
|
|
t1=$(date +%s)
|
|
n=$(PYTHONPATH="$W/src" python - "$OUT.$arm.dom" <<'PY'
|
|
import copy, sys
|
|
from homemaker_layout import dom, fitness
|
|
conf, cost = fitness.load_config(".")
|
|
_, f = fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(dom.load(sys.argv[1])))
|
|
print(len(f))
|
|
PY
|
|
)
|
|
echo -e "$TAG\t$seed\t$arm\t$n\t$((t1-t0))" >> "$OUT"
|
|
done
|
|
done
|