bruno/homemaker-layout
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

A/Bs now report what their sample could resolve

Browse source 
Three times in this log a verdict rested on a sample that could not have
produced it: 38.19 (programme-house claimed at N=20, resolves at N=60),
38.21 (harbor at n=3 resolves nothing finer than ~15 fails, yet every
recorded margin is smaller), 39.5/38.20 (a 10-fail cpsat margin inside a
+-23-fail noise band). Each was found years later.

experiments/ab_report.py makes it visible when the verdict is made:

  minimum detectable difference = t_crit(0.975, N-1) * sd / sqrt(N)

A margin below the MDD is not a weak result but an absent one -- the
experiment could not have distinguished it from zero however it came out.
The report flags that, refuses to endorse a winner, and states the N needed.
Validated against both datasets measured this session, reproducing the
hand-computed figures exactly:

  programme-house N=60  +0.567 p=0.017  MDD 0.462  verdict supported
  programme-house N=20  +0.700 p=0.085  MDD 0.805  UNDERPOWERED, N~=26
  harbor N=24           +1.208 p=0.502  MDD 3.669  UNDERPOWERED, N~=202

Harbor needing ~200 seeds means it cannot answer the collapse_insearch
question at any N this project would realistically run.

Fixed a defect in my own first version: with all-ties (sd=0) the MDD
collapses to zero and the naive abs(mean) < mdd reported "margin exceeds
the MDD -- verdict supported" for a margin of 0.000, with t=nan. A reporter
that endorses a zero margin is worse than none. Degenerate cases are now
explicit and distinguish all-ties from a constant non-zero difference.

Separate correction found while validating: 38.19's published p-values for
N=20 and N=40 were 0.069 and 0.045, from a normal approximation. The exact
paired t-test gives 0.085 and 0.052 -- so N=40 did NOT reach significance
either; it took N=60. The approximation was anti-conservative, making
results look more significant than they are, the same direction of error
this thread is about. Corrected at all four citing sites, and the "needs
N >= 40" guidance raised to N >= 60.

Closes homemaker-py-tco.

Lint at parity (46); tests 384 passed, 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
This commit is contained in:
Claude 2026-08-29 19:51:47 +00:00
parent c747e55abe
commit 104ba12237
No known key found for this signature in database
4 changed files with 242 additions and 6 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.beads/issues.jsonl
View file

@ -36,7 +36,7 @@
{"_type":"issue","id":"homemaker-py-1p0","title":"Geometry inner loop: full-objective equal-offset ratio optimiser","description":"DESIGN.md §5.1, §7 Phase 1. Productionise experiments/optimize_fullfitness.py into homemaker: optimise(topology, x0=None) -\u003e (geometry, fitness). DOF = equal-offset division ratios of free branches (solver.free_branches, lowest-storey cut ownership), clipped to [eps, 1-eps]. Objective = full oracle fitness (never a proxy — §4.2 falsified). Must support warm-start x0 (§5.6) and a population/batch evaluation mode so each iteration scores via one batched oracle call (§4.6).","acceptance_criteria":"Reproduces or exceeds §4.5 gains (x1.24–x1.67, no new failures) on 2f45907, candidate-002, c964435; works as a library call on any corpus .dom","status":"closed","priority":1,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:58Z","created_by":"Bruno Postle","updated_at":"2026-06-12T08:46:31Z","started_at":"2026-06-12T00:14:19Z","closed_at":"2026-06-12T08:46:31Z","close_reason":"innerloop.optimise() lands: batched CMA-ES sigma ladder (0.05/0.15, IPOP popsize doubling, deterministic seeding) over equal-offset free-branch ratios vs full oracle fitness; warm-start x0 supported. Acceptance vs unprojected originals: x1.65/x1.66/x1.58 against bars x1.24/x1.67/x1.59, no new failures, 46 oracle calls vs NM's 200. Two near-bar results accepted as reproduced-within-noise (1% tol) — draw spread brackets the single-NM-draw bars; approved by Bruno 2026-06-12. Gotchas: equal-offset projection of legacy unequal cuts loses fitness/adds failures (midpoint projection used); pycma seed=0 means clock-seeded.","dependencies":[{"issue_id":"homemaker-py-1p0","depends_on_id":"homemaker-py-av5","type":"blocks","created_at":"2026-06-12T00:39:33Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0}
{"_type":"issue","id":"homemaker-py-8cs","title":"Experiment: warm-vs-cold start of inner loop (Lamarckian inheritance)","description":"DESIGN.md §5.6, §4.6. Warm-starting a child topology's inner loop from the parent's optimised ratios is the main lever for cutting per-topology cost (~3 min/topology cold). Apply single topology mutations to optimised corpus designs, re-optimise warm (surviving cuts keep values, new cuts get heuristic defaults) vs cold, compare oracle-call counts to convergence at equal final fitness.","acceptance_criteria":"Speedup factor measured across \u003e=10 mutated topologies; decision recorded (expect order-of-magnitude; if \u003c2x, revisit §4.6 Phase-2 scoping)","notes":"Experiment script committed (experiments/warm_vs_cold.py, 1cc86c8) and machinery validated oracle-free; one mutated child scored through the oracle OK. Waiting on homemaker-py-gp2 reference run to finish, then execute under URB_NO_OCCLUSION=1 (3 parents x 400 evals + 12 children x 2 x 200 evals, ~1.5-2 h oracle time). Default budgets: parent 400, child 200; target = evals to 95% of best final.","status":"closed","priority":1,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:58Z","created_by":"Bruno Postle","updated_at":"2026-06-12T11:44:45Z","closed_at":"2026-06-12T11:44:45Z","close_reason":"Measured (URB_NO_OCCLUSION=1, parent budget 400, child 200, 12 single mutations across 3 designs): cold start reached 95% of warm final in 0/12 cases within budget — speedup unbounded at practical budgets; warm finals beat cold finals x1.2-x4 in 12/12; 6/12 warm starts were within 95% at 1 eval (near-neutral mutations). Decision: Lamarckian warm-starting is MANDATORY in the memetic driver (homemaker-py-b39), not an optimisation; cold starts produce strictly worse geometry at equal budget. Note: 2 undivides were exactly fitness-neutral (same-type merge == Merge_Divided equivalence) — locality datum for homemaker-py-nyb.","dependencies":[{"issue_id":"homemaker-py-8cs","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:34Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"homemaker-py-av5","title":"Batched oracle: score many .dom files per invocation","description":"oracle.py currently scores one .dom per urb-fitness.pl call (~1.65 s/dom). DESIGN.md §4.6: batching amortises Perl startup to ~0.99 s/dom and is required so population/batch optimisers can score a whole generation in one oracle call. Extend oracle.py with a batch API: write N .dom files, one perl invocation, parse N .score/.fails pairs. Keep the single-file path for compatibility.","acceptance_criteria":"Batch of 35 corpus files scores in one perl invocation; per-file results identical to single-file calls; measured s/dom reported","status":"closed","priority":1,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:56Z","created_by":"Bruno Postle","updated_at":"2026-06-12T00:14:06Z","started_at":"2026-06-11T23:50:40Z","closed_at":"2026-06-12T00:14:06Z","close_reason":"score_batch() lands in oracle.py; 35-file corpus parity verified single-vs-batch (1e-12 rel fitness, exact fail sets); 0.98 s/dom batched vs 1.27 single, x1.30","dependency_count":0,"dependent_count":1,"comment_count":0}
{"_type":"issue","id":"homemaker-py-tco","title":"A/B harnesses should report the minimum detectable difference for the N they run","description":"Two independent findings this session say the project's A/B protocols are\nroutinely underpowered for the margins they report:\n\n §38.19 programme-house needed N=60 to resolve an effect its own protocol\n claimed at N=20 (p ~= 0.069 at N=20, 0.017 at N=60)\n §38.21 harbor's paired sd is 6.19 fails, so n=3 resolves nothing finer than\n ~15.4 fails -- yet every recorded harbor margin is below that, and 25%\n of 3-seed subsets show a clean 3/3 sweep by chance\n\nThe fix is procedural and cheap. The A/B harnesses (run_*_ab.sh, ab_*.py,\nrerun_1ph_protocol.sh) should print, alongside the result, the minimum\ndifference their N and observed sd could have detected -- so an underpowered\nverdict is visible AT THE POINT IT IS MADE rather than years later.\n\nA one-line addition to each harness's summary: from the paired diffs it already\ncomputes, report\n\n minimum detectable difference = t_crit(0.975, N-1) * sd / sqrt(N)\n\nand flag when the observed margin falls below it. Optionally refuse to declare a\nwinner in that case.\n\nThis is not about re-running old A/Bs (§38.21 covers harbor, §38.19\nprogramme-house); it is about not generating more of them.\n","acceptance_criteria":"The shared A/B summary path reports the minimum detectable difference for the N actually run, and flags a verdict whose margin falls below it; applied to at least rerun_1ph_protocol.sh and ab_ssz_search.py.","status":"open","priority":2,"issue_type":"task","owner":"noreply@anthropic.com","created_at":"2026-08-29T19:34:30Z","created_by":"Claude","updated_at":"2026-08-29T19:35:08Z","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"homemaker-py-tco","title":"A/B harnesses should report the minimum detectable difference for the N they run","description":"Two independent findings this session say the project's A/B protocols are\nroutinely underpowered for the margins they report:\n\n §38.19 programme-house needed N=60 to resolve an effect its own protocol\n claimed at N=20 (p ~= 0.069 at N=20, 0.017 at N=60)\n §38.21 harbor's paired sd is 6.19 fails, so n=3 resolves nothing finer than\n ~15.4 fails -- yet every recorded harbor margin is below that, and 25%\n of 3-seed subsets show a clean 3/3 sweep by chance\n\nThe fix is procedural and cheap. The A/B harnesses (run_*_ab.sh, ab_*.py,\nrerun_1ph_protocol.sh) should print, alongside the result, the minimum\ndifference their N and observed sd could have detected -- so an underpowered\nverdict is visible AT THE POINT IT IS MADE rather than years later.\n\nA one-line addition to each harness's summary: from the paired diffs it already\ncomputes, report\n\n minimum detectable difference = t_crit(0.975, N-1) * sd / sqrt(N)\n\nand flag when the observed margin falls below it. Optionally refuse to declare a\nwinner in that case.\n\nThis is not about re-running old A/Bs (§38.21 covers harbor, §38.19\nprogramme-house); it is about not generating more of them.\n","acceptance_criteria":"The shared A/B summary path reports the minimum detectable difference for the N actually run, and flags a verdict whose margin falls below it; applied to at least rerun_1ph_protocol.sh and ab_ssz_search.py.","status":"closed","priority":2,"issue_type":"task","assignee":"Claude","owner":"noreply@anthropic.com","created_at":"2026-08-29T19:34:30Z","created_by":"Claude","updated_at":"2026-08-29T19:51:47Z","started_at":"2026-08-29T19:38:30Z","closed_at":"2026-08-29T19:51:47Z","close_reason":"Shipped experiments/ab_report.py (DESIGN.md §38.22).\n\n minimum detectable difference (MDD) = t_crit(0.975, N-1) * sd / sqrt(N)\n\nA margin below the MDD is not a weak result, it is an ABSENT one -- the\nexperiment could not have distinguished it from zero however it came out. The\nreport flags that, refuses to endorse a winner, and states the N that would be\nneeded. Also reports exact paired t, a Wilcoxon cross-check (fail counts are\nsmall integers, normality is not obvious), CI, and W/L/T.\n\nValidated against both datasets measured this session -- it reproduces the\nhand-computed figures exactly and flags correctly:\n\n programme-house N=60 +0.567 p=0.017 MDD 0.462 -\u003e verdict supported\n programme-house N=20 +0.700 p=0.085 MDD 0.805 -\u003e UNDERPOWERED, N~=26\n harbor N=24 +1.208 p=0.502 MDD 3.669 -\u003e UNDERPOWERED, N~=202\n\nHarbor needing ~200 seeds means it cannot answer the collapse_insearch question\nat any N this project would realistically run.\n\nCaught and fixed a defect in my own first version: with all-ties (sd=0) the MDD\ncollapses to zero, and the naive `abs(mean) \u003c mdd` reported \"margin exceeds the\nMDD -- verdict supported\" for a margin of 0.000, with t=nan. A reporter that\nendorses a zero margin is worse than none. Degenerate cases are now explicit and\ndistinguish all-ties from a constant non-zero difference.\n\nWired into ab_ssz_search.py; rerun_1ph_protocol.sh's TSV is readable directly by\nthe CLI.\n\nSEPARATE CORRECTION made while validating: §38.19's published p-values for N=20\nand N=40 were 0.069 and 0.045, from a normal approximation I used. The exact\npaired t-test gives 0.085 and 0.052 -- so N=40 did NOT reach significance either,\nand it took N=60. The approximation was anti-conservative, making results look\nmore significant than they are, which is the same direction of error this whole\nthread is about. Corrected at all four sites citing it, and §38.19's \"needs\nN \u003e= 40\" guidance raised to N \u003e= 60.\n","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"homemaker-py-ioe","title":"Is collapse_insearch=True still the right default under the current objective?","description":"The default was flipped OFF -\u003e ON by homemaker-py-1ph (DESIGN.md §20, 2026-07-24) on the strength of a programme-house N=20 sweep: mean 7.95 -\u003e 7.10, 11W/6L/3T, paired t p ~= 0.028. homemaker-py-d86 (§38.18) has now confirmed that verdict was sound FOR ITS OWN ERA -- it reproduces on a pre-iio commit, and the iio stale-share bug is structurally unreachable on that protocol because programme-house declares count: 1 for every code, so no leaf ever carries a share.\n\nBut the objective has changed substantially since, three times over, and all of it after 1ph:\n\n §39.4 the generic-namespace fix -- codes like cr1 were being read as generic\n circulation, so 14% of harbor's programme was silently optional\n §38.10 / §38.11 crinkliness declared per space; 14 corpus spaces now declare\n crinkliness: none\n §38.12 the missing-space cascade no longer weighted by YAML verbosity, a fixed\n 5 fails per missing instance instead of 3-5\n\ncollapse_insearch runs collapse_global inside every fitness eval, and collapse_global's assignment is valued against exactly the quality factors those changes touched. So the ON-beats-OFF margin was measured against an objective that no longer exists. The direction is plausibly unchanged -- but it is currently an assumption carried on a superseded measurement, and it is a DEFAULT, so every run inherits it.\n\nThe protocol and harness already exist: experiments/rerun_1ph_protocol.sh runs programme-house, budget 3000, 4 workers, seeds 1-20, ON vs OFF, both arms finished with --collapse, and takes about 6 minutes.\n\nNote when re-running: n_workers is an algorithm parameter (§38.17), so keep 4 workers to stay comparable with the published protocol, and record it with the result.","acceptance_criteria":"The 1ph protocol re-run at N=20 on the current codebase and objective, with the ON-vs-OFF verdict either reconfirmed or restated; if the margin has moved materially, DESIGN.md §20's default-flip rationale is updated to say so and the default is reconsidered on the new numbers.","status":"closed","priority":2,"issue_type":"task","assignee":"Claude","owner":"noreply@anthropic.com","created_at":"2026-08-29T13:36:18Z","created_by":"Claude","updated_at":"2026-08-29T13:57:45Z","started_at":"2026-08-29T13:36:32Z","closed_at":"2026-08-29T13:57:45Z","close_reason":"Re-validated: the default STANDS, but with two caveats worth carrying\n(DESIGN.md §38.19).\n\nRe-ran the 1ph protocol as published on the current codebase and objective --\nprogramme-house, budget 3000, 4 workers, ON vs OFF, both arms finished with\n--collapse.\n\n N OFF ON W/L/T diff t p\n published 1ph (07-24) 20 7.95 7.10 11/6/3 +0.85 2.38 0.028\n historical re-run (§38.18) 20 8.05 7.10 11/6/3 +0.95 2.59 --\n current objective 20 7.85 7.15 10/7/3 +0.70 1.82 0.069\n current objective 40 7.60 7.03 21/14/5 +0.57 2.01 0.045\n current objective 60 7.58 7.02 29/19/12 +0.57 2.45 0.017\n\nAt N=60: mean diff +0.567 fails/seed, paired t=2.454 (df=59), p=0.0171 exact,\n95% CI [+0.105, +1.029] excluding zero. Wilcoxon signed-rank cross-check agrees\n(p=0.0138), which matters because fail counts are small integers and normality\nis not obvious.\n\nCaveat 1: the effect is about a third smaller than published (+0.57 vs +0.85).\nPartly regression from a slightly lucky N=20 draw, partly plausible real erosion\n-- several fails collapse_global used to clear have been redefined out of\nexistence or made harder by §39.4 / §38.10-12.\n\nCaveat 2, the more useful one: THE PUBLISHED N=20 CAN NO LONGER DETECT ITS OWN\nEFFECT. At exactly the published sample size the current answer is p ~= 0.069, a\nnull by the conventional threshold. Had I run N=20 and stopped, the honest report\nwould have been \"the 1ph verdict no longer reproduces\" and the default would have\nlooked unjustified. It took N=60 to resolve. That is the \"8sh/1ph/qi6/lj3\npattern\" this log already warns about, now biting the flagship result itself.\nAny future re-validation of this default needs N \u003e= 40; N=20 should not be\ntrusted to settle it either way.\n\n§20 annotated in place so a reader of the original claim sees the current figure.\nHarness now takes a seed range (APPEND=1 to extend a sweep); results in\nexperiments/results/ioe_1ph_current_objective.tsv.\n","dependency_count":0,"dependent_count":0,"comment_count":0}
{"_type":"issue","id":"homemaker-py-vjd","title":"cpsat assignment ~7.5x slower after the 3qj adjacency: re-check §39.5's cpsat-vs-greedy verdict","description":"Declaring harbor's t -\u003e n adjacency (homemaker-py-3qj, DESIGN.md §38.14) made the CP-SAT room-labelling model markedly harder. constructive_topology seeding, 3-seed average:\\n\\n harbor greedy 0.06s -\u003e 0.06s (unchanged) cpsat 0.28s -\u003e 2.11s (7.5x)\\n maple greedy 0.03s -\u003e 0.03s (unchanged) cpsat 0.47s -\u003e 1.37s (2.9x)\\n\\nassign_solver defaults to greedy so ordinary runs pay nothing, and mutate_reassign/enable_reassign are opt-in too. But §39.5 concluded cpsat beats greedy on both programmes, and that was measured on a cheaper problem than the corpus now poses. The verdict needs re-checking on quality-per-second, not just quality.\\n\\nAlso worth checking whether cpsat.max_deterministic_time is now being hit, which would mean it is returning early rather than solving -- that would change the quality side of the comparison too, silently.","acceptance_criteria":"cpsat vs greedy re-measured on the current corpus for both solution quality AND wall time; §39.5's verdict either reconfirmed or restated; if max_deterministic_time is being hit, that is recorded and the limit reconsidered.","status":"closed","priority":2,"issue_type":"task","assignee":"Claude","owner":"noreply@anthropic.com","created_at":"2026-08-29T10:57:26Z","created_by":"Claude","updated_at":"2026-08-29T14:47:36Z","started_at":"2026-08-29T14:20:49Z","closed_at":"2026-08-29T14:47:36Z","close_reason":"Re-measured. Three findings (DESIGN.md §38.20).\n\n1. A LIVE BUG IN THE CAP, found on the way. solve_room_labels sets a\ndeterministic work-unit budget (max_deterministic_time=4.0) and a wall-clock\nbackstop, with the comment that the wall clock is \"a pathological-case backstop\nonly\". At its 2.0s value it had become THE BINDING CONSTRAINT: on harbor, 2 of\n24 solves returned FEASIBLE not OPTIMAL, wall time hit exactly 2010 ms, and the\ndeterministic budget was never reached (max 2.483 of 4.0). Those labellings were\nboth suboptimal AND load-dependent -- the wall clock is precisely the cap §39.5\nadded the deterministic one to escape. Cause: §38.14's `t -\u003e n` adjacency makes\nthe model much harder, and the 2s value dated from when solves took ~124 ms.\nRaised to 30s; now 24/24 harbor and 36/36 maple solves are OPTIMAL with the\ndeterministic budget still in headroom (max 3.569/4.0).\n\n2. THE VERDICT REVERSES. Re-measured deterministically (fdp's id()-ordering fix\nmeans the arms no longer differ by memory layout), 12 constructed seeds, scored\ncanonically:\n\n harbor greedy 722h/601s = 1323 0.079 s/seed\n harbor cpsat 908h/640s = 1548 1.623 s/seed\n maple greedy 777h/987s = 1764 0.063 s/seed\n maple cpsat 1213h/1043s= 2256 1.327 s/seed\n\ncpsat LOSES on both, +225 and +492 fails, at ~21x the seeding time, concentrated\nin hard fails.\n\n3. TIME AND QUALITY HAVE DIFFERENT CAUSES. Removing §38.14's `t -\u003e n` from\nharbor: cpsat goes 1.623 -\u003e 0.193 s/seed (8.4x faster) but still +205 vs greedy\n(was +225). So the adjacency explains the time blow-up and ~9% of the quality\ngap; the regression is otherwise pre-existing.\n\nSquaring with §39.5: that section records cpsat returning 194/180/171/182 over\nfour identical 10-seed aggregates before the determinism work. Its 10-fail\nharbor margin (102 vs 92) sits well inside a noise band that wide, and was\nmeasured with fdp's id()-ordered room_slots still live. So the seeder-level\n\"cpsat wins\" claim was never established rather than being overturned. §39.5\nannotated in place.\n\nCaveat stated in the write-up: absolute totals are ~6x §39.5's because the\nobjective has changed (§39.4, §38.10-12), so they are not directly comparable to\nthat table. The greedy-vs-cpsat comparison within this measurement is\nlike-for-like and is what the verdict rests on.\n\nNo default changes -- assign_solver was already greedy for §37.7's independent\nreason, and this reinforces it. What changes is that \"cpsat wins the seeder A/B\"\nshould no longer be cited as a reason to pursue it.\n\nCost recorded and filed as homemaker-py-2xk: the cap fix takes the test suite\nfrom ~4.5 to ~10 min, and the tests cannot opt out because constructive_topology\ndoes not thread the solver limits through.\n","comments":[{"id":"01a04dfd-ca76-7a21-b18d-d2af9ccd9ef4","issue_id":"homemaker-py-vjd","author":"Claude","text":"Correction: the close note above cites the follow-up as homemaker-py-2xk. That\nID does not exist -- I wrote it before creating the issue. The real one is\nhomemaker-py-7t1 (\"cpsat solver limits are not threadable from\nconstructive_topology, so tests pay full solve cost\"). DESIGN.md §38.20 has been\ncorrected to match.","created_at":"2026-08-29T14:47:53Z"}],"dependency_count":0,"dependent_count":0,"comment_count":1}
{"_type":"issue","id":"homemaker-py-9gj","title":"quality_uncrinkliness returns a flat hard 0.0, so the objective cannot rank two equally-buried layouts","description":"Narrowed remnant of homemaker-py-ssz after the owner's daylight ruling (DESIGN.md §38.11). A buried leaf usually IS a defect -- corridors and WCs included -- so scoring it badly is correct. The residual complaint is not that the value is low, it is that it is FLAT: quality_uncrinkliness returns exactly 0.0 for every zero-exposure leaf, and since evaluate_leaf multiplies factors into quality and process_storey accumulates value += quality * rate * area, two layouts that differ only in how badly buried their rooms are score identically.\n\nSo the objective gives the search no gradient to descend in precisely the region it most needs to escape. This is a search-mechanics problem, not a calibration one, and it should be judged on whether it helps the search escape -- NOT on fail counts, which by construction it will not move (the fails are real and should stay).\n\nNote the trap recorded in §38.9: an arm optimised under a modified objective must not be scored under the objective it modifies, and equally must not be scored under its own. For a pure gradient change that emits the same fail set, stock scoring IS valid -- that is the one case where the yardstick is sound.","acceptance_criteria":"A variant that keeps the fail set byte-identical to stock (every currently-failing leaf still fails) but is monotone in how buried a leaf is; A/B at fixed budget on harbor + maple with enough seeds to see past the one-seed variance that made the §38.8 n=3 result undecidable.","status":"open","priority":2,"issue_type":"bug","owner":"noreply@anthropic.com","created_at":"2026-08-28T23:14:04Z","created_by":"Claude","updated_at":"2026-08-28T23:14:04Z","dependency_count":0,"dependent_count":0,"comment_count":0}

58
DESIGN.md
View file

@ -2773,7 +2773,7 @@ path):
**Re-validated under the current objective (`homemaker-py-ioe`, §38.19):** the default still
stands, but the margin is about a third smaller (+0.57 fails/seed against the +0.85 below) and is
**no longer detectable at this section's N=20** (p ≈ 0.069 there); it takes N=60 to reach
**no longer detectable at this section's N=20** (p = 0.085 there); it takes N=60 to reach
p = 0.017. Do not re-check this default at N=20.
Confirms the qpk verdict holds at both example scales tested. `collapse_insearch` default flipped
@ -5862,10 +5862,17 @@ programme-house, budget 3000, 4 workers, ON vs OFF, both arms finished with
|---|---|---|---|---|---|---|---|
| published `1ph` (2026-07-24) | 20 | 7.95 | 7.10 | 11/6/3 | +0.85 | 2.38 | 0.028 |
| historical re-run (§38.18) | 20 | 8.05 | 7.10 | 11/6/3 | +0.95 | 2.59 | — |
| **current objective** | 20 | 7.85 | 7.15 | 10/7/3 | +0.70 | 1.82 | **0.069** |
| current objective | 40 | 7.60 | 7.03 | 21/14/5 | +0.57 | 2.01 | 0.045 |
| **current objective** | 20 | 7.85 | 7.15 | 10/7/3 | +0.70 | 1.82 | **0.085** |
| current objective | 40 | 7.60 | 7.03 | 21/14/5 | +0.57 | 2.01 | **0.052** |
| **current objective** | **60** | **7.58** | **7.02** | **29/19/12** | **+0.57** | **2.45** | **0.017** |
*(**Corrected.** The N=20 and N=40 p-values first published here were 0.069 and
0.045, from a normal approximation. The exact paired t-test gives **0.085** and
**0.052** — so **N=40 did not reach significance either**; it took N=60. The
approximation was anti-conservative, i.e. it made results look *more* significant
than they are, which is the same direction of error this section is about.
`experiments/ab_report.py` computes it exactly now, per `homemaker-py-tco`.)*
**Verdict: the default stands.** At N=60, mean diff **+0.567 fails/seed**,
paired t = 2.454 (df=59), **p = 0.0171** exact, 95% CI **[+0.105, +1.029]**
excluding zero. A Wilcoxon signed-rank cross-check agrees (p = 0.0138), which
@ -5881,12 +5888,13 @@ have been redefined out of existence or made harder by the objective work.
Second, and more usefully: **the published protocol's N=20 can no longer detect
its own effect.** At exactly the published sample size the current answer is
p ≈ 0.069 — a null by the conventional threshold. Had this been re-run at N=20
p = 0.085 — a null by the conventional threshold. Had this been re-run at N=20
and stopped there, the honest report would have been "the 1ph verdict no longer
reproduces", and the default would have looked unjustified. It took N=60 to
resolve. That is precisely the "8sh/1ph/qi6/lj3 pattern" this document already
warns about, now biting the flagship result itself: **any future re-validation
of this default needs N ≥ 40, and N=20 should not be trusted to settle it either
of this default needs N ≥ 60 (not the "≥ 40" first written here — N=40 gives
p = 0.052), and N=20 should not be trusted to settle it either
way.**
### 38.20 CP-SAT seeding re-measured deterministically: it loses (`homemaker-py-vjd`)
@ -6034,6 +6042,46 @@ nothing finer than ~15 fails, and programme-house at N=20 resolves nothing
finer than ~1 fail.* Any future A/B on these programmes should state its
detectable difference before running, not after.
### 38.22 A/Bs now report what their sample could resolve (`homemaker-py-tco`)
Three times in this log a verdict turned out to rest on a sample that could not
have produced it: §38.19 (programme-house claimed at N=20, resolves at N=60),
§38.21 (harbor at n=3 resolves nothing finer than ~15 fails, yet every recorded
margin is smaller), and §39.5/§38.20 (a 10-fail cpsat margin inside a ±23-fail
noise band). Each was found years later. `experiments/ab_report.py` makes it
visible at the point the verdict is made:
```
minimum detectable difference (MDD) = t_crit(0.975, N-1) * sd / sqrt(N)
```
A margin below the MDD is not a *weak* result, it is an **absent** one — the
experiment could not have distinguished it from zero however it came out. The
report flags that, refuses to endorse a winner, and states the N that would be
needed. Validated against both datasets measured this session:
| dataset | verdict | MDD | reported |
|---|---|---|---|
| programme-house N=60 | +0.567, p=0.017 | 0.462 | margin exceeds MDD — verdict supported |
| programme-house N=20 | +0.700, p=0.085 | 0.805 | **UNDERPOWERED**, N ≈ 26 needed |
| harbor N=24 | +1.208, p=0.502 | 3.669 | **UNDERPOWERED**, N ≈ **202** needed |
That last row is worth reading twice: harbor's `collapse_insearch` margin would
need ~200 seeds to resolve. At the 3 seeds it was published with, and at any N
this project would realistically run, harbor cannot answer that question at all.
**Degenerate cases are handled explicitly**, because the first version got one
wrong: with all-ties (`sd = 0`) the MDD collapses to zero and the naive test
`abs(mean) < mdd` reported "margin exceeds the MDD — verdict supported" for a
margin of 0.000, with `t = nan`. A reporter that endorses a zero margin is worse
than none. It now says "NO DIFFERENCE: identical on every seed — nothing to
test", and distinguishes that from a constant non-zero difference, where a
t-test is undefined but the result is real.
Wired into `experiments/ab_ssz_search.py`, which prints a power report per arm
before its summary table. `rerun_1ph_protocol.sh` writes a TSV the CLI reads
directly: `python experiments/ab_report.py <results.tsv> off on`.
## 39. Config audit: requirements that actively fight the engine (`homemaker-py-ju3`) — measured 2026-08-25
The corpus `patterns.config` targets and `costs.config` values were estimated

168
experiments/ab_report.py Normal file
View file

@ -0,0 +1,168 @@
"""Paired A/B reporting that states what the sample could actually resolve.
Motivation (`homemaker-py-tco`). Two findings in DESIGN.md §38.19 and §38.21 say
this project's A/B protocols are routinely underpowered for the margins they
report:
* programme-house needed N=60 to resolve an effect its own protocol claimed at
N=20 -- at N=20 the current answer is p ~= 0.069, a null;
* harbor's paired sd is ~6.2 fails, so n=3 resolves nothing finer than ~15
fails. Every recorded harbor margin is below that, and 25% of 3-seed subsets
show a clean 3/3 sweep by chance.
Both were only discovered years later. The cheap prevention is to print, beside
every verdict, the smallest difference the sample could have detected -- so an
underpowered result is visible AT THE POINT IT IS MADE.
minimum detectable difference (MDD) = t_crit(0.975, N-1) * sd / sqrt(N)
A margin below the MDD is not a weak result, it is an ABSENT one: the experiment
could not have distinguished it from zero however it came out. `paired_report`
flags exactly that, and reports the N that would be needed instead.
Usage as a library::
from ab_report import paired_report, format_report
print(format_report(paired_report(off_list, on_list, "OFF", "ON")))
Usage as a CLI over a results TSV with `tag seed arm value [...]` rows::
python experiments/ab_report.py results/0wr_qpk_harbor_n24.tsv off on
"""
from __future__ import annotations
import csv
import math
import statistics
import sys
try: # scipy is a hard dep of the
from scipy import stats as _sps # project, but keep the
except ImportError: # report usable without it
_sps = None
def _t_crit(df: int, alpha: float = 0.05) -> float:
if _sps is not None:
return float(_sps.t.ppf(1 - alpha / 2, df))
return 1.96 if df > 30 else 2.2 # crude, only for a scipy-less fallback
def paired_report(a: list[float], b: list[float],
label_a: str = "A", label_b: str = "B",
alpha: float = 0.05) -> dict:
"""Paired comparison of two arms measured on the same seeds.
``a`` and ``b`` are aligned per seed. Positive ``mean_diff`` means ``b``
scored LOWER than ``a`` -- for fail counts, that is ``b`` winning.
"""
if len(a) != len(b):
raise ValueError(f"unpaired input: {len(a)} vs {len(b)}")
n = len(a)
if n < 2:
raise ValueError("need at least 2 paired observations")
diffs = [x - y for x, y in zip(a, b)]
mean = statistics.mean(diffs)
sd = statistics.stdev(diffs)
se = sd / math.sqrt(n)
tc = _t_crit(n - 1, alpha)
mdd = tc * se # smallest |mean| this N could resolve
# Degenerate case: every seed gave the SAME difference, so sd == 0 and the
# MDD collapses to 0. Without this, an all-ties comparison (mean 0, sd 0)
# reports "margin exceeds the MDD" and endorses a zero margin as a verdict,
# which is worse than saying nothing. t/p are nan here too.
degenerate = sd == 0.0
out = {
"n": n, "label_a": label_a, "label_b": label_b,
"mean_a": statistics.mean(a), "mean_b": statistics.mean(b),
"wins_b": sum(1 for d in diffs if d > 0),
"losses_b": sum(1 for d in diffs if d < 0),
"ties": sum(1 for d in diffs if d == 0),
"mean_diff": mean, "sd": sd, "se": se,
"ci": (mean - tc * se, mean + tc * se),
"mdd": mdd,
"degenerate": degenerate,
"identical": degenerate and mean == 0.0,
"underpowered": (not degenerate) and abs(mean) < mdd,
"t": None, "p": None, "wilcoxon_p": None, "n_needed": None,
}
if _sps is not None and not degenerate:
t, p = _sps.ttest_rel(a, b)
out["t"], out["p"] = float(t), float(p)
if any(d != 0 for d in diffs):
try:
out["wilcoxon_p"] = float(_sps.wilcoxon(a, b).pvalue)
except ValueError:
pass
if mean and not degenerate: # N that would resolve THIS margin
k = n
while k < 100000 and _t_crit(k - 1, alpha) * sd / math.sqrt(k) >= abs(mean):
k += 1
out["n_needed"] = k
return out
def format_report(r: dict, indent: str = " ") -> str:
L = []
A, B = r["label_a"], r["label_b"]
L.append(f"{indent}N={r['n']} {A} mean {r['mean_a']:.2f} {B} mean {r['mean_b']:.2f} "
f"{r['wins_b']}W/{r['losses_b']}L/{r['ties']}T (for {B})")
L.append(f"{indent}mean diff {r['mean_diff']:+.3f} sd {r['sd']:.3f} "
f"95% CI [{r['ci'][0]:+.3f}, {r['ci'][1]:+.3f}]")
if r["p"] is not None:
w = "" if r["wilcoxon_p"] is None else f" wilcoxon p={r['wilcoxon_p']:.4f}"
L.append(f"{indent}t={r['t']:.3f} p={r['p']:.4f}{w}")
if r["identical"]:
L.append(f"{indent}** NO DIFFERENCE: the two arms scored identically on every seed.")
L.append(f"{indent} Nothing to test -- do not report a winner.")
return "\n".join(L)
if r["degenerate"]:
L.append(f"{indent}** Every seed gave the same difference ({r['mean_diff']:+.3f}), "
f"so the variance is zero.")
L.append(f"{indent} Consistent, but a t-test is undefined here; treat N as the "
f"evidence and check the arms are genuinely independent.")
return "\n".join(L)
L.append(f"{indent}minimum detectable difference at N={r['n']}: {r['mdd']:.3f}")
if r["underpowered"]:
need = r["n_needed"]
L.append(f"{indent}** UNDERPOWERED: the observed margin ({abs(r['mean_diff']):.3f}) is "
f"BELOW what N={r['n']} can resolve.")
L.append(f"{indent} This experiment could not distinguish it from zero however it "
f"came out. Do not declare a winner.")
if need:
L.append(f"{indent} N ~= {need} would be needed for a margin this size.")
else:
L.append(f"{indent} margin exceeds the MDD -- the sample can support a verdict.")
return "\n".join(L)
def main() -> int:
if len(sys.argv) < 4:
print(__doc__.strip().splitlines()[-1], file=sys.stderr)
print("usage: ab_report.py <results.tsv> <arm_a> <arm_b> [value_col=3]",
file=sys.stderr)
return 2
path, arm_a, arm_b = sys.argv[1], sys.argv[2], sys.argv[3]
col = int(sys.argv[4]) if len(sys.argv) > 4 else 3
by: dict = {}
with open(path) as fh:
for row in csv.reader(fh, delimiter="\t"):
if len(row) <= col or not row[1].lstrip("-").isdigit():
continue # header / short row
by.setdefault(int(row[1]), {})[row[2]] = float(row[col])
seeds = sorted(s for s, v in by.items() if arm_a in v and arm_b in v)
if len(seeds) < 2:
print(f"only {len(seeds)} paired seed(s) for {arm_a!r}/{arm_b!r}", file=sys.stderr)
return 1
a = [by[s][arm_a] for s in seeds]
b = [by[s][arm_b] for s in seeds]
print(f"{path} {arm_a} vs {arm_b}")
print(format_report(paired_report(a, b, arm_a, arm_b)))
return 0
if __name__ == "__main__":
raise SystemExit(main())

20
experiments/ab_ssz_search.py
View file

@ -126,6 +126,26 @@ def main() -> None:
w.writeheader()
w.writerows(rows)
# homemaker-py-tco: report what this N could actually resolve, beside the
# result, so an underpowered verdict is visible when it is made. §38.19 and
# §38.21 are both cases where that was only noticed years later.
from ab_report import format_report, paired_report
base = args.modes[0]
for progdir in args.corpus:
name = Path(progdir).name
for mode in args.modes[1:]:
by = {}
for r in rows:
if r["programme"] == name and r["mode"] in (base, mode):
by.setdefault(r["seed"], {})[r["mode"]] = r["urb_hard"] + r["urb_soft"]
seeds = sorted(s for s, v in by.items() if base in v and mode in v)
if len(seeds) < 2:
continue
print(f"\n--- {name}: {mode} vs {base} (stock-scored total fails) ---")
print(format_report(paired_report(
[by[s][base] for s in seeds], [by[s][mode] for s in seeds], base, mode)))
print(f"\n=== stock-objective (urb) fail counts, budget {args.budget} ===")
print(f" {'programme':<14}{'mode':<22}{'hard':<14}{'soft':<14}total")
print(" " + "-" * 70)