diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index 4df8970..97ecda8 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -112,7 +112,7 @@ {"_type":"issue","id":"homemaker-py-sd3","title":"driver.collapse_best bakes collapse_insearch=True into its finish evaluator, making the 94g keep-better guard vacuous","description":"Found by the homemaker-py-zrx expert review; same family as homemaker-py-7ua but in the PRODUCT (driver.py), not the experiment script. driver.collapse_best builds its evaluator as _fitness_for(str(programme_dir), leaf_sharing, superpose, multi_use=multi_use) — so collapse_insearch silently takes _fitness_for's default True. collapse_best has no collapse_insearch parameter, so evolve.py cannot thread the run's --collapse-insearch flag through even if it wanted to.\n\nConsequences, verified on 5 evolved harbor-house trees today: (1) the keep-better guard of collapse_finish is VACUOUS — base_fails is measured on a deepcopy that _evaluate_full re-collapses in-eval, so base == collapsed on 5/5 files (e.g. evolved-3M-nols-3.dom logs '12 -\u003e 12 (applied)' where the canonical evaluator shows the collapse actually did 15 -\u003e 12). The 94g safety property 'kept only if the fail count does not increase' is therefore not being checked against the true pre-collapse tree: a canonically fail-INCREASING collapse would be silently applied (collapse_global is 'monotone on harbor-house but not proven in general' per its own docstring — the guard exists precisely for that case). (2) The '[finish] collapse: N -\u003e M' log line under-reports the collapse's real effect (experiment logs quoting it understate 94g's contribution). (3) In a --no-collapse-insearch run the finish evaluator contradicts the run's objective outright — the deterministic 7ua mechanism, now in the default pipeline. (4) Minor: max_share and conn_grade are also not forwarded (matters for kpu/anneal and qi6 runs). Same pattern in search_annealed's final rescore branch: _evaluate(..., leaf_sharing=False, superpose=superpose) leaves _evaluate's collapse_insearch default True, and search_annealed has no way to pass the flag to it.\n\nOn the 5 probed files the returned tree's canonical fails happened to equal the reported number (the tree is a collapse fixpoint after iters=6 + 2-opt, so the extra in-eval collapse found nothing) — but that is not guaranteed, and the vacuous guard + misleading log line are unconditional.\n\nRecommended fix: add a collapse_insearch (and max_share/conn_grade) parameter to collapse_best, thread it from evolve.py, and make collapse_finish's keep-better measurement use a CANONICAL (collapse_insearch=False) evaluator regardless — the guard's job is to protect the canonical fail count of the written .dom, which homemaker-fitness scores with the on-disk config (no insearch override). Decide explicitly which objective the final 'best: N fails' report should quote (canonical is what the .dom.fails sidecar will say).","notes":"Fixed. Two changes: (1) fitness.collapse_finish now forces collapse_insearch=False (canonical) for its own base_fails/cand_fails measurement, saving/restoring self._collapse_insearch around the two score_with_fails calls -- regardless of how the Fitness instance itself was configured, so the guard can never again compare a pre-collapsed base against a pre-collapsed candidate. (2) driver.collapse_best now builds its evaluator with collapse_insearch=False explicitly (hardcoded, not threaded -- canonical is always the right objective for the 94g guard and the final reported fail count, matching what homemaker-fitness reports for the written .dom with no override), and threads max_share/conn_grade through to _fitness_for for config parity. Also fixed the same-family bug in search_annealed's no-polish-budget final rescore branch (was silently defaulting to collapse_insearch=True via _evaluate's default; now reads collapse_insearch/multi_use from search_kw). evolve.py forwards conn_grade to collapse_best. Added a regression test (test_collapse_finish_guard_is_canonical_even_with_insearch_collapse_on) that builds a Fitness with collapse_insearch=True in conf and asserts base_f still reflects the true pre-collapse fail count. Full suite: 405 passed (same 5 pre-existing CP-SAT/reassign failures, confirmed present on main before this change, unrelated).","status":"closed","priority":3,"issue_type":"bug","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-08-02T08:19:41Z","created_by":"Bruno Postle","updated_at":"2026-08-05T06:47:15Z","started_at":"2026-08-04T23:41:55Z","closed_at":"2026-08-05T06:47:15Z","close_reason":"94g keep-better guard is now non-vacuous; canonical scoring enforced","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"homemaker-py-d86","title":"Rigorously re-verify qpk/1ph historical numbers against the homemaker-py-iio fix","description":"homemaker-py-iio (fixed 2026-08-02) found a stale-leaf-share metadata leak\nin Fitness._collapse_value/_usage_quality that could corrupt one cell of\ncollapse_global's Hungarian assignment during any leaf_sharing+collapse\nrun -- i.e. essentially the entire \"full default stack\" used from\nhomemaker-py-x3b (leaf_sharing default-on) onward, including the very\nstudies that justified defaulting collapse_insearch on (94g, qpk/1ph, 8sh).\n\nA same-codebase fix-vs-no-fix re-run of the qpk protocol (harbor-house,\nbudget 2500, seeds 1-3) confirmed the bug demonstrably perturbs real\nper-seed outcomes under collapse_insearch=ON (2/3 seeds diverged by 5-8\nfails, non-directionally) -- see DESIGN.md §35 for full details. That\nre-run used TODAY's codebase, not the actual historical commit, and only 3\nharbor-house seeds, not the original seed sets -- so it establishes the bug\nwas real and non-trivial but does NOT establish whether 1ph's aggregate\nN=20 programme-house verdict (mean 7.95-\u003e7.10, paired t-test p~=0.028)\nwould have changed under the fix.\n\nThis issue is to do the rigorous version: check out the codebase near the\n1ph commit (~2026-07-24, \"post-qpk commits through 161\"), backport the iio\nfix there in an isolated worktree, and re-run the ACTUAL historical seed\nsets (programme-house N=20 seeds 1-20, harbor-house N=3 seeds 1-3) at the\n1ph protocol's exact parameters, comparing per-seed and aggregate results\nagainst the published numbers. Low priority: the qualitative direction of\nthe qpk/1ph conclusion is probably still right (noise is non-directional\nand the N=20 statistical margin is comfortably above the observed per-seed\nswing), this is about tightening confidence, not expecting a reversal.","notes":"homemaker-py-r5a (fixed 2026-08-02) also affects this: it is the COMMIT-door companion to iio (a leaf relabelled back to its own stale share_type resurrects a stale multiplicity credit). Any re-verification run here should use the codebase state after BOTH iio and r5a, not iio alone.","status":"open","priority":3,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-08-02T06:53:51Z","created_by":"Bruno Postle","updated_at":"2026-08-02T09:44:35Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"homemaker-py-7ua","title":"run_staged_search.py final rescore omits collapse_insearch override, causing false MISMATCH under leaf-sharing","description":"experiments/run_staged_search.py's _native_score() (used for the final 're-scored (native): ... -\u003e OK/MISMATCH' sanity line) calls fitness.load_config(programme_dir) with NO overrides, but driver.search_staged's internal evaluator always runs with collapse_insearch=True (baked into driver.search's default, search_staged has no param to disable it). The script's monkeypatched fitness.load_config only injects leaf_sharing/share_edge_cap/multi_use, not collapse_insearch, so the final rescore conf silently diverges from the search-time conf whenever leaf_sharing is on (the current default stack). Observed during homemaker-py-91f: a WORKERS=4 budget=2000 harbor-house run reported best fails=38 during search but re-scored fails=34 -\u003e MISMATCH (partly parallel non-determinism per homemaker-py-b8g, but the missing collapse_insearch override is a separate, deterministic contributor). Fix: add collapse_insearch=True to the monkeypatched conf alongside leaf_sharing/share_edge_cap.","status":"closed","priority":3,"issue_type":"bug","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-08-01T11:32:58Z","created_by":"Bruno Postle","updated_at":"2026-08-05T09:04:05Z","started_at":"2026-08-05T07:33:29Z","closed_at":"2026-08-05T09:04:05Z","close_reason":"Fixed: added collapse_insearch=True to the monkeypatched load_config conf in run_staged_search.py's leaf_share/multi_use branch. Verified with a smoke run (programme-house, LEAFSHARE=1, budget=150): pre-fix reported MISMATCH (1.51708e-08 vs 1.56663e-08), post-fix reports OK with identical values. Full pytest suite: 405 passed, 5 pre-existing failures in test_cpsat.py/test_operators.py unrelated to this change (confirmed failing on main before this fix too).","dependency_count":0,"dependent_count":0,"comment_count":0} -{"_type":"issue","id":"homemaker-py-b8g","title":"Investigate parallel/BLAS non-determinism noise source in n_workers\u003e1 runs","description":"DESIGN.md §14 (psk, island-model experiment) flagged a real, uninvestigated noise source: 'Phase A is unaffected by the probe, yet harbor seed 2 scored 71 then 73 on byte-identical re-runs -- parallel/BLAS non-determinism, the same +/-2-3 effect §12.4 flagged.' This is DISTINCT from the homemaker-py-xcy bug (ProcessPoolExecutor as_completed ordering), which was fixed and made same-worker-count parallel runs reproducible for the SEARCH TRAJECTORY. This remaining noise is at the SCORING level (a single fitness eval on a fixed genome apparently returning different fail counts across runs), plausibly numpy/scipy BLAS thread nondeterminism in the geometry/inner-loop math. It was never root-caused or fixed, and it widens the error bars on every A/B in this log run at n_workers\u003e1 (the great majority of them, since serial sweeps are expensive). Investigate: reproduce minimally (score the same frozen .dom N times under workers\u003e1), bisect whether it's BLAS threading (try OMP_NUM_THREADS=1/OPENBLAS_NUM_THREADS=1), floating-point summation order, or something else; fix or document a mitigation (e.g. pin thread count in worker processes).","design":"Reference: DESIGN.md §14 'Noise caveat (carry forward)', §12.4 (homemaker-py-xcy, the related-but-distinct trajectory-ordering bug already fixed). If the cause is BLAS thread count, the fix is likely a one-line env pin in the worker pool initializer (driver.py's ProcessPoolExecutor setup).","notes":"homemaker-py-zrx review (2026-08-02) found a concrete, non-BLAS candidate mechanism for part of this noise in PARALLEL STAGED runs: homemaker-py-cvw — substrate_readiness in the parent process reads stale id()-keyed geometry cache entries (24/300 corrupted in a churn probe, worst error ~1.0), perturbing stage-1 selection address-dependently across byte-identical re-runs. Does not explain fixed-genome single-eval divergence (if that was ever actually isolated); re-test after cvw lands before chasing BLAS.","status":"open","priority":3,"issue_type":"bug","owner":"bruno@postle.net","created_at":"2026-08-01T10:07:45Z","created_by":"Bruno Postle","updated_at":"2026-08-02T08:20:13Z","dependency_count":0,"dependent_count":1,"comment_count":0} +{"_type":"issue","id":"homemaker-py-b8g","title":"Investigate parallel/BLAS non-determinism noise source in n_workers\u003e1 runs","description":"DESIGN.md §14 (psk, island-model experiment) flagged a real, uninvestigated noise source: 'Phase A is unaffected by the probe, yet harbor seed 2 scored 71 then 73 on byte-identical re-runs -- parallel/BLAS non-determinism, the same +/-2-3 effect §12.4 flagged.' This is DISTINCT from the homemaker-py-xcy bug (ProcessPoolExecutor as_completed ordering), which was fixed and made same-worker-count parallel runs reproducible for the SEARCH TRAJECTORY. This remaining noise is at the SCORING level (a single fitness eval on a fixed genome apparently returning different fail counts across runs), plausibly numpy/scipy BLAS thread nondeterminism in the geometry/inner-loop math. It was never root-caused or fixed, and it widens the error bars on every A/B in this log run at n_workers\u003e1 (the great majority of them, since serial sweeps are expensive). Investigate: reproduce minimally (score the same frozen .dom N times under workers\u003e1), bisect whether it's BLAS threading (try OMP_NUM_THREADS=1/OPENBLAS_NUM_THREADS=1), floating-point summation order, or something else; fix or document a mitigation (e.g. pin thread count in worker processes).","design":"Reference: DESIGN.md §14 'Noise caveat (carry forward)', §12.4 (homemaker-py-xcy, the related-but-distinct trajectory-ordering bug already fixed). If the cause is BLAS thread count, the fix is likely a one-line env pin in the worker pool initializer (driver.py's ProcessPoolExecutor setup).","notes":"homemaker-py-zrx review (2026-08-02) found a concrete, non-BLAS candidate mechanism for part of this noise in PARALLEL STAGED runs: homemaker-py-cvw — substrate_readiness in the parent process reads stale id()-keyed geometry cache entries (24/300 corrupted in a churn probe, worst error ~1.0), perturbing stage-1 selection address-dependently across byte-identical re-runs. Does not explain fixed-genome single-eval divergence (if that was ever actually isolated); re-test after cvw lands before chasing BLAS.","status":"closed","priority":3,"issue_type":"bug","assignee":"Claude","owner":"bruno@postle.net","created_at":"2026-08-01T10:07:45Z","created_by":"Bruno Postle","updated_at":"2026-08-29T12:52:33Z","started_at":"2026-08-29T12:19:37Z","closed_at":"2026-08-29T12:52:33Z","close_reason":"Investigated. The premise does not survive measurement: there is no\nnon-determinism (DESIGN.md §38.17).\n\n score a frozen .dom, 20 repeats in one process bit-identical\n same .dom, 8 processes, varied PYTHONHASHSEED bit-identical\n full search, harbor seeds 0-3, n_workers 1..4,\n repeated across processes bit-identical PER worker count\n the same with OMP/OPENBLAS/MKL_NUM_THREADS=1 IDENTICAL to unpinned\n\nThat last line is the important one. This issue's proposed remedy was \"likely a\none-line env pin in the worker pool initializer\". Pinning the BLAS thread count\nchanges nothing bit-for-bit, so shipping it would have looked like a fix, done\nnothing, and retired the issue. BLAS is not implicated. The bead's other\nhypothesis -- \"a single fitness eval on a fixed genome returning different fail\ncounts\" -- is also false.\n\nWhat IS real, and is not noise: the trajectory is a deterministic FUNCTION of\nn_workers. harbor seed 3, budget 1500:\n\n w=1 64 fails 1.6264880162149419e-22\n w=2 64 fails, same bits\n w=3 64 fails, same bits\n w=4 65 fails 7.685882216045091e-23\n\nEach stable across processes. Mechanism, driver.py's batch loop:\n batch_n = min(n_workers, max(1, (budget - n_evals + child_budget - 1) // child_budget))\nbatch_n children are bred from ONE population snapshot before any is admitted,\nand the shared rng is consumed in a different pattern. At w=1 each child sees\nthe population its predecessor updated. So a 4-worker run is partly generational\nand a 1-worker run is steady-state -- same seed, different search. Divergence is\noccasional (seeds 0/1/2 agreed across worker counts, seed 3 did not), which is\nexactly how it reads as \"noise\" when sampled.\n\n§14's \"71 then 73 on byte-identical re-runs\" was most likely homemaker-py-xcy,\nthe as_completed admission-ordering bug, which WAS genuinely non-deterministic\nand is fixed. No residual noise behind it.\n\nShipped instead of a no-op env pin:\n - driver.search's docstring states n_workers is an ALGORITHM parameter and\n that results from different worker counts are not comparable.\n - test_search_is_reproducible_at_a_fixed_worker_count, parametrised over\n 2/3/4 workers -- asserts each is internally stable, and deliberately does\n NOT assert they agree with each other.\n - test_scoring_a_frozen_design_is_deterministic guards the floor.\n\nPractical exposure: the run_*_ab.sh harnesses already pin WORKERS=4 within a\nrun, so arms inside one harness are sound. The risk is comparing across\nharnesses, or against a historical figure whose worker count was not recorded.\n","dependency_count":0,"dependent_count":1,"comment_count":0} {"_type":"issue","id":"homemaker-py-7xb","title":"Validate full winning construction stack generalises to health-centre","description":"The whole positive construction-quality stack (adjacency-aware + proportion-aware seeding, depth-balanced growth, leaf-sharing factor 3, interior-O odiv=3, share-aware edge cap) has only ever been measured end-to-end on harbor-house and maple-court (DESIGN.md §11-§13, cumulative -54%/-41% vs the leu.2 baseline per §13.7). examples/health-centre exists (built for homemaker-py-9yx, a non-synthetic ~20-room programme of a different building type -- primary care, not house/co-housing) but has only ever been used to NULL-test ruin_recreate; the positive stack itself has never been run there. Run the current default full stack (staged search, matching the §13.9/§13.10 default config) on health-centre at a comparable budget/seed count to harbor/maple's Phase-8 measurements, and report whether the fail-count reduction pattern (dominated by leaf-sharing, then depth-balance synergy, then interior-O) holds on a structurally different programme mix, or whether health-centre's room-type diversity (19 distinct codes, mostly single-instance, per §32) changes which lever dominates.","design":"Reference: DESIGN.md §13.3/§13.5/§13.6/§13.9 (the levers to validate), §32 (9yx, health-centre's construction and room-code tiering). No new code expected -- this is a measurement run with the existing default-on stack, comparable to the leu.1/§12.1 benchmark-establishment style.","status":"open","priority":3,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-08-01T10:07:29Z","created_by":"Bruno Postle","updated_at":"2026-08-01T10:07:29Z","dependencies":[{"issue_id":"homemaker-py-7xb","depends_on_id":"homemaker-py-7b7","type":"blocks","created_at":"2026-08-26T16:20:35Z","created_by":"Claude","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"homemaker-py-fe2","title":"Experiment: 2-opt local-search polish inside collapse_insearch hot loop","description":"collapse_global's optional 2-opt adjacency polish (homemaker-py-9wi, §25) is proven positive and default-ON at finish-time (homemaker-py-cdl, §28: 46-file sweep, 0 regressions, 2 improvements incl. harbor evolved-anneal-3M 21-\u003e19). In-search collapse (collapse_insearch, homemaker-py-qpk/1ph, §20) is separately proven positive and default-ON (~11% mean fail reduction on both example programmes at N=15/20). But the two have never been combined: §28 explicitly left collapse_global's method-level local_search default OFF because 2-opt running inside the per-eval hot loop (thousands of calls per search) was 'untested and likely-costly, out of scope' for that issue -- only the one-shot finish-time cost (\u003c1s even on the largest file) was measured. This issue is the measurement: A/B collapse_insearch with local_search=True vs False (both already default-on baseline), on harbor-house and maple-court, staged search, matching the qpk/1ph protocol (equal budget, keep-better guard already monotone by construction). Report both the wall-clock cost multiplier and any fail-count effect; only recommend a default flip if positive and the cost is not prohibitive.","design":"Reference: DESIGN.md §20 (qpk), §25 (9wi), §28 (cdl) 'Where the default did NOT change' paragraph. Protocol: mirror experiments/run_qi6_ab.sh / run_lj3_qjg_ab.sh style equal-budget A/B, finish with standard --collapse, canonical homemaker-fitness re-score.","status":"open","priority":3,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-08-01T10:07:12Z","created_by":"Bruno Postle","updated_at":"2026-08-01T10:07:12Z","dependency_count":0,"dependent_count":0,"comment_count":0} {"_type":"issue","id":"homemaker-py-9yx","title":"Non-synthetic third example programme to isolate ruin_recreate room-count threshold","description":"y51/xyu follow-up (option b, not run by xyu). The synthetic n=10/14/18/22 sweep scales room count by duplicating already-interchangeable programme-house room codes (count: on b1/t1/b2/t2) -- the same mechanism harbor-house itself uses 'to reduce complexity'. xyu extended n=18 to N=15 seeds (DESIGN.md 31): trend weakened but did not evaporate (9.3%-\u003e6.4%, two-sided Wilcoxon p 0.098-\u003e0.059), still ambiguous. A genuinely distinct third example programme with real room-type diversity at an intermediate room count (not a duplicated-code scale-up) would avoid the interchangeable-room confound and better isolate room count as the driving variable behind the wing-rebuild-fraction hypothesis from f1d (DESIGN.md 23).","notes":"RESOLVED (2026-07-30, DESIGN.md §32): built examples/health-centre, a 19-code/\nn=20 real health-centre programme (not duplicated-count). Wilcoxon N=15 vs\nxyu's own protocol: 8W/5L/2T, mean fails 46.13-\u003e45.13, delta=2.2%, two-sided\np=0.40, one-sided p=0.20 -- a clean null, weaker even than xyu's own\ninconclusive 6.4%/p=0.059 reading at the same room count. Converges with\nharbor-house's null-to-negative result rather than y51's synthetic sweep.\nConclusion: the y51/xyu signal was substantially an artifact of the\nduplicated-interchangeable-code mechanism, not a real room-count effect.\nenable_ruin_recreate stays OFF. No further follow-up filed.\n\nNote en route: first draft of health-centre's room sizes auto-derived into\none 19-code interchange class (9o5's transitive chain) -- fixed by tiering\nroom widths with \u003e1.3x gaps at 3 boundaries into 3 bounded classes. Worth\nremembering for any future non-synthetic programme design.","status":"closed","priority":3,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-07-29T09:07:48Z","created_by":"Bruno Postle","updated_at":"2026-07-30T07:07:09Z","started_at":"2026-07-29T14:05:00Z","closed_at":"2026-07-30T07:07:09Z","close_reason":"Closed","dependency_count":0,"dependent_count":0,"comment_count":0} diff --git a/DESIGN.md b/DESIGN.md index 46ae21a..8b419f3 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -5717,6 +5717,67 @@ that the fail *count* matched and only the continuous score moved, which the this dangerous: a harness that reports MISMATCH on its own control, in a way the metric-of-record cannot see, trains everyone to ignore the warning. +### 38.17 `n_workers` is an algorithm parameter, not noise (`homemaker-py-b8g`) + +§14 recorded "harbor seed 2 scored 71 then 73 on byte-identical re-runs — +parallel/BLAS non-determinism", and `b8g` carried that forward as an +uninvestigated noise source widening the error bars on every A/B run at +`n_workers>1`. **The premise does not survive measurement.** Nothing is +non-deterministic: + +| test | result | +|---|---| +| score a frozen `.dom`, 20 repeats in one process | bit-identical | +| same `.dom`, 8 processes, different `PYTHONHASHSEED` | bit-identical | +| full search, harbor, seeds 0/1/2/3, `n_workers` 1..4, repeated across processes | bit-identical **per worker count** | +| the same, with `OMP_NUM_THREADS=OPENBLAS_NUM_THREADS=MKL_NUM_THREADS=1` | **identical to unpinned** | + +That last row matters most. `b8g`'s proposed remedy was "likely a one-line env +pin in the worker pool initializer". Pinning the BLAS thread count changes +nothing at all — bit-for-bit — so shipping that mitigation would have looked +like a fix and done nothing, while retiring the issue. BLAS is not implicated. + +**What is real, and it is not noise.** The trajectory is a deterministic +*function of* `n_workers`. harbor seed 3, budget 1500: + +| `n_workers` | best | +|---|---| +| 1 | 64 fails, 1.6264880162149419e-22 | +| 2 | 64 fails, same bits | +| 3 | 64 fails, same bits | +| 4 | **65 fails, 7.685882216045091e-23** | + +Each is perfectly stable on its own across processes. The mechanism is at +`driver.py`'s batch loop: + +```python +batch_n = min(n_workers, max(1, (budget - n_evals + child_budget - 1) // child_budget)) +``` + +`batch_n` children are bred from **one population snapshot** before any of them +is admitted, and the shared `rng` is consumed in a different pattern. At +`n_workers=1` each child sees the population its predecessor updated. So a +4-worker run is a partly-generational algorithm and a 1-worker run is +steady-state — the same seed, a different search. (Seeds 0/1/2 happened to agree +across worker counts and seed 3 did not; divergence is occasional, not constant, +which is exactly how it reads as "noise" when sampled.) + +**Consequence for the A/B record.** `n_workers` must be treated as part of an +arm's configuration. Comparing a result measured at 4 workers against one +measured at 1 compares two algorithms. The `run_*_ab.sh` harnesses already pin +`WORKERS=4` within a run, so arms inside one harness are sound; the exposure is +comparing across harnesses, or against a historical figure whose worker count +was not recorded. + +**§14's original observation was most likely `homemaker-py-xcy`** — the +`as_completed` admission-ordering bug, which was genuinely non-deterministic and +has since been fixed. There is no residual noise behind it. + +Guarded by `test_search_is_reproducible_at_a_fixed_worker_count` (parametrised +over 2/3/4 workers, asserting each is internally stable and deliberately not +asserting they agree with each other) and +`test_scoring_a_frozen_design_is_deterministic`. + ## 39. Config audit: requirements that actively fight the engine (`homemaker-py-ju3`) — measured 2026-08-25 The corpus `patterns.config` targets and `costs.config` values were estimated diff --git a/src/homemaker_layout/driver.py b/src/homemaker_layout/driver.py index 527de42..98dd6b6 100644 --- a/src/homemaker_layout/driver.py +++ b/src/homemaker_layout/driver.py @@ -331,7 +331,16 @@ def search( legacy single-seed path (appropriate for warm starts from existing designs). ``n_workers=1`` (default) runs serially; ``n_workers > 1`` evaluates - children in parallel using ``ProcessPoolExecutor``. The bootstrap batch + children in parallel using ``ProcessPoolExecutor``. + + **``n_workers`` is an ALGORITHM parameter, not just a speed knob** + (homemaker-py-b8g, DESIGN.md §38.17). ``batch_n = min(n_workers, ...)`` + children are bred from ONE population snapshot before any of them is + admitted, and the shared ``rng`` is consumed in a different pattern, so a + run at ``n_workers=4`` explores a different trajectory from the same seed at + ``n_workers=1``. Each worker count is bit-reproducible on its own; results + from DIFFERENT worker counts are not comparable, and an A/B whose arms differ + in ``n_workers`` is comparing two algorithms, not two configurations. The bootstrap batch is fully parallel; the main loop generates ``n_workers`` children per iteration from the current population snapshot and evaluates them in parallel. Results are admitted in completion order (fastest first), so diff --git a/tests/test_driver.py b/tests/test_driver.py index 4383936..a56df28 100644 --- a/tests/test_driver.py +++ b/tests/test_driver.py @@ -6,7 +6,7 @@ from pathlib import Path import numpy as np import pytest -from homemaker_layout import dom, driver, innerloop, solver +from homemaker_layout import dom, driver, fitness, innerloop, solver CORPUS = Path(__file__).parent.parent / "examples" / "programme-house" SEED_FILE = CORPUS / "c964435454c459f86c3ed9a5a7621132.dom" @@ -506,23 +506,50 @@ def test_search_parallel_smoke(): assert r.n_topologies >= 2 # at least the bootstrap individuals -def test_search_parallel_is_reproducible(): - """Two same-seed parallel runs must be byte-identical (homemaker-py-xcy). +@pytest.mark.parametrize("workers", [2, 3, 4]) +def test_search_is_reproducible_at_a_fixed_worker_count(workers): + """Same seed + SAME worker count => byte-identical (homemaker-py-xcy/b8g). ``_run_batch`` used to admit futures in completion order (``as_completed``), which varies run-to-run; with the order-sensitive ``admit`` (n_evals accrual, first-of-tie wins ``best``) that made parallel searches non-reproducible. - Admitting in submission order fixed it. Guard the invariant directly: same - seed + same worker count ⇒ identical best (n_fails, fitness, signature) and - identical improvement history.""" + Admitting in submission order fixed it. + + Note the invariant is per worker count, and deliberately so. `n_workers` is + an algorithm parameter: `batch_n = min(n_workers, ...)` children are bred + from one population snapshot before any is admitted, so different worker + counts explore different trajectories from the same seed (§38.17). This + parametrises over several counts to check each is internally stable; it does + NOT assert that they agree with each other, because they legitimately need + not. + """ def run(): r = driver.search(dom.load(str(INIT_FILE)), CORPUS, budget=1200, - pop_size=8, child_budget=80, seed=0, n_workers=3) + pop_size=8, child_budget=80, seed=0, n_workers=workers) return (r.best.n_fails, r.best.fitness, r.best.sig, tuple(r.history)) - a = run() - b = run() - assert a == b, "parallel search is not reproducible run-to-run" + assert run() == run(), ( + f"search at n_workers={workers} is not reproducible run-to-run") + + +def test_scoring_a_frozen_design_is_deterministic(): + """No floating-point/BLAS nondeterminism in a single eval (homemaker-py-b8g). + + b8g suspected "a single fitness eval on a fixed genome returning different + fail counts across runs", plausibly BLAS threading. It does not: measured + bit-identical over 20 in-process repeats and 8 processes with different + PYTHONHASHSEED, and pinning OMP/OPENBLAS/MKL to one thread changes nothing. + This guards the floor the reproducibility argument stands on. + """ + import copy + + conf, cost = fitness.load_config(CORPUS) + root = dom.load(str(INIT_FILE)) + results = { + fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(root)) + for _ in range(8) + } + assert len(results) == 1, "scoring a frozen design is not deterministic" def _shared_best_result() -> driver.SearchResult: