n_workers is an algorithm parameter, not noise

14 recorded "harbor seed 2 scored 71 then 73 on byte-identical re-runs --
parallel/BLAS non-determinism", and b8g carried that forward as noise
widening the error bars on every A/B run at n_workers>1. The premise does
not survive measurement. Nothing is non-deterministic:

  score a frozen .dom, 20 repeats in one process   bit-identical
  same .dom, 8 processes, varied PYTHONHASHSEED    bit-identical
  full search, harbor seeds 0-3, n_workers 1..4,
    repeated across processes                      bit-identical PER count
  the same with OMP/OPENBLAS/MKL_NUM_THREADS=1     IDENTICAL to unpinned

The last line matters most: b8g proposed "likely a one-line env pin in the
worker pool initializer". Pinning BLAS threads changes nothing bit-for-bit,
so shipping that would have looked like a fix, done nothing, and retired
the issue.

What is real is not noise: the trajectory is a deterministic function of
n_workers. harbor seed 3, budget 1500 -- w=1/2/3 all give 64 fails with
identical bits, w=4 gives 65. Each stable across processes. The mechanism
is batch_n = min(n_workers, ...) children bred from ONE population snapshot
before any is admitted, with the shared rng consumed in a different
pattern; at w=1 each child sees the population its predecessor updated. A
4-worker run is partly generational, a 1-worker run steady-state -- same
seed, different search. Divergence is occasional (seeds 0/1/2 agreed, seed
3 did not), which is how it reads as noise when sampled.

14's observation was most likely homemaker-py-xcy, the as_completed
admission-ordering bug, which WAS non-deterministic and is fixed.

Shipped instead of a no-op env pin: driver.search's docstring states the
contract; test_search_is_reproducible_at_a_fixed_worker_count parametrises
over 2/3/4 workers, asserting each is internally stable and deliberately
NOT that they agree; test_scoring_a_frozen_design_is_deterministic guards
the floor.

The run_*_ab.sh harnesses already pin WORKERS=4, so arms inside one harness
are sound. The exposure is comparing across harnesses, or against a
historical figure whose worker count was never recorded.

Closes homemaker-py-b8g.

Lint at parity (46); tests 384 passed (3 new), 0 failed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
This commit is contained in:
Claude 2026-08-29 12:52:33 +00:00
parent 225e673ae3
commit f6eeee7191
No known key found for this signature in database
4 changed files with 109 additions and 12 deletions

File diff suppressed because one or more lines are too long

View file

@ -5717,6 +5717,67 @@ that the fail *count* matched and only the continuous score moved, which the
this dangerous: a harness that reports MISMATCH on its own control, in a way the
metric-of-record cannot see, trains everyone to ignore the warning.
### 38.17 `n_workers` is an algorithm parameter, not noise (`homemaker-py-b8g`)
§14 recorded "harbor seed 2 scored 71 then 73 on byte-identical re-runs —
parallel/BLAS non-determinism", and `b8g` carried that forward as an
uninvestigated noise source widening the error bars on every A/B run at
`n_workers>1`. **The premise does not survive measurement.** Nothing is
non-deterministic:
| test | result |
|---|---|
| score a frozen `.dom`, 20 repeats in one process | bit-identical |
| same `.dom`, 8 processes, different `PYTHONHASHSEED` | bit-identical |
| full search, harbor, seeds 0/1/2/3, `n_workers` 1..4, repeated across processes | bit-identical **per worker count** |
| the same, with `OMP_NUM_THREADS=OPENBLAS_NUM_THREADS=MKL_NUM_THREADS=1` | **identical to unpinned** |
That last row matters most. `b8g`'s proposed remedy was "likely a one-line env
pin in the worker pool initializer". Pinning the BLAS thread count changes
nothing at all — bit-for-bit — so shipping that mitigation would have looked
like a fix and done nothing, while retiring the issue. BLAS is not implicated.
**What is real, and it is not noise.** The trajectory is a deterministic
*function of* `n_workers`. harbor seed 3, budget 1500:
| `n_workers` | best |
|---|---|
| 1 | 64 fails, 1.6264880162149419e-22 |
| 2 | 64 fails, same bits |
| 3 | 64 fails, same bits |
| 4 | **65 fails, 7.685882216045091e-23** |
Each is perfectly stable on its own across processes. The mechanism is at
`driver.py`'s batch loop:
```python
batch_n = min(n_workers, max(1, (budget - n_evals + child_budget - 1) // child_budget))
```
`batch_n` children are bred from **one population snapshot** before any of them
is admitted, and the shared `rng` is consumed in a different pattern. At
`n_workers=1` each child sees the population its predecessor updated. So a
4-worker run is a partly-generational algorithm and a 1-worker run is
steady-state — the same seed, a different search. (Seeds 0/1/2 happened to agree
across worker counts and seed 3 did not; divergence is occasional, not constant,
which is exactly how it reads as "noise" when sampled.)
**Consequence for the A/B record.** `n_workers` must be treated as part of an
arm's configuration. Comparing a result measured at 4 workers against one
measured at 1 compares two algorithms. The `run_*_ab.sh` harnesses already pin
`WORKERS=4` within a run, so arms inside one harness are sound; the exposure is
comparing across harnesses, or against a historical figure whose worker count
was not recorded.
**§14's original observation was most likely `homemaker-py-xcy`** — the
`as_completed` admission-ordering bug, which was genuinely non-deterministic and
has since been fixed. There is no residual noise behind it.
Guarded by `test_search_is_reproducible_at_a_fixed_worker_count` (parametrised
over 2/3/4 workers, asserting each is internally stable and deliberately not
asserting they agree with each other) and
`test_scoring_a_frozen_design_is_deterministic`.
## 39. Config audit: requirements that actively fight the engine (`homemaker-py-ju3`) — measured 2026-08-25
The corpus `patterns.config` targets and `costs.config` values were estimated

View file

@ -331,7 +331,16 @@ def search(
legacy single-seed path (appropriate for warm starts from existing designs).
``n_workers=1`` (default) runs serially; ``n_workers > 1`` evaluates
children in parallel using ``ProcessPoolExecutor``. The bootstrap batch
children in parallel using ``ProcessPoolExecutor``.
**``n_workers`` is an ALGORITHM parameter, not just a speed knob**
(homemaker-py-b8g, DESIGN.md §38.17). ``batch_n = min(n_workers, ...)``
children are bred from ONE population snapshot before any of them is
admitted, and the shared ``rng`` is consumed in a different pattern, so a
run at ``n_workers=4`` explores a different trajectory from the same seed at
``n_workers=1``. Each worker count is bit-reproducible on its own; results
from DIFFERENT worker counts are not comparable, and an A/B whose arms differ
in ``n_workers`` is comparing two algorithms, not two configurations. The bootstrap batch
is fully parallel; the main loop generates ``n_workers`` children per
iteration from the current population snapshot and evaluates them in
parallel. Results are admitted in completion order (fastest first), so

View file

@ -6,7 +6,7 @@ from pathlib import Path
import numpy as np
import pytest
from homemaker_layout import dom, driver, innerloop, solver
from homemaker_layout import dom, driver, fitness, innerloop, solver
CORPUS = Path(__file__).parent.parent / "examples" / "programme-house"
SEED_FILE = CORPUS / "c964435454c459f86c3ed9a5a7621132.dom"
@ -506,23 +506,50 @@ def test_search_parallel_smoke():
assert r.n_topologies >= 2 # at least the bootstrap individuals
def test_search_parallel_is_reproducible():
"""Two same-seed parallel runs must be byte-identical (homemaker-py-xcy).
@pytest.mark.parametrize("workers", [2, 3, 4])
def test_search_is_reproducible_at_a_fixed_worker_count(workers):
"""Same seed + SAME worker count => byte-identical (homemaker-py-xcy/b8g).
``_run_batch`` used to admit futures in completion order (``as_completed``),
which varies run-to-run; with the order-sensitive ``admit`` (n_evals accrual,
first-of-tie wins ``best``) that made parallel searches non-reproducible.
Admitting in submission order fixed it. Guard the invariant directly: same
seed + same worker count identical best (n_fails, fitness, signature) and
identical improvement history."""
Admitting in submission order fixed it.
Note the invariant is per worker count, and deliberately so. `n_workers` is
an algorithm parameter: `batch_n = min(n_workers, ...)` children are bred
from one population snapshot before any is admitted, so different worker
counts explore different trajectories from the same seed (§38.17). This
parametrises over several counts to check each is internally stable; it does
NOT assert that they agree with each other, because they legitimately need
not.
"""
def run():
r = driver.search(dom.load(str(INIT_FILE)), CORPUS, budget=1200,
pop_size=8, child_budget=80, seed=0, n_workers=3)
pop_size=8, child_budget=80, seed=0, n_workers=workers)
return (r.best.n_fails, r.best.fitness, r.best.sig, tuple(r.history))
a = run()
b = run()
assert a == b, "parallel search is not reproducible run-to-run"
assert run() == run(), (
f"search at n_workers={workers} is not reproducible run-to-run")
def test_scoring_a_frozen_design_is_deterministic():
"""No floating-point/BLAS nondeterminism in a single eval (homemaker-py-b8g).
b8g suspected "a single fitness eval on a fixed genome returning different
fail counts across runs", plausibly BLAS threading. It does not: measured
bit-identical over 20 in-process repeats and 8 processes with different
PYTHONHASHSEED, and pinning OMP/OPENBLAS/MKL to one thread changes nothing.
This guards the floor the reproducibility argument stands on.
"""
import copy
conf, cost = fitness.load_config(CORPUS)
root = dom.load(str(INIT_FILE))
results = {
fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(root))
for _ in range(8)
}
assert len(results) == 1, "scoring a frozen design is not deterministic"
def _shared_best_result() -> driver.SearchResult: