n_workers is an algorithm parameter, not noise
14 recorded "harbor seed 2 scored 71 then 73 on byte-identical re-runs --
parallel/BLAS non-determinism", and b8g carried that forward as noise
widening the error bars on every A/B run at n_workers>1. The premise does
not survive measurement. Nothing is non-deterministic:
score a frozen .dom, 20 repeats in one process bit-identical
same .dom, 8 processes, varied PYTHONHASHSEED bit-identical
full search, harbor seeds 0-3, n_workers 1..4,
repeated across processes bit-identical PER count
the same with OMP/OPENBLAS/MKL_NUM_THREADS=1 IDENTICAL to unpinned
The last line matters most: b8g proposed "likely a one-line env pin in the
worker pool initializer". Pinning BLAS threads changes nothing bit-for-bit,
so shipping that would have looked like a fix, done nothing, and retired
the issue.
What is real is not noise: the trajectory is a deterministic function of
n_workers. harbor seed 3, budget 1500 -- w=1/2/3 all give 64 fails with
identical bits, w=4 gives 65. Each stable across processes. The mechanism
is batch_n = min(n_workers, ...) children bred from ONE population snapshot
before any is admitted, with the shared rng consumed in a different
pattern; at w=1 each child sees the population its predecessor updated. A
4-worker run is partly generational, a 1-worker run steady-state -- same
seed, different search. Divergence is occasional (seeds 0/1/2 agreed, seed
3 did not), which is how it reads as noise when sampled.
14's observation was most likely homemaker-py-xcy, the as_completed
admission-ordering bug, which WAS non-deterministic and is fixed.
Shipped instead of a no-op env pin: driver.search's docstring states the
contract; test_search_is_reproducible_at_a_fixed_worker_count parametrises
over 2/3/4 workers, asserting each is internally stable and deliberately
NOT that they agree; test_scoring_a_frozen_design_is_deterministic guards
the floor.
The run_*_ab.sh harnesses already pin WORKERS=4, so arms inside one harness
are sound. The exposure is comparing across harnesses, or against a
historical figure whose worker count was never recorded.
Closes homemaker-py-b8g.
Lint at parity (46); tests 384 passed (3 new), 0 failed.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
This commit is contained in:
parent
225e673ae3
commit
f6eeee7191
4 changed files with 109 additions and 12 deletions
File diff suppressed because one or more lines are too long
61
DESIGN.md
61
DESIGN.md
|
|
@ -5717,6 +5717,67 @@ that the fail *count* matched and only the continuous score moved, which the
|
|||
this dangerous: a harness that reports MISMATCH on its own control, in a way the
|
||||
metric-of-record cannot see, trains everyone to ignore the warning.
|
||||
|
||||
### 38.17 `n_workers` is an algorithm parameter, not noise (`homemaker-py-b8g`)
|
||||
|
||||
§14 recorded "harbor seed 2 scored 71 then 73 on byte-identical re-runs —
|
||||
parallel/BLAS non-determinism", and `b8g` carried that forward as an
|
||||
uninvestigated noise source widening the error bars on every A/B run at
|
||||
`n_workers>1`. **The premise does not survive measurement.** Nothing is
|
||||
non-deterministic:
|
||||
|
||||
| test | result |
|
||||
|---|---|
|
||||
| score a frozen `.dom`, 20 repeats in one process | bit-identical |
|
||||
| same `.dom`, 8 processes, different `PYTHONHASHSEED` | bit-identical |
|
||||
| full search, harbor, seeds 0/1/2/3, `n_workers` 1..4, repeated across processes | bit-identical **per worker count** |
|
||||
| the same, with `OMP_NUM_THREADS=OPENBLAS_NUM_THREADS=MKL_NUM_THREADS=1` | **identical to unpinned** |
|
||||
|
||||
That last row matters most. `b8g`'s proposed remedy was "likely a one-line env
|
||||
pin in the worker pool initializer". Pinning the BLAS thread count changes
|
||||
nothing at all — bit-for-bit — so shipping that mitigation would have looked
|
||||
like a fix and done nothing, while retiring the issue. BLAS is not implicated.
|
||||
|
||||
**What is real, and it is not noise.** The trajectory is a deterministic
|
||||
*function of* `n_workers`. harbor seed 3, budget 1500:
|
||||
|
||||
| `n_workers` | best |
|
||||
|---|---|
|
||||
| 1 | 64 fails, 1.6264880162149419e-22 |
|
||||
| 2 | 64 fails, same bits |
|
||||
| 3 | 64 fails, same bits |
|
||||
| 4 | **65 fails, 7.685882216045091e-23** |
|
||||
|
||||
Each is perfectly stable on its own across processes. The mechanism is at
|
||||
`driver.py`'s batch loop:
|
||||
|
||||
```python
|
||||
batch_n = min(n_workers, max(1, (budget - n_evals + child_budget - 1) // child_budget))
|
||||
```
|
||||
|
||||
`batch_n` children are bred from **one population snapshot** before any of them
|
||||
is admitted, and the shared `rng` is consumed in a different pattern. At
|
||||
`n_workers=1` each child sees the population its predecessor updated. So a
|
||||
4-worker run is a partly-generational algorithm and a 1-worker run is
|
||||
steady-state — the same seed, a different search. (Seeds 0/1/2 happened to agree
|
||||
across worker counts and seed 3 did not; divergence is occasional, not constant,
|
||||
which is exactly how it reads as "noise" when sampled.)
|
||||
|
||||
**Consequence for the A/B record.** `n_workers` must be treated as part of an
|
||||
arm's configuration. Comparing a result measured at 4 workers against one
|
||||
measured at 1 compares two algorithms. The `run_*_ab.sh` harnesses already pin
|
||||
`WORKERS=4` within a run, so arms inside one harness are sound; the exposure is
|
||||
comparing across harnesses, or against a historical figure whose worker count
|
||||
was not recorded.
|
||||
|
||||
**§14's original observation was most likely `homemaker-py-xcy`** — the
|
||||
`as_completed` admission-ordering bug, which was genuinely non-deterministic and
|
||||
has since been fixed. There is no residual noise behind it.
|
||||
|
||||
Guarded by `test_search_is_reproducible_at_a_fixed_worker_count` (parametrised
|
||||
over 2/3/4 workers, asserting each is internally stable and deliberately not
|
||||
asserting they agree with each other) and
|
||||
`test_scoring_a_frozen_design_is_deterministic`.
|
||||
|
||||
## 39. Config audit: requirements that actively fight the engine (`homemaker-py-ju3`) — measured 2026-08-25
|
||||
|
||||
The corpus `patterns.config` targets and `costs.config` values were estimated
|
||||
|
|
|
|||
|
|
@ -331,7 +331,16 @@ def search(
|
|||
legacy single-seed path (appropriate for warm starts from existing designs).
|
||||
|
||||
``n_workers=1`` (default) runs serially; ``n_workers > 1`` evaluates
|
||||
children in parallel using ``ProcessPoolExecutor``. The bootstrap batch
|
||||
children in parallel using ``ProcessPoolExecutor``.
|
||||
|
||||
**``n_workers`` is an ALGORITHM parameter, not just a speed knob**
|
||||
(homemaker-py-b8g, DESIGN.md §38.17). ``batch_n = min(n_workers, ...)``
|
||||
children are bred from ONE population snapshot before any of them is
|
||||
admitted, and the shared ``rng`` is consumed in a different pattern, so a
|
||||
run at ``n_workers=4`` explores a different trajectory from the same seed at
|
||||
``n_workers=1``. Each worker count is bit-reproducible on its own; results
|
||||
from DIFFERENT worker counts are not comparable, and an A/B whose arms differ
|
||||
in ``n_workers`` is comparing two algorithms, not two configurations. The bootstrap batch
|
||||
is fully parallel; the main loop generates ``n_workers`` children per
|
||||
iteration from the current population snapshot and evaluates them in
|
||||
parallel. Results are admitted in completion order (fastest first), so
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ from pathlib import Path
|
|||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from homemaker_layout import dom, driver, innerloop, solver
|
||||
from homemaker_layout import dom, driver, fitness, innerloop, solver
|
||||
|
||||
CORPUS = Path(__file__).parent.parent / "examples" / "programme-house"
|
||||
SEED_FILE = CORPUS / "c964435454c459f86c3ed9a5a7621132.dom"
|
||||
|
|
@ -506,23 +506,50 @@ def test_search_parallel_smoke():
|
|||
assert r.n_topologies >= 2 # at least the bootstrap individuals
|
||||
|
||||
|
||||
def test_search_parallel_is_reproducible():
|
||||
"""Two same-seed parallel runs must be byte-identical (homemaker-py-xcy).
|
||||
@pytest.mark.parametrize("workers", [2, 3, 4])
|
||||
def test_search_is_reproducible_at_a_fixed_worker_count(workers):
|
||||
"""Same seed + SAME worker count => byte-identical (homemaker-py-xcy/b8g).
|
||||
|
||||
``_run_batch`` used to admit futures in completion order (``as_completed``),
|
||||
which varies run-to-run; with the order-sensitive ``admit`` (n_evals accrual,
|
||||
first-of-tie wins ``best``) that made parallel searches non-reproducible.
|
||||
Admitting in submission order fixed it. Guard the invariant directly: same
|
||||
seed + same worker count ⇒ identical best (n_fails, fitness, signature) and
|
||||
identical improvement history."""
|
||||
Admitting in submission order fixed it.
|
||||
|
||||
Note the invariant is per worker count, and deliberately so. `n_workers` is
|
||||
an algorithm parameter: `batch_n = min(n_workers, ...)` children are bred
|
||||
from one population snapshot before any is admitted, so different worker
|
||||
counts explore different trajectories from the same seed (§38.17). This
|
||||
parametrises over several counts to check each is internally stable; it does
|
||||
NOT assert that they agree with each other, because they legitimately need
|
||||
not.
|
||||
"""
|
||||
def run():
|
||||
r = driver.search(dom.load(str(INIT_FILE)), CORPUS, budget=1200,
|
||||
pop_size=8, child_budget=80, seed=0, n_workers=3)
|
||||
pop_size=8, child_budget=80, seed=0, n_workers=workers)
|
||||
return (r.best.n_fails, r.best.fitness, r.best.sig, tuple(r.history))
|
||||
|
||||
a = run()
|
||||
b = run()
|
||||
assert a == b, "parallel search is not reproducible run-to-run"
|
||||
assert run() == run(), (
|
||||
f"search at n_workers={workers} is not reproducible run-to-run")
|
||||
|
||||
|
||||
def test_scoring_a_frozen_design_is_deterministic():
|
||||
"""No floating-point/BLAS nondeterminism in a single eval (homemaker-py-b8g).
|
||||
|
||||
b8g suspected "a single fitness eval on a fixed genome returning different
|
||||
fail counts across runs", plausibly BLAS threading. It does not: measured
|
||||
bit-identical over 20 in-process repeats and 8 processes with different
|
||||
PYTHONHASHSEED, and pinning OMP/OPENBLAS/MKL to one thread changes nothing.
|
||||
This guards the floor the reproducibility argument stands on.
|
||||
"""
|
||||
import copy
|
||||
|
||||
conf, cost = fitness.load_config(CORPUS)
|
||||
root = dom.load(str(INIT_FILE))
|
||||
results = {
|
||||
fitness.Fitness(conf, cost).score_with_fails(copy.deepcopy(root))
|
||||
for _ in range(8)
|
||||
}
|
||||
assert len(results) == 1, "scoring a frozen design is not deterministic"
|
||||
|
||||
|
||||
def _shared_best_result() -> driver.SearchResult:
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue