bruno/homemaker-layout
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

psk/§14: island model — null (best-of-N at equal budget wins)

Browse source 
Prime a population from N independent converged elites + crossover-heavy
migration phase, vs best-of-N at equal total budget. Island does NOT win:
harbor 68 vs control 67 (within parallel noise), maple 124 vs control 116
(decisive). Default-off child_probe hook on driver.search instruments the
deciding mechanism: area-matched crossover across independently-converged
elites rarely synthesizes (1/65 harbor, 3/63 maple beat the better parent,
max fail-drop 2-5), confirming the alignment hypothesis (non-canonical 9gp
encoding -> disruptive splice). Search-machinery null #3; residual stays
geometry/shape-bound. 233 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bruno Postle 2026-06-29 06:20:29 +01:00
parent abe9dcad6c
commit d627ee5fb2
4 changed files with 285 additions and 12 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

24
.beads/issues.jsonl
View file

File diff suppressed because one or more lines are too long

68
DESIGN.md
View file

@ -2045,3 +2045,71 @@ ON, `HOMEMAKER_LEAF_SHARING`) and `--leaf-share-factor N` (default 3,
Default-OFF parity holds: `overrides=None` leaves `load_config` byte-identical and
`_share_rooms` is never reached. Smoke-checked end-to-end on harbor-house (sharing
on 37 fails vs `--no-leaf-sharing` 95 at budget 160). 233 tests pass.
## 14. Island model: multi-run recombination (`homemaker-py-psk`) — DONE (null)
**Lever (user-proposed).** Perl Urb ran the search many times and kept the best,
because independent runs settle into different local minima. The Python tool is
deterministic per `--seed`, so the analog is an *island model with synchronous
migration*: run N independent seeds to convergence (Phase A), then PRIME a fresh
population with those N converged elites and run a second, crossover-heavy phase
(Phase B) to recombine basins. Distinct from §11.5 (`c4c.5`), which injected
**fresh** random/constructive seeds for raw diversity and landed null — here the
migrants are **fully-converged elites**, high-quality building blocks, so the
"diversity does not help" result does not directly refute it. The one untested
sub-mechanism: can crossover *stack* wins across independent basins (run A solved
cluster X, run B solved cluster Y, child inherits both)?
**Design (`experiments/run_island_ab.py`).** Three numbers per programme, all
`leaf_sharing` OFF so controls track the §12.2 baselines (maple 136 / harbor 74),
all on **equal actual eval budget** (the staged search has a hard ~`pop·child·2`
bootstrap floor, so we account `r.n_evals`, never the request):
- **`bestN@A`** — best-of-N over Phase A (the FREE reference; these N runs happen
anyway — the legitimate descendant of Urb's multi-run habit).
- **`island`** — Phase B result: a population primed from the N Phase-A elites via
the existing `seed_factory`+`bootstrap` path (no new representation), evolved at
`p_crossover=0.7`. Total budget = Phase A + migration.
- **`bestN@T`** — best-of-N over N independent runs at the *same total* per seed
(the "N+ longer independent runs" control). **THE BAR**: island must beat it.
A default-off `child_probe` hook (`driver.search`) instruments the deciding
mechanism: for every crossover child it records whether the spliced child beats
`max`/`min(parent fails)`. Parent fails are appended to the child lineage as
`|pf=a,b` (only when the probe is set) so the signal survives the
`ProcessPoolExecutor` pickle round-trip an `id(root)` key cannot.
**Result (N=4, master_seed 0, 28160 actual evals/arm, 4 workers):**
| programme | bestN@A | island | **bestN@T** | verdict | crossover beat-min-parent |
|-----------|--------:|-------:|------------:|---------|--------------------------:|
| harbor | 73 | 68 | **67** | loses by 1 (within noise) | 1 / 65 |
| maple | 134 | 124 | **116** | loses by 8 (decisive) | 3 / 63 |
**Verdict: NULL / negative.** The island model does **not** beat best-of-N at
equal total budget. On harbor it ties-to-loses inside the parallel noise band; on
maple it loses clearly (124 vs 116) — a single *longer* independent run reached
116 while the migration phase, given the same budget, stalled at 124. The migration
phase buys nothing a longer independent run does not.
**The mechanistic probe explains why (the deciding diagnostic).** Crossover across
independently-converged elites almost never synthesizes: of ~64 crossover children
only **1/65 (harbor) and 3/63 (maple)** beat the *better* parent, with a best
fail-drop of just 2 and 5. This confirms the issue's **alignment** hypothesis:
`operators.crossover` is *area-matched* subtree exchange, but two independently
evolved trees encode similar arrangements at different paths/areas (the encoding
is non-canonical — `9gp` closed negative), so the splice is mostly disruptive, not
combinatorial, and the inner loop re-solves ratios at the boundary (spliced quality
not preserved). The null is therefore **mechanistic, not budget**.
**Noise caveat (carry forward).** Phase A is unaffected by the probe, yet harbor
seed 2 scored 71 then 73 on byte-identical re-runs — parallel/BLAS
non-determinism, the same ±2-3 effect §12.4 flagged. Sub-±3 verdicts under
`n_workers>1` are noise; both arms here ran at the same worker count so the
*comparison* stays fair, and maple's −8 is safely outside the band.
This is the third search-machinery null after §11.4 (graded objective) and §11.5
(niching+restarts) / §12.3 (M3 + shape filter), against four construction/seed
wins (§11.6, §11.7, §12.2, §13.x). best-of-N at the Phase-A budget remains a free,
worthwhile habit; a dedicated migration phase is not worth its budget. The residual
stays geometry/shape-bound. NOT gated on canonical encoding (`9gp` closed); the
`child_probe` hook is kept default-off for reuse.

188
experiments/run_island_ab.py Normal file
View file

@ -0,0 +1,188 @@
#!/usr/bin/env python3
"""Island-model A/B (homemaker-py-psk, DESIGN.md §14).
Tests the user-proposed lever: the Perl Urb workflow ran the search many times
and kept the best because runs settled into different local minima. The Python
tool is deterministic per --seed, so the analog is to run N independent seeds,
then PRIME a fresh population with those N converged elites and run a second,
crossover-heavy migration phase (an island model with synchronous migration).
Three numbers per programme (all leaf_sharing OFF, so the controls reproduce the
§12.2 baselines maple 136.0 / harbor 74.0):
bestN@A best-of-N over Phase A (N runs at B_A each). The FREE reference
(these N runs happen anyway); the legitimate descendant of Urb's
multi-run habit.
island Phase B migration result: a fresh population primed from the N
Phase-A elites, evolved with high p_crossover. Total budget
T = N*B_A + B_B.
bestN@T best-of-N over N independent runs at T/N each (the "N+ longer
independent runs" control). Same TOTAL budget T as island. THE BAR:
island must beat this to count.
Mechanistic instrument (the key diagnostic, §14): a child_probe over Phase B
counts how many crossover children ever beat max(parent fails) / min(parent
fails). If crossover children are never net-positive, the null is mechanistic
(area-matched splice across non-canonical encodings is disruptive, cf. 9gp),
not a budget shortfall.
Usage:
URB_NO_OCCLUSION=1 python3 experiments/run_island_ab.py \
[programme_dir] [N] [B_A] [B_B] [master_seed] [workers] [out_dir]
Defaults: harbor-house, N=5, B_A=1500, B_B=5000, master_seed=0, workers=4.
"""
from __future__ import annotations
import copy
import sys
import time
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from homemaker_layout import dom, driver # noqa: E402
REPO = Path(__file__).resolve().parents[1]
def _staged(seed_root, programme_dir, budget, seed, workers):
return driver.search_staged(
seed_root, programme_dir, budget=budget, pop_size=16,
child_budget=80, seed_budget=300, stage1_frac=0.4, base_p=0.15,
p_crossover=0.2, seed=seed, n_workers=workers, leaf_sharing=False,
)
def main() -> int:
programme_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else (
REPO / "examples" / "harbor-house")
N = int(sys.argv[2]) if len(sys.argv) > 2 else 5
B_A = int(sys.argv[3]) if len(sys.argv) > 3 else 1500
B_B = int(sys.argv[4]) if len(sys.argv) > 4 else 5000
master_seed = int(sys.argv[5]) if len(sys.argv) > 5 else 0
workers = int(sys.argv[6]) if len(sys.argv) > 6 else 4
out_dir = Path(sys.argv[7]) if len(sys.argv) > 7 else (REPO / "scratch" / "island_ab")
out_dir.mkdir(parents=True, exist_ok=True)
seed_file = programme_dir / "init.dom"
if not seed_file.exists():
print(f"ERROR: no seed .dom at {seed_file}", file=sys.stderr)
return 1
seed_root = dom.load(str(seed_file))
print(f"programme : {programme_dir.name}")
print(f"N seeds : {N}")
print(f"B_A (phase A): {B_A}/seed requested")
print(f"B_B (migrate): {B_B} requested")
print(f"control : {N} runs sized to island's ACTUAL total evals")
print(f"master_seed : {master_seed} workers {workers} leaf_sharing OFF")
print(flush=True)
t_start = time.perf_counter()
# NB: staged search has a hard floor of ~pop*child_budget*2 evals (the
# two bootstrap stages), so a requested per-seed budget below that overshoots.
# We account for ACTUAL evals consumed (r.n_evals), never the request.
# --- Phase A: N independent converged elites ---------------------------
print("=== Phase A: N independent runs ===", flush=True)
elites = []
phaseA_fails = []
evA = 0
for i in range(N):
s = master_seed * 1000 + i
t0 = time.perf_counter()
r = _staged(seed_root, programme_dir, B_A, s, workers)
phaseA_fails.append(r.best.n_fails)
evA += r.n_evals
elites.append(copy.deepcopy(r.best.root))
print(f" seed {s}: {r.best.n_fails} fails ({r.best.fitness:.6g}), "
f"{r.n_evals} evals, {time.perf_counter() - t0:.0f}s", flush=True)
bestN_A = min(phaseA_fails)
print(f" -> bestN@A = {bestN_A} fails (pool {sorted(phaseA_fails)}), "
f"{evA} actual evals\n", flush=True)
# --- Phase B: island migration ----------------------------------------
print("=== Phase B: migration (prime pop from N elites, high crossover) ===",
flush=True)
counter = {"i": 0}
def island_factory(rng):
root = copy.deepcopy(elites[counter["i"] % len(elites)])
counter["i"] += 1
return root
# Instrument crossover children: did the spliced child beat its parents?
# The driver appends "|pf=a,b" (parent fail counts) to a crossover child's
# lineage when child_probe is set; this survives the worker pickle round-trip.
xstats = {"xover": 0, "beat_min": 0, "beat_max": 0, "best_drop": 0}
def child_probe(ind):
if ind.lineage.startswith("pruned/") or "|pf=" not in ind.lineage:
return
pa, pb = (int(x) for x in ind.lineage.split("|pf=")[1].split(","))
xstats["xover"] += 1
if ind.n_fails < min(pa, pb):
xstats["beat_min"] += 1
if ind.n_fails < max(pa, pb):
xstats["beat_max"] += 1
drop = max(pa, pb) - ind.n_fails
if drop > xstats["best_drop"]:
xstats["best_drop"] = drop
t0 = time.perf_counter()
r_island = driver.search(
seed_root, programme_dir, budget=B_B, pop_size=N, child_budget=80,
seed_budget=300, p_crossover=0.7, seed=master_seed, n_workers=workers,
leaf_sharing=False, bootstrap=True, seed_factory=island_factory,
child_probe=child_probe,
)
island_fails = r_island.best.n_fails
evB = r_island.n_evals
island_total_ev = evA + evB
dom.dump(r_island.best.root, str(out_dir / f"{programme_dir.name}_island_s{master_seed}.dom"))
print(f" island = {island_fails} fails ({r_island.best.fitness:.6g}), "
f"{evB} migration evals (island total {island_total_ev}), "
f"{time.perf_counter() - t0:.0f}s", flush=True)
print(f" crossover children: {xstats['xover']} evaluated, "
f"{xstats['beat_max']} beat max(parent), {xstats['beat_min']} beat "
f"min(parent), best fail-drop {xstats['best_drop']}\n", flush=True)
# --- Control: N longer independent runs at equal total -----------------
# Match the island's ACTUAL total evals, not the requested budget.
B_T = max(B_A, island_total_ev // N)
print(f"=== Control: best-of-N @ {B_T}/seed (~equal total {island_total_ev}) ===",
flush=True)
control_fails = []
evC = 0
for i in range(N):
s = master_seed * 1000 + 500 + i
t0 = time.perf_counter()
r = _staged(seed_root, programme_dir, B_T, s, workers)
control_fails.append(r.best.n_fails)
evC += r.n_evals
print(f" seed {s}: {r.best.n_fails} fails, {r.n_evals} evals, "
f"{time.perf_counter() - t0:.0f}s", flush=True)
bestN_T = min(control_fails)
print(f" -> bestN@T = {bestN_T} fails (pool {sorted(control_fails)}), "
f"{evC} actual evals\n", flush=True)
# --- Verdict ----------------------------------------------------------
print("=" * 64)
print(f"RESULT {programme_dir.name} (master_seed {master_seed})")
print(f" bestN@A (free ref, {evA} ev) : {bestN_A} fails")
print(f" island (Phase A+migration, {island_total_ev} ev) : {island_fails} fails")
print(f" bestN@T (control, {evC} ev) : {bestN_T} fails <- BAR")
verdict = ("ISLAND WINS" if island_fails < bestN_T
else "tie" if island_fails == bestN_T else "ISLAND LOSES")
print(f" verdict : island vs control = {island_fails} vs {bestN_T} -> {verdict}")
print(f" crossover net-positive: {xstats['beat_max']}/{xstats['xover']} "
f"beat max(parent); mechanism "
f"{'LIVE' if xstats['beat_max'] else 'DEAD (alignment null)'}")
print(f" wall: {time.perf_counter() - t_start:.0f}s")
print("=" * 64, flush=True)
return 0
if __name__ == "__main__":
sys.exit(main())

17
src/homemaker_layout/driver.py
View file

@ -185,6 +185,7 @@ def search(
rank_bonus_weight: float = 1.0,
seed_factory=None,
base_p: float = 1.0,
child_probe=None,
use_grade: bool = False,
niche_by_signature: bool = False,
restart_patience: int | None = None,
@ -339,6 +340,16 @@ def search(
pop: list[Individual] = []
# homemaker-py-psk (island model §14): optional per-child instrumentation
# hook, default off (no behaviour change). ``child_probe(ind)`` is called
# once per evaluated child. Used by the island-migration A/B to measure
# whether area-matched crossover across independently-converged elites EVER
# yields a child that beats max(parent fails) — distinguishing a mechanistic
# (alignment) null from a budget null. The crossover parents' fail counts are
# appended to the child's lineage as ``|pf=a,b`` (only when the probe is set),
# so the signal survives the ProcessPoolExecutor pickle round-trip that an
# id(root) key cannot (the worker returns a deserialised, distinct object).
# Set up optional process pool for parallel child evaluation.
_pool = None
if n_workers > 1:
@ -376,11 +387,15 @@ def search(
for f in futs:
ind, used = f.result()
n_evals += used
if child_probe is not None:
child_probe(ind)
admit(ind, pop)
else:
for t in full:
ind, used = _evaluate(*t)
n_evals += used
if child_probe is not None:
child_probe(ind)
admit(ind, pop)
# A fresh seed individual (used for the initial bootstrap and for §11.5
@ -465,6 +480,8 @@ def search(
if len(pop) >= 2 and rng.random() < p_crossover:
a, b = _tournament(pop, rng, _key), _tournament(pop, rng, _key)
child_root, _, desc = operators.crossover(a.root, b.root, rng)
if child_probe is not None:
desc = f"{desc}|pf={a.n_fails},{b.n_fails}"
ratios = {**b.ratios, **a.ratios} # primary parent wins
else:
parent = _tournament(pop, rng, _key)