diff --git a/.beads/issues.jsonl b/.beads/issues.jsonl index b315456..4754ed2 100644 --- a/.beads/issues.jsonl +++ b/.beads/issues.jsonl @@ -9,7 +9,7 @@ {"id":"homemaker-py-1p0","title":"Geometry inner loop: full-objective equal-offset ratio optimiser","description":"DESIGN.md §5.1, §7 Phase 1. Productionise experiments/optimize_fullfitness.py into homemaker: optimise(topology, x0=None) -\u003e (geometry, fitness). DOF = equal-offset division ratios of free branches (solver.free_branches, lowest-storey cut ownership), clipped to [eps, 1-eps]. Objective = full oracle fitness (never a proxy — §4.2 falsified). Must support warm-start x0 (§5.6) and a population/batch evaluation mode so each iteration scores via one batched oracle call (§4.6).","acceptance_criteria":"Reproduces or exceeds §4.5 gains (x1.24–x1.67, no new failures) on 2f45907, candidate-002, c964435; works as a library call on any corpus .dom","status":"closed","priority":1,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:58Z","created_by":"Bruno Postle","updated_at":"2026-06-12T08:46:31Z","started_at":"2026-06-12T00:14:19Z","closed_at":"2026-06-12T08:46:31Z","close_reason":"innerloop.optimise() lands: batched CMA-ES sigma ladder (0.05/0.15, IPOP popsize doubling, deterministic seeding) over equal-offset free-branch ratios vs full oracle fitness; warm-start x0 supported. Acceptance vs unprojected originals: x1.65/x1.66/x1.58 against bars x1.24/x1.67/x1.59, no new failures, 46 oracle calls vs NM's 200. Two near-bar results accepted as reproduced-within-noise (1% tol) — draw spread brackets the single-NM-draw bars; approved by Bruno 2026-06-12. Gotchas: equal-offset projection of legacy unequal cuts loses fitness/adds failures (midpoint projection used); pycma seed=0 means clock-seeded.","dependencies":[{"issue_id":"homemaker-py-1p0","depends_on_id":"homemaker-py-av5","type":"blocks","created_at":"2026-06-12T00:39:33Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":3,"comment_count":0} {"id":"homemaker-py-8cs","title":"Experiment: warm-vs-cold start of inner loop (Lamarckian inheritance)","description":"DESIGN.md §5.6, §4.6. Warm-starting a child topology's inner loop from the parent's optimised ratios is the main lever for cutting per-topology cost (~3 min/topology cold). Apply single topology mutations to optimised corpus designs, re-optimise warm (surviving cuts keep values, new cuts get heuristic defaults) vs cold, compare oracle-call counts to convergence at equal final fitness.","acceptance_criteria":"Speedup factor measured across \u003e=10 mutated topologies; decision recorded (expect order-of-magnitude; if \u003c2x, revisit §4.6 Phase-2 scoping)","notes":"Experiment script committed (experiments/warm_vs_cold.py, 1cc86c8) and machinery validated oracle-free; one mutated child scored through the oracle OK. Waiting on homemaker-py-gp2 reference run to finish, then execute under URB_NO_OCCLUSION=1 (3 parents x 400 evals + 12 children x 2 x 200 evals, ~1.5-2 h oracle time). Default budgets: parent 400, child 200; target = evals to 95% of best final.","status":"closed","priority":1,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:58Z","created_by":"Bruno Postle","updated_at":"2026-06-12T11:44:45Z","closed_at":"2026-06-12T11:44:45Z","close_reason":"Measured (URB_NO_OCCLUSION=1, parent budget 400, child 200, 12 single mutations across 3 designs): cold start reached 95% of warm final in 0/12 cases within budget — speedup unbounded at practical budgets; warm finals beat cold finals x1.2-x4 in 12/12; 6/12 warm starts were within 95% at 1 eval (near-neutral mutations). Decision: Lamarckian warm-starting is MANDATORY in the memetic driver (homemaker-py-b39), not an optimisation; cold starts produce strictly worse geometry at equal budget. Note: 2 undivides were exactly fitness-neutral (same-type merge == Merge_Divided equivalence) — locality datum for homemaker-py-nyb.","dependencies":[{"issue_id":"homemaker-py-8cs","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:34Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0} {"id":"homemaker-py-av5","title":"Batched oracle: score many .dom files per invocation","description":"oracle.py currently scores one .dom per urb-fitness.pl call (~1.65 s/dom). DESIGN.md §4.6: batching amortises Perl startup to ~0.99 s/dom and is required so population/batch optimisers can score a whole generation in one oracle call. Extend oracle.py with a batch API: write N .dom files, one perl invocation, parse N .score/.fails pairs. Keep the single-file path for compatibility.","acceptance_criteria":"Batch of 35 corpus files scores in one perl invocation; per-file results identical to single-file calls; measured s/dom reported","status":"closed","priority":1,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:56Z","created_by":"Bruno Postle","updated_at":"2026-06-12T00:14:06Z","started_at":"2026-06-11T23:50:40Z","closed_at":"2026-06-12T00:14:06Z","close_reason":"score_batch() lands in oracle.py; 35-file corpus parity verified single-vs-batch (1e-12 rel fitness, exact fail sets); 0.98 s/dom batched vs 1.27 single, x1.30","dependency_count":0,"dependent_count":1,"comment_count":0} -{"id":"homemaker-py-ccw","title":"Scaled topology search on native fitness","description":"DESIGN.md §7 Phase 3 closing step. Once native fitness passes corpus parity, re-run the Phase-2 memetic search at real scale (population/generations comparable to urb-evolve) on the native objective. This is the first point where the §1 scaling question gets a real answer.","acceptance_criteria":"Full-scale run on programme-house beats both urb-evolve and the small-scale Phase-2 result; larger programme attempted","status":"open","priority":2,"issue_type":"feature","owner":"bruno@postle.net","created_at":"2026-06-11T23:38:59Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:38:59Z","dependencies":[{"issue_id":"homemaker-py-ccw","depends_on_id":"homemaker-py-uxz","type":"blocks","created_at":"2026-06-12T00:39:44Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-ccw","depends_on_id":"homemaker-py-way","type":"blocks","created_at":"2026-06-12T00:39:45Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0} +{"id":"homemaker-py-ccw","title":"Scaled topology search on native fitness","description":"DESIGN.md §7 Phase 3 closing step. Once native fitness passes corpus parity, re-run the Phase-2 memetic search at real scale (population/generations comparable to urb-evolve) on the native objective. This is the first point where the §1 scaling question gets a real answer.","acceptance_criteria":"Full-scale run on programme-house beats both urb-evolve and the small-scale Phase-2 result; larger programme attempted","status":"closed","priority":2,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:38:59Z","created_by":"Bruno Postle","updated_at":"2026-06-13T21:09:28Z","started_at":"2026-06-13T20:49:27Z","closed_at":"2026-06-13T21:09:28Z","close_reason":"Phase 3 scaled search complete. programme-house: 1.04e-02 (2 fails) at budget 20000 beats Phase-2 (7.65e-03, 1.36x) and urb-evolve p128 (4.00e-03, 2.60x). harbor-house 16-room programme: 3.73e-18 (49 fails) at budget 10000 — oracle-impossible, native fitness makes it feasible. DESIGN.md updated with gate results.","dependencies":[{"issue_id":"homemaker-py-ccw","depends_on_id":"homemaker-py-uxz","type":"blocks","created_at":"2026-06-12T00:39:44Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-ccw","depends_on_id":"homemaker-py-way","type":"blocks","created_at":"2026-06-12T00:39:45Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0} {"id":"homemaker-py-way","title":"Benchmark: memetic loop vs urb-evolve at equal oracle-call budget (Phase 2 gate)","description":"DESIGN.md §7 Phase 2 gate. Compare against urb-evolve from the same seeds/programmes at equal oracle-evaluation budget — NOT generations (urb-evolve has diversity injection/culling baked in, so generations are not comparable). Go/no-go: memetic loop must beat equal-budget urb-evolve. Scaling up waits for native fitness.","acceptance_criteria":"Best-fitness and failure-count comparison at \u003e=2 budgets, \u003e=3 seeds; go/no-go decision recorded in DESIGN.md","status":"closed","priority":2,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:28Z","created_by":"Bruno Postle","updated_at":"2026-06-13T08:55:03Z","started_at":"2026-06-12T21:13:20Z","closed_at":"2026-06-13T08:55:03Z","close_reason":"Phase-2 gate run (benchmark_vs_urbevolve.py, 2026-06-13, 2000 evals, URB_NO_OCCLUSION=1): 2/3 seeds → REVIEW. Memetic beats urb-evolve by 1.91x/1.63x on seeded designs; blank-slate init.dom stalls at 18 fails vs urb-evolve's 6 (random-pop init advantage). Fix: patterns.config was missing from re-score cwd (run_search.py), giving false near-zero finals in first run. Results recorded in DESIGN.md §7 Phase 2 gate.","dependencies":[{"issue_id":"homemaker-py-way","depends_on_id":"homemaker-py-b39","type":"blocks","created_at":"2026-06-12T00:39:39Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-way","depends_on_id":"homemaker-py-gp2","type":"blocks","created_at":"2026-06-12T08:27:45Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0} {"id":"homemaker-py-b39","title":"Memetic search driver, small-scale (budgets in oracle evaluations)","description":"DESIGN.md §5, §7 Phase 2, §4.6 arithmetic. Memetic EA/SA over topology genomes wrapping the geometry inner loop (warm-started per §5.6); score = best full fitness over the inner loop. Explicitly small-scale on the batched oracle: tens of topologies, budget accounted in oracle evaluations, not generations. Population evaluation batched into single oracle calls.","acceptance_criteria":"End-to-end run on programme-house completes within a stated oracle-call budget and logs evaluations; produces valid .dom output","status":"closed","priority":2,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:27Z","created_by":"Bruno Postle","updated_at":"2026-06-12T21:10:05Z","started_at":"2026-06-12T13:13:53Z","closed_at":"2026-06-12T21:10:05Z","close_reason":"driver.search() lands: steady-state memetic GA, tournament selection, operators + crossover, warm-started inner loop (Lamarckian write-back), budgets accounted in oracle evaluations. Acceptance run (URB_NO_OCCLUSION=1, budget 2000, seed c964435): 2010 evals / 23 topologies, best 0.00765/2 fails via crossover = x1.14 over the geometry-only optimum; output .dom re-scores standalone at exactly the recorded fitness. En route: found + fixed Urb ratio_o/ratio_type first-match nondeterminism (class-sum patch, 35/35 corpus parity) after the search reward-hacked it; operators now emit canonical uppercase generics (Bruno's correction: C=circulation, Is_Covered is a predicate).","dependencies":[{"issue_id":"homemaker-py-b39","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:37Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-b39","depends_on_id":"homemaker-py-nyb","type":"blocks","created_at":"2026-06-12T00:39:38Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0} {"id":"homemaker-py-nyb","title":"High-locality topology operators (mutation + subtree crossover)","description":"DESIGN.md §5, §7 Phase 2, §8.4. Mutation moves: divide/undivide leaf, swap children, rotate cut, retype leaf, per-floor delta edits, storey add/delete (cf. Urb Mutate.pm — but geometry sliding belongs to the inner loop, not the operator set). Crossover: area-matched subtree exchange (a subtree = a contiguous region, so crossover is meaningful — Crossover.pm). Operators must be high-locality: small genome change =\u003e small phenotype change, so warm-started inner loops stay cheap.","acceptance_criteria":"Each operator produces valid genomes (oracle scores them without error); locality measured (mean fitness/geometry perturbation per operator)","status":"closed","priority":2,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:27Z","created_by":"Bruno Postle","updated_at":"2026-06-12T13:07:37Z","started_at":"2026-06-12T12:54:23Z","closed_at":"2026-06-12T13:07:37Z","close_reason":"operators.py lands: 7 mutations + area-matched crossover, valid-by-construction via genome.encode repair. 115/115 oracle-valid children; locality measured: geom-pert 0.07-0.33 per op, fitness-pert 0.68-0.99 (0.5^n cliff flags raw moves — warm restart + penalty reshaping confirmed load-bearing). Also fixed dom._link stale below-links on structural mutation.","dependencies":[{"issue_id":"homemaker-py-nyb","depends_on_id":"homemaker-py-k2g","type":"blocks","created_at":"2026-06-12T00:39:36Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0} diff --git a/DESIGN.md b/DESIGN.md index 2983a55..eb937e8 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -385,6 +385,40 @@ outside ratios, min internal area.** Source of truth: set against the *flagged* oracle across the 35-file corpus (float tolerance, identical failure sets). Swap behind the same interface; retire the oracle. Then re-run Phase 2 at scale. + + **Gate result (homemaker-py-ccw, 2026-06-13, `URB_NO_OCCLUSION=1`, budget 20000):** + `experiments/run_search_scaled.py`; native fitness only, no oracle. pop_size=16, + child_budget=80, seed_budget=300. 71.8 evals/s, 279.8s elapsed. + + programme-house, seed c964435 vs Phase-2 and urb-evolve references: + + | seed | system | budget | best | fails | + |------|--------|--------|------|-------| + | c964435 | memetic Phase-2 (oracle) | 2000 | 7.65e-03 | 2 | + | c964435 | urb-evolve p16 | — | 4.00e-03 | 3 | + | c964435 | urb-evolve p128 | — | 4.00e-03 | 3 | + | c964435 | **memetic Phase-3 (native)** | **20000** | **1.04e-02** | **2** | + + **Verdict: PASS.** + - Best 1.04e-02 beats Phase-2 oracle run (7.65e-03) by **1.36×** and urb-evolve p128 + (4.00e-03) by **2.60×**; both at 2 fails. + - Winning topology found at eval 10357 via `rotate 1/ll` — unreachable within the + Phase-2 budget of 2000. + - Population diverse: 16 members, all at 2 fails (top 15), range 5.99e-03–1.04e-02. + - Throughput 71.8 evals/s vs ~0.5 evals/s for the batched oracle (≈140× speedup). + - harbor-house (16 rooms, oracle-impossible): run attempted, results below. + + harbor-house (16 rooms, budget 10000): seed `2b51b05` (best corpus design, 48 fails raw): + + | system | budget | best | fails | evals/s | + |--------|--------|------|-------|---------| + | oracle | — | *impossible* | — | — | + | memetic Phase-3 (native) | 10000 | 3.73e-18 | 49 | 15.8 | + + Search found 3.73e-18 vs seed inner-loop baseline 8.73e-19 (4.3× lift). 638 topologies + in 633s. 49-fail landscape: still many fails, but topology search is finding structure + (best 3 population members all at 49 fails). The 16-room programme is qualitatively + beyond the oracle's capability — this run is only possible with native fitness. - **Phase 4 — penalty reshaping**: replace `0.5^n` with additive/soft, lexicographic, or multi-objective (easier once fitness is native), while preserving the inner loop's no-new-failures protection (§5.4) and the diff --git a/experiments/run_search_scaled.py b/experiments/run_search_scaled.py new file mode 100644 index 0000000..b6b479e --- /dev/null +++ b/experiments/run_search_scaled.py @@ -0,0 +1,119 @@ +#!/usr/bin/env python3 +"""Scaled topology search on native fitness (homemaker-py-ccw, Phase 3). + +Runs driver.search at larger budget using the native Python fitness exclusively +— no Perl oracle, no wall-clock bottleneck from Perl startup. + +This unlocks two things the Phase-2 oracle-bounded run could not do: + 1. Larger evaluation budgets on programme-house (beat Phase-2 best). + 2. Programmes too large for the oracle (e.g. harbor-house, 16 rooms). + +Usage: + URB_NO_OCCLUSION=1 python3 experiments/run_search_scaled.py [programme_dir] [budget] [rng_seed] [seed.dom] [out.dom] + +Defaults: programme-house, budget=20000, rng_seed=0, best corpus seed. + +Phase-2 reference bars (URB_NO_OCCLUSION=1, budget=2000, native fitness): + c964435 seed → 7.65e-03 (2 fails) beats urb-evolve p128 4.00e-03 + 2f45907 seed → 2.13e-02 (2 fails) beats urb-evolve p128 1.30e-02 +""" + +from __future__ import annotations + +import math +import sys +import time +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src")) +from homemaker import dom, driver, fitness, innerloop # noqa: E402 + +URB_EX = Path("/home/bruno/src/urb/examples") +PH_DIR = URB_EX / "programme-house" +PH_SEED = PH_DIR / "c964435454c459f86c3ed9a5a7621132.dom" + + +def _native_score(root: dom.Node, programme_dir: Path) -> tuple[float, int]: + """Re-score a design with native fitness (no oracle dependency).""" + import copy + + conf, cost = fitness.load_config(programme_dir) + fit = fitness.Fitness(conf, cost) + score, fails = fit.score_with_fails(copy.deepcopy(root)) + return score, len(fails) + + +def main() -> int: + programme_dir = Path(sys.argv[1]) if len(sys.argv) > 1 else PH_DIR + budget = int(sys.argv[2]) if len(sys.argv) > 2 else 20000 + rng_seed = int(sys.argv[3]) if len(sys.argv) > 3 else 0 + seed_file = Path(sys.argv[4]) if len(sys.argv) > 4 else None + out = Path(sys.argv[5]) if len(sys.argv) > 5 else ( + Path(__file__).resolve().parents[1] / "scratch" / "scaled_best.dom" + ) + + if seed_file is None: + # pick the default seed for known programmes, else fall back to init.dom + if programme_dir.name == "programme-house": + seed_file = PH_SEED + else: + seed_file = programme_dir / "init.dom" + if not seed_file.exists(): + candidates = sorted(programme_dir.glob("*.dom")) + seed_file = candidates[0] if candidates else None + if seed_file is None: + print(f"ERROR: no .dom seed found in {programme_dir}", file=sys.stderr) + return 1 + + print(f"programme : {programme_dir.name}") + print(f"seed : {seed_file.name}") + print(f"budget : {budget} native evals") + print(f"rng seed : {rng_seed}") + print(flush=True) + + seed_root = dom.load(str(seed_file)) + t0 = time.perf_counter() + + r = driver.search( + seed_root, + programme_dir, + budget=budget, + pop_size=16, # up from 8 — more diversity at scale + child_budget=80, + seed_budget=300, # up from 200 — thorough seed optimisation + p_crossover=0.2, + seed=rng_seed, + log=lambda m: print(m, flush=True), + # urb_root not needed: use_native=True is the default + ) + + elapsed = time.perf_counter() - t0 + evals_per_sec = r.n_evals / elapsed + + print(f"\n--- done ---") + print(f"elapsed : {elapsed:.1f}s ({evals_per_sec:.1f} evals/s)") + print(f"evals : {r.n_evals} across {r.n_topologies} topologies") + print(f"best : {r.best.fitness:.6g} ({r.best.n_fails} fails) via {r.best.lineage}") + print("population: " + ", ".join( + f"{p.fitness:.4g}/{p.n_fails}f" for p in r.population + )) + + if r.history: + print("\nimprovement history:") + for ev, fit_val, lin in r.history: + print(f" [{ev:6d}] {fit_val:.6g} ({lin})") + + out.parent.mkdir(parents=True, exist_ok=True) + dom.dump(r.best.root, str(out)) + + # native re-score as ground truth (oracle retired in Phase 3) + rs, rf = _native_score(r.best.root, programme_dir) + ok = math.isclose(rs, r.best.fitness, rel_tol=1e-9) + print(f"\n{out.name} re-scored (native): {rs:.6g} ({rf} fails) " + f"→ {'OK' if ok else 'MISMATCH'}") + + return 0 if ok else 1 + + +if __name__ == "__main__": + sys.exit(main())