Phase-2 gate benchmark: memetic loop vs urb-evolve at equal eval budgets
experiments/benchmark_vs_urbevolve.py (homemaker-py-way): 3 seed designs (init from scratch, c964435 weak, 2f45907 strong) x 2000-eval runs, the 1000-eval tier read from each run's best-so-far log; urb-evolve gets two population sizes (default 128 = ~16 generations at this budget, and 16 = ~130) and credit for its better one. Counts via the MAX_EVALS counter patch in urb-evolve.pl (committed in the urb repo); both systems under URB_NO_OCCLUSION=1; all finals re-scored through urb-fitness.pl as the common deterministic yardstick. run_search.py generalised to (budget, rng-seed, seed.dom, out.dom); innerloop.optimise now handles 0-DOF topologies (an undivided plot like init.dom scores once instead of crashing CMA). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
6459949e41
commit
e2b3e20070
4 changed files with 152 additions and 12 deletions
|
|
@ -7,17 +7,17 @@
|
|||
{"id":"homemaker-py-8cs","title":"Experiment: warm-vs-cold start of inner loop (Lamarckian inheritance)","description":"DESIGN.md §5.6, §4.6. Warm-starting a child topology's inner loop from the parent's optimised ratios is the main lever for cutting per-topology cost (~3 min/topology cold). Apply single topology mutations to optimised corpus designs, re-optimise warm (surviving cuts keep values, new cuts get heuristic defaults) vs cold, compare oracle-call counts to convergence at equal final fitness.","acceptance_criteria":"Speedup factor measured across \u003e=10 mutated topologies; decision recorded (expect order-of-magnitude; if \u003c2x, revisit §4.6 Phase-2 scoping)","notes":"Experiment script committed (experiments/warm_vs_cold.py, 1cc86c8) and machinery validated oracle-free; one mutated child scored through the oracle OK. Waiting on homemaker-py-gp2 reference run to finish, then execute under URB_NO_OCCLUSION=1 (3 parents x 400 evals + 12 children x 2 x 200 evals, ~1.5-2 h oracle time). Default budgets: parent 400, child 200; target = evals to 95% of best final.","status":"closed","priority":1,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:58Z","created_by":"Bruno Postle","updated_at":"2026-06-12T11:44:45Z","closed_at":"2026-06-12T11:44:45Z","close_reason":"Measured (URB_NO_OCCLUSION=1, parent budget 400, child 200, 12 single mutations across 3 designs): cold start reached 95% of warm final in 0/12 cases within budget — speedup unbounded at practical budgets; warm finals beat cold finals x1.2-x4 in 12/12; 6/12 warm starts were within 95% at 1 eval (near-neutral mutations). Decision: Lamarckian warm-starting is MANDATORY in the memetic driver (homemaker-py-b39), not an optimisation; cold starts produce strictly worse geometry at equal budget. Note: 2 undivides were exactly fitness-neutral (same-type merge == Merge_Divided equivalence) — locality datum for homemaker-py-nyb.","dependencies":[{"issue_id":"homemaker-py-8cs","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:34Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
|
||||
{"id":"homemaker-py-av5","title":"Batched oracle: score many .dom files per invocation","description":"oracle.py currently scores one .dom per urb-fitness.pl call (~1.65 s/dom). DESIGN.md §4.6: batching amortises Perl startup to ~0.99 s/dom and is required so population/batch optimisers can score a whole generation in one oracle call. Extend oracle.py with a batch API: write N .dom files, one perl invocation, parse N .score/.fails pairs. Keep the single-file path for compatibility.","acceptance_criteria":"Batch of 35 corpus files scores in one perl invocation; per-file results identical to single-file calls; measured s/dom reported","status":"closed","priority":1,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:56Z","created_by":"Bruno Postle","updated_at":"2026-06-12T00:14:06Z","started_at":"2026-06-11T23:50:40Z","closed_at":"2026-06-12T00:14:06Z","close_reason":"score_batch() lands in oracle.py; 35-file corpus parity verified single-vs-batch (1e-12 rel fitness, exact fail sets); 0.98 s/dom batched vs 1.27 single, x1.30","dependency_count":0,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-ccw","title":"Scaled topology search on native fitness","description":"DESIGN.md §7 Phase 3 closing step. Once native fitness passes corpus parity, re-run the Phase-2 memetic search at real scale (population/generations comparable to urb-evolve) on the native objective. This is the first point where the §1 scaling question gets a real answer.","acceptance_criteria":"Full-scale run on programme-house beats both urb-evolve and the small-scale Phase-2 result; larger programme attempted","status":"open","priority":2,"issue_type":"feature","owner":"bruno@postle.net","created_at":"2026-06-11T23:38:59Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:38:59Z","dependencies":[{"issue_id":"homemaker-py-ccw","depends_on_id":"homemaker-py-uxz","type":"blocks","created_at":"2026-06-12T00:39:44Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-ccw","depends_on_id":"homemaker-py-way","type":"blocks","created_at":"2026-06-12T00:39:45Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-way","title":"Benchmark: memetic loop vs urb-evolve at equal oracle-call budget (Phase 2 gate)","description":"DESIGN.md §7 Phase 2 gate. Compare against urb-evolve from the same seeds/programmes at equal oracle-evaluation budget — NOT generations (urb-evolve has diversity injection/culling baked in, so generations are not comparable). Go/no-go: memetic loop must beat equal-budget urb-evolve. Scaling up waits for native fitness.","acceptance_criteria":"Best-fitness and failure-count comparison at \u003e=2 budgets, \u003e=3 seeds; go/no-go decision recorded in DESIGN.md","status":"open","priority":2,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:28Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:37:28Z","dependencies":[{"issue_id":"homemaker-py-way","depends_on_id":"homemaker-py-b39","type":"blocks","created_at":"2026-06-12T00:39:39Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-way","depends_on_id":"homemaker-py-gp2","type":"blocks","created_at":"2026-06-12T08:27:45Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-way","title":"Benchmark: memetic loop vs urb-evolve at equal oracle-call budget (Phase 2 gate)","description":"DESIGN.md §7 Phase 2 gate. Compare against urb-evolve from the same seeds/programmes at equal oracle-evaluation budget — NOT generations (urb-evolve has diversity injection/culling baked in, so generations are not comparable). Go/no-go: memetic loop must beat equal-budget urb-evolve. Scaling up waits for native fitness.","acceptance_criteria":"Best-fitness and failure-count comparison at \u003e=2 budgets, \u003e=3 seeds; go/no-go decision recorded in DESIGN.md","status":"in_progress","priority":2,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:28Z","created_by":"Bruno Postle","updated_at":"2026-06-12T21:13:20Z","started_at":"2026-06-12T21:13:20Z","dependencies":[{"issue_id":"homemaker-py-way","depends_on_id":"homemaker-py-b39","type":"blocks","created_at":"2026-06-12T00:39:39Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-way","depends_on_id":"homemaker-py-gp2","type":"blocks","created_at":"2026-06-12T08:27:45Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-b39","title":"Memetic search driver, small-scale (budgets in oracle evaluations)","description":"DESIGN.md §5, §7 Phase 2, §4.6 arithmetic. Memetic EA/SA over topology genomes wrapping the geometry inner loop (warm-started per §5.6); score = best full fitness over the inner loop. Explicitly small-scale on the batched oracle: tens of topologies, budget accounted in oracle evaluations, not generations. Population evaluation batched into single oracle calls.","acceptance_criteria":"End-to-end run on programme-house completes within a stated oracle-call budget and logs evaluations; produces valid .dom output","status":"closed","priority":2,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:27Z","created_by":"Bruno Postle","updated_at":"2026-06-12T21:10:05Z","started_at":"2026-06-12T13:13:53Z","closed_at":"2026-06-12T21:10:05Z","close_reason":"driver.search() lands: steady-state memetic GA, tournament selection, operators + crossover, warm-started inner loop (Lamarckian write-back), budgets accounted in oracle evaluations. Acceptance run (URB_NO_OCCLUSION=1, budget 2000, seed c964435): 2010 evals / 23 topologies, best 0.00765/2 fails via crossover = x1.14 over the geometry-only optimum; output .dom re-scores standalone at exactly the recorded fitness. En route: found + fixed Urb ratio_o/ratio_type first-match nondeterminism (class-sum patch, 35/35 corpus parity) after the search reward-hacked it; operators now emit canonical uppercase generics (Bruno's correction: C=circulation, Is_Covered is a predicate).","dependencies":[{"issue_id":"homemaker-py-b39","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:37Z","created_by":"Bruno Postle","metadata":"{}"},{"issue_id":"homemaker-py-b39","depends_on_id":"homemaker-py-nyb","type":"blocks","created_at":"2026-06-12T00:39:38Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":2,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-nyb","title":"High-locality topology operators (mutation + subtree crossover)","description":"DESIGN.md §5, §7 Phase 2, §8.4. Mutation moves: divide/undivide leaf, swap children, rotate cut, retype leaf, per-floor delta edits, storey add/delete (cf. Urb Mutate.pm — but geometry sliding belongs to the inner loop, not the operator set). Crossover: area-matched subtree exchange (a subtree = a contiguous region, so crossover is meaningful — Crossover.pm). Operators must be high-locality: small genome change =\u003e small phenotype change, so warm-started inner loops stay cheap.","acceptance_criteria":"Each operator produces valid genomes (oracle scores them without error); locality measured (mean fitness/geometry perturbation per operator)","status":"closed","priority":2,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:27Z","created_by":"Bruno Postle","updated_at":"2026-06-12T13:07:37Z","started_at":"2026-06-12T12:54:23Z","closed_at":"2026-06-12T13:07:37Z","close_reason":"operators.py lands: 7 mutations + area-matched crossover, valid-by-construction via genome.encode repair. 115/115 oracle-valid children; locality measured: geom-pert 0.07-0.33 per op, fitness-pert 0.68-0.99 (0.5^n cliff flags raw moves — warm restart + penalty reshaping confirmed load-bearing). Also fixed dom._link stale below-links on structural mutation.","dependencies":[{"issue_id":"homemaker-py-nyb","depends_on_id":"homemaker-py-k2g","type":"blocks","created_at":"2026-06-12T00:39:36Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-k2g","title":"Topology genome: base-floor tree + per-floor deltas + type assignment","description":"DESIGN.md §5.2, §7 Phase 2. Genome = base-floor slicing topology (primary) + per-leaf type assignment + per-floor divide/undivide deltas (Below-inheritance as regulariser; cut owned by lowest storey where its path is divided — §10). Must round-trip to/from dom.py Node trees so the oracle and inner loop consume it directly. Includes storey count and per-floor type overrides.","acceptance_criteria":"Genome \u003c-\u003e .dom round-trip on all 35 corpus files preserves fitness; multi-storey wall stacking preserved","status":"closed","priority":2,"issue_type":"feature","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:37:26Z","created_by":"Bruno Postle","updated_at":"2026-06-12T12:52:34Z","started_at":"2026-06-12T10:55:21Z","closed_at":"2026-06-12T12:52:34Z","close_reason":"genome.py encode/decode lands. 35/35 oracle fitness parity after round-trip (flag-on); genome fixed-point + owned-projection tests. Dead-field discovery: corpus upper storeys carry drifted dead divisions (97) and rotations (187) — canonicalised by decode, validated fitness-neutral.","dependency_count":0,"dependent_count":1,"comment_count":0}
|
||||
{"id":"homemaker-py-d0s","title":"Experiment: inner-loop optimiser bake-off at equal oracle budgets","description":"DESIGN.md §7 Phase 1, §8.3. DOF is only ~rooms-1 (6–7 on corpus). Compare Nelder-Mead vs CMA-ES vs batched multi-start pattern search at equal oracle-call budgets, measuring fitness gained per oracle call and wall-clock (batch-friendliness matters — §4.6). Measure, don't commit blind.","acceptance_criteria":"Table of fitness-per-budget across \u003e=3 candidates; one optimiser chosen and recorded in DESIGN.md","status":"open","priority":2,"issue_type":"task","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:59Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:36:59Z","dependencies":[{"issue_id":"homemaker-py-d0s","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:35Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
|
||||
{"id":"homemaker-py-d0s","title":"Experiment: inner-loop optimiser bake-off at equal oracle budgets","description":"DESIGN.md §7 Phase 1, §8.3. DOF is only ~rooms-1 (6–7 on corpus). Compare Nelder-Mead vs CMA-ES vs batched multi-start pattern search at equal oracle-call budgets, measuring fitness gained per oracle call and wall-clock (batch-friendliness matters — §4.6). Measure, don't commit blind.","acceptance_criteria":"Table of fitness-per-budget across \u003e=3 candidates; one optimiser chosen and recorded in DESIGN.md","status":"in_progress","priority":2,"issue_type":"task","assignee":"Bruno Postle","owner":"bruno@postle.net","created_at":"2026-06-11T23:36:59Z","created_by":"Bruno Postle","updated_at":"2026-06-12T21:22:15Z","started_at":"2026-06-12T21:22:15Z","dependencies":[{"issue_id":"homemaker-py-d0s","depends_on_id":"homemaker-py-1p0","type":"blocks","created_at":"2026-06-12T00:39:35Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
|
||||
{"id":"homemaker-py-can","title":"Programme width defaults: t3 contradiction (impossible width_inside default)","description":"DESIGN.md §8.2, confirmed in source. t3 (3 m2 WC) has no width spec so inherits width_inside [4.0, 1.0] (Fitness/Base.pm:60) — geometrically impossible; designs 'pass' only by failing size instead. Fix AFTER faithful-port validation (port-faithfully-first policy, §8.1): a sane width default scaled to area (e.g. sqrt(area/proportion)) or per-room widths in patterns.config. Applies to native fitness; optionally upstream to Urb.","acceptance_criteria":"No programme space has a default width incompatible with its target area; corpus re-scored and effect documented","status":"open","priority":3,"issue_type":"bug","owner":"bruno@postle.net","created_at":"2026-06-11T23:39:01Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:39:01Z","dependencies":[{"issue_id":"homemaker-py-can","depends_on_id":"homemaker-py-uxz","type":"blocks","created_at":"2026-06-12T00:39:47Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
|
||||
{"id":"homemaker-py-yg5","title":"Penalty reshaping: replace 0.5^n while preserving inner-loop protection","description":"DESIGN.md §4.7, §5.4, §7 Phase 4, §8.5. The 0.5^n cliff gives the outer search no gradient and rewards flag-count over geometry, but it also PROTECTS the inner loop from trading into new failures (§4.5). One fitness shape cannot naively be both soft outside and cliff-protected inside. Candidates: cliff-inside-inner-loop only, lexicographic (failure count first, score second), additive/soft, multi-objective Pareto. Must preserve the missing-space failure hierarchy (worse to drop a room than to have a poor one). Measure landscape + search outcomes; this helps Urb today too.","acceptance_criteria":"Chosen scheme documented with measurements: search improves while inner loop still never trades into new failures","status":"open","priority":3,"issue_type":"feature","owner":"bruno@postle.net","created_at":"2026-06-11T23:39:00Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:39:00Z","dependencies":[{"issue_id":"homemaker-py-yg5","depends_on_id":"homemaker-py-uxz","type":"blocks","created_at":"2026-06-12T00:39:46Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
|
||||
{"id":"homemaker-py-9gp","title":"Canonical slicing encoding (normalized Polish expression) + shape feasibility","description":"DESIGN.md §5.5, §7 Phase 5. Representation upgrade once core lands: normalized Polish expression / skewed slicing tree (Wong–Liu) for redundancy-free, high-locality topology moves (M1/M2/M3); bottom-up shape-feasibility checks to prune infeasible topologies before the inner loop. Goal: scale to larger programmes. Excluded representations stay excluded (§2): no sequence-pair/B*-tree (non-slicing).","acceptance_criteria":"Encoding round-trips with the genome; M1/M2/M3 moves implemented; measured search improvement on a larger-than-house programme","status":"open","priority":4,"issue_type":"feature","owner":"bruno@postle.net","created_at":"2026-06-11T23:39:02Z","created_by":"Bruno Postle","updated_at":"2026-06-11T23:39:02Z","dependencies":[{"issue_id":"homemaker-py-9gp","depends_on_id":"homemaker-py-ccw","type":"blocks","created_at":"2026-06-12T00:39:48Z","created_by":"Bruno Postle","metadata":"{}"}],"dependency_count":1,"dependent_count":0,"comment_count":0}
|
||||
{"id":"homemaker-py-2g5","title":"Rebuild occlusion/daylight/sun subsystem in Python (post-Phase-5, after optimisation fully native)","description":"DESIGN.md §6 port scope — a whole subsystem, not a term. quality_daylight (Leaf.pm:281-296) needs Urb::Misc::Sun + Urb::Field::Occlusion (+CIESky); quality_uncrinkliness also takes the occlusion object. Indoor spaces return 1 for daylight; cost is outdoor spaces + crinkliness. Port Sun_horizontal (262980-minute normalisation) and the occlusion wall set from Dom-\u003eWalls.","acceptance_criteria":"Daylight and crinkliness factors match Perl (float tolerance) across the corpus, including multi-storey cases","notes":"Re-scoped 2026-06-12: occlusion disabled in the Urb oracle instead of ported (see homemaker-py-gp2). Native fitness ships with simple crinkliness (illumination factor = 1, in homemaker-py-gnw). This issue is now the eventual Python occlusion rebuild, only after optimisation works entirely in Python. Restores outdoor-daylight and shaded-wall selection pressure.","status":"open","priority":4,"issue_type":"feature","owner":"bruno@postle.net","created_at":"2026-06-11T23:38:25Z","created_by":"Bruno Postle","updated_at":"2026-06-12T07:27:48Z","dependency_count":0,"dependent_count":0,"comment_count":0}
|
||||
{"_type":"memory","key":"strategy-decision-2026-06-12-bruno-occlusion-daylight","value":"Strategy decision 2026-06-12 (Bruno): occlusion/daylight is ORTHOGONAL to building a scalable optimiser. Disable it in Urb (env flag, homemaker-py-gp2) rather than port it; native fitness uses simple crinkliness (illumination factor = 1); rebuild occlusion in Python only after optimisation is fully native (homemaker-py-2g5, now P4). Consequence: all scores change when the flag flips — re-baseline corpus/.score, DESIGN \\$4.5 gains, gate bars at one clean boundary AFTER homemaker-py-1p0 closes; Phase-2 urb-evolve benchmark must run with the same flag."}
|
||||
{"_type":"memory","key":"user-preference-bruno-this-is-a-fedora-system","value":"User preference (Bruno): this is a Fedora system — NEVER install Python packages via pip without asking first; always ask whether to install the rpm via dnf (e.g. python3-cma) before considering pip. Applies to any dependency additions."}
|
||||
{"_type":"memory","key":"urb-fitness-bug-found-fixed-2026-06-12","value":"Urb fitness bug found+fixed 2026-06-12 (patch in /home/bruno/src/urb, uncommitted): ProgrammeDriven.pm ratio_o/ratio_type grepped case-insensitively over the ratios hash and took the FIRST key — nondeterministic (x4.5 score swings) for designs with mixed-case type classes (both 'c' circulation and 'C' covered). Fixed to SUM the class (matches Is_Circulation//Is_Outside semantics); 35/35 corpus scores unchanged. CRITICAL for homemaker-py-3y7/gnw: the native port must implement class-SUM ratios. Building.pm has the same unpatched pattern (site-driven path, not used by our oracle). Also: the memetic search reward-hacked this bug before the fix — search results predating it are noise artifacts."}
|
||||
{"_type":"memory","key":"correction-to-urb-fitness-bug-memory-bruno-2026","value":"CORRECTION to urb-fitness-bug memory (Bruno, 2026-06-12): 'C' is NOT a 'covered' type — Is_Covered is a geometric predicate (indoor space above). Urb's generic types are canonically UPPERCASE: C=circulation, O=outside, S=sahn (get_space_types qw/C O S/; corpus is 100% uppercase, never 'c'/'o' leaves). The mixed-case designs that fired the latent ratio_type first-match bug were created by homemaker's own operator type pool emitting lowercase 'c'/'o' — fixed: driver/operators now emit uppercase generics only, and class checks use t[0].lower() in 'cos'. The Urb class-sum patch stays as defensive hardening (zero impact on canonical designs). Native port (3y7/gnw): treat type classes case-insensitively, generics canonically uppercase."}
|
||||
{"_type":"memory","key":"urb-oracle-nondeterminism-urb-fitness-pl-output-varies","value":"Urb oracle nondeterminism: urb-fitness.pl output varies run-to-run from Perl hash-order randomisation — .fails line ORDER shuffles (compare sorted, use oracle.Score.fail_lines) and the score float can flip by ~1 ULP (compare with math.isclose rel_tol=1e-12, never ==). Not a batching artifact; affects single runs too. Matters for the Phase 3 native-fitness parity gate (homemaker-py-uxz)."}
|
||||
{"_type":"memory","key":"user-preference-bruno-this-is-a-fedora-system","value":"User preference (Bruno): this is a Fedora system — NEVER install Python packages via pip without asking first; always ask whether to install the rpm via dnf (e.g. python3-cma) before considering pip. Applies to any dependency additions."}
|
||||
{"_type":"memory","key":"strategy-decision-2026-06-12-bruno-occlusion-daylight","value":"Strategy decision 2026-06-12 (Bruno): occlusion/daylight is ORTHOGONAL to building a scalable optimiser. Disable it in Urb (env flag, homemaker-py-gp2) rather than port it; native fitness uses simple crinkliness (illumination factor = 1); rebuild occlusion in Python only after optimisation is fully native (homemaker-py-2g5, now P4). Consequence: all scores change when the flag flips — re-baseline corpus/.score, DESIGN \\$4.5 gains, gate bars at one clean boundary AFTER homemaker-py-1p0 closes; Phase-2 urb-evolve benchmark must run with the same flag."}
|
||||
{"_type":"memory","key":"urb-fitness-bug-found-fixed-2026-06-12","value":"Urb fitness bug found+fixed 2026-06-12 (patch in /home/bruno/src/urb, uncommitted): ProgrammeDriven.pm ratio_o/ratio_type grepped case-insensitively over the ratios hash and took the FIRST key — nondeterministic (x4.5 score swings) for designs with mixed-case type classes (both 'c' circulation and 'C' covered). Fixed to SUM the class (matches Is_Circulation//Is_Outside semantics); 35/35 corpus scores unchanged. CRITICAL for homemaker-py-3y7/gnw: the native port must implement class-SUM ratios. Building.pm has the same unpatched pattern (site-driven path, not used by our oracle). Also: the memetic search reward-hacked this bug before the fix — search results predating it are noise artifacts."}
|
||||
|
|
|
|||
133
experiments/benchmark_vs_urbevolve.py
Normal file
133
experiments/benchmark_vs_urbevolve.py
Normal file
|
|
@ -0,0 +1,133 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Phase-2 gate: memetic loop vs urb-evolve at equal oracle-eval budgets
|
||||
(homemaker-py-way).
|
||||
|
||||
Protocol: 3 seed designs x 2000-evaluation runs; the 1000-eval tier is read
|
||||
from each run's own best-so-far log (no extra oracle time). urb-evolve runs
|
||||
at two population sizes (its default 128, which only allows ~16 generations
|
||||
at this budget, and 16, which allows ~130) and gets credit for its better
|
||||
one. urb-evolve counts evaluations via the MAX_EVALS patch; the memetic
|
||||
driver accounts natively. Both run under URB_NO_OCCLUSION=1 with the same
|
||||
patterns.config; every final design is re-scored through urb-fitness.pl as
|
||||
the common deterministic yardstick.
|
||||
|
||||
URB_NO_OCCLUSION=1 python3 experiments/benchmark_vs_urbevolve.py [budget]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
||||
from homemaker import oracle # noqa: E402
|
||||
|
||||
URB = Path("/home/bruno/src/urb")
|
||||
EX = URB / "examples" / "programme-house"
|
||||
HERE = Path(__file__).resolve().parent
|
||||
WORK = HERE.parent / "scratch" / "benchmark"
|
||||
|
||||
SEEDS = ["init.dom", "c964435454c459f86c3ed9a5a7621132.dom",
|
||||
"2f45907abd9accac2a124d311732f749.dom"]
|
||||
CHECKPOINT_FRACTION = 0.5
|
||||
|
||||
|
||||
def run_homemaker(seed: str, budget: int, cell: Path) -> dict:
|
||||
out = cell / "best.dom"
|
||||
proc = subprocess.run(
|
||||
[sys.executable, str(HERE / "run_search.py"), str(budget), "0",
|
||||
str(EX / seed), str(out)],
|
||||
capture_output=True, text=True, env={**os.environ, "URB_NO_OCCLUSION": "1"},
|
||||
)
|
||||
(cell / "run.log").write_text(proc.stdout + proc.stderr)
|
||||
series = [(int(m[1]), float(m[2])) for m in
|
||||
re.finditer(r"\[\s*(\d+) evals\] best ([\d.e-]+)", proc.stdout)]
|
||||
return {"series": series, "best_dom": out, "log_ok": proc.returncode == 0}
|
||||
|
||||
|
||||
def run_urbevolve(seed: str, budget: int, pop: int, cell: Path) -> dict:
|
||||
shutil.copy(EX / "patterns.config", cell)
|
||||
shutil.copy(EX / seed, cell)
|
||||
proc = subprocess.run(
|
||||
["perl", f"-I{URB}/lib", str(URB / "bin" / "urb-evolve.pl"), seed],
|
||||
cwd=cell, capture_output=True, text=True,
|
||||
env={**os.environ, "URB_NO_OCCLUSION": "1", "MAX_EVALS": str(budget),
|
||||
"MAX_POP": str(pop), "MAX_ITERATIONS": "100000"},
|
||||
)
|
||||
(cell / "run.log").write_text(proc.stderr)
|
||||
series = []
|
||||
evals = None
|
||||
for line in proc.stderr.splitlines():
|
||||
if m := re.search(r"Evals: (\d+)", line):
|
||||
evals = int(m[1])
|
||||
elif (m := re.search(r"Fitness: ([\d.e-]+)", line)) and evals is not None:
|
||||
series.append((evals, float(m[1])))
|
||||
out_hash = proc.stdout.strip().split()[-1] if proc.stdout.strip() else None
|
||||
best = cell / f"{out_hash}.dom" if out_hash else None
|
||||
return {"series": series, "best_dom": best,
|
||||
"log_ok": proc.returncode == 0 and best is not None and best.exists()}
|
||||
|
||||
|
||||
def best_at(series: list[tuple[int, float]], budget: int) -> float:
|
||||
vals = [f for e, f in series if e <= budget]
|
||||
return max(vals) if vals else float("nan")
|
||||
|
||||
|
||||
def main() -> int:
|
||||
budget = int(sys.argv[1]) if len(sys.argv) > 1 else 2000
|
||||
checkpoint = int(budget * CHECKPOINT_FRACTION)
|
||||
if WORK.exists():
|
||||
shutil.rmtree(WORK)
|
||||
|
||||
cells = []
|
||||
for seed in SEEDS:
|
||||
cells.append((seed, "memetic", run_homemaker, {}))
|
||||
cells.append((seed, "urb-evolve p16", run_urbevolve, {"pop": 16}))
|
||||
cells.append((seed, "urb-evolve p128", run_urbevolve, {"pop": 128}))
|
||||
|
||||
def run_cell(args):
|
||||
seed, label, fn, kw = args
|
||||
cell = WORK / f"{seed.split('.')[0][:8]}_{label.replace(' ', '_')}"
|
||||
cell.mkdir(parents=True, exist_ok=True)
|
||||
print(f"start: {seed} / {label}", flush=True)
|
||||
res = fn(seed, budget, cell, **kw)
|
||||
print(f"done: {seed} / {label} ({len(res['series'])} log points)", flush=True)
|
||||
return (seed, label), res
|
||||
|
||||
with ThreadPoolExecutor(max_workers=3) as pool:
|
||||
results = dict(pool.map(run_cell, cells))
|
||||
|
||||
# common yardstick: re-score every final design through urb-fitness.pl
|
||||
for key, res in results.items():
|
||||
if res["best_dom"] and Path(res["best_dom"]).exists():
|
||||
s = oracle.score(res["best_dom"], URB)
|
||||
res["final"], res["fails"] = s.fitness, s.n_fails
|
||||
else:
|
||||
res["final"], res["fails"] = float("nan"), -1
|
||||
|
||||
print(f"\n{'seed':10s} {'system':16s} {'best@' + str(checkpoint):>12s} "
|
||||
f"{'final@' + str(budget):>12s} {'fails':>6s}")
|
||||
verdicts = []
|
||||
for seed in SEEDS:
|
||||
rows = {label: results[(seed, label)] for label in
|
||||
("memetic", "urb-evolve p16", "urb-evolve p128")}
|
||||
for label, res in rows.items():
|
||||
print(f"{seed[:10]:10s} {label:16s} {best_at(res['series'], checkpoint):12.6g} "
|
||||
f"{res['final']:12.6g} {res['fails']:6d}")
|
||||
hm = rows["memetic"]["final"]
|
||||
urb = max(rows["urb-evolve p16"]["final"], rows["urb-evolve p128"]["final"])
|
||||
verdicts.append(hm > urb)
|
||||
print(f"{'':10s} -> memetic {'BEATS' if hm > urb else 'LOSES TO'} "
|
||||
f"urb-evolve ({hm:.6g} vs {urb:.6g})")
|
||||
print(f"\nGATE: memetic wins {sum(verdicts)}/{len(verdicts)} seeds "
|
||||
f"-> {'GO' if all(verdicts) else 'REVIEW'}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
|
|
@ -11,7 +11,7 @@ DESIGN §4.7 reference: c964435 -> 0.00674). Anything above that is value
|
|||
added by *topology* search.
|
||||
|
||||
Run under the go-forward fitness:
|
||||
URB_NO_OCCLUSION=1 python3 experiments/run_search.py [budget] [rng-seed]
|
||||
URB_NO_OCCLUSION=1 python3 experiments/run_search.py [budget] [rng-seed] [seed.dom] [out.dom]
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -32,21 +32,23 @@ OUT = Path(__file__).resolve().parents[1] / "scratch" / "search_best.dom"
|
|||
def main() -> int:
|
||||
budget = int(sys.argv[1]) if len(sys.argv) > 1 else 2000
|
||||
rng_seed = int(sys.argv[2]) if len(sys.argv) > 2 else 0
|
||||
print(f"seed={SEED_FILE.name} budget={budget} oracle evals, rng seed {rng_seed}",
|
||||
seed_file = Path(sys.argv[3]) if len(sys.argv) > 3 else SEED_FILE
|
||||
out = Path(sys.argv[4]) if len(sys.argv) > 4 else OUT
|
||||
print(f"seed={seed_file.name} budget={budget} oracle evals, rng seed {rng_seed}",
|
||||
flush=True)
|
||||
|
||||
r = driver.search(dom.load(str(SEED_FILE)), EX, budget=budget,
|
||||
r = driver.search(dom.load(str(seed_file)), EX, budget=budget,
|
||||
urb_root=URB, seed=rng_seed, log=lambda m: print(m, flush=True))
|
||||
|
||||
print(f"\ndone: {r.n_evals} oracle evals across {r.n_topologies} topologies")
|
||||
print(f"best: {r.best.fitness:.6g} (fails {r.best.n_fails}) via {r.best.lineage}")
|
||||
print("population: " + ", ".join(f"{p.fitness:.4g}/{p.n_fails}f" for p in r.population))
|
||||
|
||||
OUT.parent.mkdir(exist_ok=True)
|
||||
dom.dump(r.best.root, str(OUT))
|
||||
s = oracle.score(OUT, URB)
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
dom.dump(r.best.root, str(out))
|
||||
s = oracle.score(out, URB)
|
||||
ok = math.isclose(s.fitness, r.best.fitness, rel_tol=1e-9)
|
||||
print(f"\n{OUT} re-scored standalone: {s.fitness:.6g} ({s.n_fails} fails) "
|
||||
print(f"\n{out} re-scored standalone: {s.fitness:.6g} ({s.n_fails} fails) "
|
||||
f"-> {'MATCHES search record' if ok else 'MISMATCH'}")
|
||||
return 0 if ok else 1
|
||||
|
||||
|
|
|
|||
|
|
@ -287,6 +287,11 @@ def optimise(
|
|||
with OracleEvaluator(root, programme_dir, urb_root) as ev:
|
||||
if x0 is None:
|
||||
x0 = ev.x_current
|
||||
if len(x0) == 0: # undivided topology (e.g. a bare plot): nothing to optimise
|
||||
s = ev.evaluate([np.empty(0)])[0]
|
||||
return Result(x=np.empty(0), fitness=s.fitness, n_fails=s.n_fails,
|
||||
fail_lines=s.fail_lines, x0_fitness=s.fitness,
|
||||
x0_n_fails=s.n_fails, n_evals=1, n_oracle_calls=1)
|
||||
result = _METHODS[method](ev, x0, budget=budget, **search_kw)
|
||||
ev.apply(result.x) # leave the tree at the optimum (Lamarckian write-back)
|
||||
return result
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue