# homemaker — Design & Plan **Status:** validated direction, pre-implementation. Reviewed against the Urb source 2026-06-12; review findings folded in (see §4.5 evidence note, §4.6 throughput arithmetic, §5 decision 6, §6 port-scope expansion, §7 re-scoped phases, §8). **Audience:** a fresh session that will break this into `bd` (beads) tasks (note: no beads database exists yet — run `bd init` first). Self-contained — assumes no memory of the originating conversation. --- ## 1. Purpose `homemaker-layout` is a clean-room Python successor to the Perl **Urb** project (`/home/bruno/src/urb`). Urb models a building as a binary **slicing tree** and evolves layouts with mutation + crossover, scored against Christopher Alexander–style pattern fitness. Two long-standing problems motivate the rewrite: 1. **It doesn't scale** — beyond a few rooms, evolution never finds layouts an architect would consider obvious. 2. **Local minima** — even small programmes converge to poor optima. The eventual goal is a **100% Python** system. During bring-up, Perl Urb is kept as a throwaway **fitness oracle** behind the `.dom` file format. --- ## 2. Constraints that fix the representation These come from the problem domain and are **not negotiable**; importantly, they *vindicate* the slicing tree rather than argue against it: - **Multi-storey with stacked walls.** An upper storey retains the storey below, except additional divisions/undivisions. Load-bearing walls must stack ⇒ every cut is a full edge-to-edge **guillotine** cut. Urb already enforces this via `Below`-inheritance (an upper quad reads its geometry from the matching quad below). - **Quadrilateral rooms only** (no L/Z shapes) — recursive bisection produces exactly this. - **No pinwheel / non-slicing layouts** — undesirable for load-bearing construction and adaptability (cf. Brand, *How Buildings Learn*). This is the one class a slicing tree *can't* express, and we don't want it anyway. - **Plots are near-rectangular but general convex quadrilaterals** (not axis-aligned). Geometry must handle skew; the slicing *combinatorics* are unaffected. **Conclusion:** the slicing tree is the correct phenotype. The rewrite is about the *genotype*, the *search*, and the *fitness shape* — not about leaving the slicing class. --- ## 3. What we built this session (all committed) Package `src/homemaker_layout/`: - **`dom.py`** — `.dom` YAML ⇄ `Node` tree. Linkage (`parent`/`below`/`position`), `wall_outer` inset on load with raw-corner stash for byte-perfect round-trip, emit. - **`geometry.py`** — faithful port of Urb's top-down geometry (`Coordinate`/`Coordinate_a`/`_b`/`Area`/`Length`) + `Coordinate_Offset` wall inset. **Memoised** (uncached recursion is exponential in depth). - **`programme.py`** — parse `patterns.config` `spaces:` into per-code size/width/proportion/adjacency/level/count requirements. - **`solver.py`** — bottom-up division-ratio solver (scipy `least_squares`). *(Outcome: falsified as a standalone component — see §4.2.)* - **`oracle.py`** — Phase-1 fitness bridge: write `.dom`, run `urb-fitness.pl`, parse `.score` + `.fails`. Experiments in `experiments/`: `dump_areas.{py,pl}`, `resolve_ratios.py`, `refine_sweep.py`, `sweep_failtypes.py`, `optimize_fullfitness.py`. --- ## 4. Empirical findings (the core of this document) ### 4.1 Geometry port — VALIDATED Per-leaf areas computed in Python are **byte-identical to Urb across all 35 programme-house `.dom` files**, including the wall inset and multi-storey wall-stacking inheritance. (`experiments/dump_areas.{py,pl}`.) The infrastructure is trustworthy. ### 4.2 Bottom-up area-proxy sizing solver — FALSIFIED The original hypothesis: give leaves *target sizes*, solve cut ratios bottom-up, let the EA search only topology. Tested by re-solving an evolved candidate's ratios from programme targets and scoring via the oracle. - `resolve_ratios.py` on candidate-002: areas recovered accurately (errors collapsed, e.g. t1/t2/t3 from +1.4/+2.4/+4.8 → ~+0.05), and it *fixed* the original's `size` failure — **but total fitness dropped** (0.00737 → 0.00065, 4 fails) because it introduced shape/relational failures. - `refine_sweep.py` (warm-start refine of all 34 candidates): **0/34 improved.** Total failures 124 → 297 (equal-offset cuts) and 124 → 626 (independent-offset cuts). - `sweep_failtypes.py` (failure-type histogram, equal-offset): | type | area-dominant Δ | shape-aware Δ | |---|---|---| | width | +82 | +29 | | proportion | +35 | +7 | | crinkliness | +18 | +4 | | adjacency | +18 | +13 | | size | **−15** | **+15** | | access | +29 | +39 | | **total added** | +173 | +110 | **Why it fails:** in Urb's fitness, every cut position is simultaneously a *size* knob **and** an *adjacency/access/shape* knob. A solver that optimises only size/shape is blind to access/adjacency and trades them away. Refining a co-evolved local optimum with a *partial* objective is **structurally unable to win**, and the `0.5^n` failure penalty makes every new failure catastrophic while fixes are only linear. The proxy solver is strictly worse than optimising real fitness. **Do not pursue it.** ### 4.3 "Perpendicular" failures were an artifact — RESOLVED Letting the two ends of a cut float independently produced skewed cuts and many `perpendicular` failures. Tying the two ends (**equal offset, `a == b`**, one DOF per cut) produces near-perpendicular walls on these near-rectangular plots and yields **zero** `perpendicular` failures. **Equal-offset cuts are the only mode to use.** This also halves the variable count and matches the slicing model. ### 4.4 DOF / over-determination — partially real, not fatal A topology with *R* rooms has ~*R−1* cut DOF but ~2–3 size/shape constraints per room, so a *fixed* topology can be over-determined: you cannot always hit area + width + proportion for every room at once (heavy shape weighting traded straight into `size`, §4.2 table). This limits any single-objective sizing pass — but it is **not** fatal, because optimising the *full* objective still found large gains (§4.5). The earlier "infeasibility" worry was overstated. ### 4.5 Full-fitness frozen-topology optimisation — VALIDATED ✅ Drive the equal-offset ratios with Nelder-Mead against the **real oracle fitness** (whole objective, no proxy), topology frozen (`experiments/optimize_fullfitness.py`): | candidate | DOF | original | optimised | gain | fails | |---|---|---|---|---|---| | 2f45907 (best evolved) | 7 | 0.012617 | 0.015684 | ×1.24 | 2→2 | | candidate-002 (MCP-refined) | 6 | 0.007375 | 0.012319 | ×1.67 | 2→2 | | c964435 (MCP baseline) | 6 | 0.003667 | 0.005836 | ×1.59 | 3→3 | **Every design improved 24–67%, none added a failure.** Headroom *widens* on weaker designs. Because the optimiser sees the whole objective (including the `0.5^n` penalty), it never trades into a new failure — **the cliff that destroys the proxy solver protects the full-objective optimiser.** **Implications:** - There is large, unclaimed **geometry headroom above every EA design** — even the best. Urb's EA under-optimises geometry: source inspection confirms `slide()` (Mutate.pm:256-269) *re-randomises* the cut position uniformly across the span — Urb has **no fine-tuning geometry operator at all**, which fully explains the headroom. - A **full-objective geometry inner loop is genuinely valuable** (the proxy solver is not). - The EA/search should therefore own **topology**; geometry is delegated to the inner loop. This is the memetic architecture (§5). - Corroboration for §4.3: Urb's own mutations use equal offsets (`Divide($division, $division)`) — equal-offset cuts match how every corpus design was generated. ### 4.6 Oracle throughput (measured) `urb-fitness.pl` scores **many `.dom` files per invocation**, so the Perl startup (~0.65 s) amortises across a batch and cached fields (e.g. occlusion) persist. Measured on the 35-file corpus: **0.99 s/dom batched** vs **1.65 s/dom** for a single-file call. The cost is **assessment-dominated** (~1 s/dom of actual work), so startup amortisation gives ~40% — useful but bounded. Consequences: - **Batching only helps when evaluations are submitted together** — favour **population/parallel-evaluating optimisers** (CMA-ES, differential evolution, island EA, pattern search) over inherently sequential ones (Nelder-Mead), both inner loop and outer search, so a whole generation scores in one oracle call. - **Do the arithmetic before scoping topology search on the oracle.** §4.5 used ~200 inner evaluations per topology ⇒ ~3 min/topology at 1 s/dom. A run comparable to `urb-evolve` (pop 128 × 768 generations) is *years* of oracle time; even 32 topologies × 100 generations with a trimmed 50-eval inner loop is ~2 days. Therefore: - The oracle supports **Phase 1 fully** and **Phase 2 only as a small-scale proof** (tens of topologies, budgets counted in oracle calls). - A **native Python fitness is effectively a gate for topology search at any real scale** — not merely a later optimisation. (It also brings independence, penalty reshaping, and large programmes.) - **Warm-starting the inner loop from the parent's optimised ratios** (Lamarckian inheritance, §5 decision 6) is the main lever for cutting the per-topology cost — with high-locality moves most cuts survive a mutation, so an order-of-magnitude reduction is plausible. Measure this in Phase 1. ### 4.7 Occlusion-disabled re-baseline (measured 2026-06-12) With the §6 descope in place (`URB_NO_OCCLUSION=1` patch in Urb), the corpus re-baseline (`experiments/rebaseline_no_occlusion.py`): all 35 scores change (mostly up, ×1.0–×1.24 — daylight terms pin to 1), exactly one failure-set change (458aa8b8 gains two `crinkliness` fails — expected mechanism: no shading discount on external wall area), batched oracle ~8% faster (0.92 s/dom). New inner-loop reference gains (deterministic seed, budget 400, `accept_innerloop.py` bars): 2f45907 0.01304→0.02128 (×1.63), candidate-002 0.00808→0.01373 (×1.70), c964435 0.00400→0.00674 (×1.68, fails 3→2); ~35 oracle calls per topology. All Phase-2+ work uses the flag; flag-off numbers above are historical. ### 4.8 The `0.5^n` failure penalty is a first-order pathology Multiplicative `0.5^n` over failure *count* (a) makes the landscape a cliff (no gradient across the huge zero-feasibility region), (b) rewards fewer *flags* over better *geometry* (the original outscored better-sized solved designs purely on flag count), and (c) is representation-independent. Reshaping it (additive / soft / multi-objective Pareto) is a high-leverage change that helps Urb today and homemaker tomorrow. ### 4.9 Penalty reshaping decision: lexicographic outer search (measured 2026-06-14) `experiments/penalty_reshape.py`, `URB_NO_OCCLUSION=1`, programme-house. **Inner-loop protection** (nm_search, budget 80, 3 files × 3 seeds = 9 runs): All runs show `n_fails ≤ x0_n_fails`. **0/9 regressions.** The `0.5^n` cliff in the native fitness scalar is unchanged and continues to protect the inner loop. **Outer-search comparison** (budget 3000, 3 seeds, seed = 2f45907): | scheme | seed | best | fails | note | |--------|------|------|-------|------| | lex | 0 | 0.01781 | 2 | | | lex | 1 | 0.01793 | 2 | | | lex | 2 | 0.01785 | 2 | | | scalar | 0 | 0.01781 | 2 | (same outcome) | | scalar | 1 | **0.01890** | **3** | trapped by high-score 3-fail design | | scalar | 2 | 0.02632 | 2 | (different topology path) | `lex` mean: 0.01786 / 2.00 fails. `scalar` mean: 0.02101 / 2.33 fails. Key result (seed 1): scalar promoted a 3-fail design whose raw score (×0.125 penalty) beat the pool's 2-fail candidates — exactly the §4.8 pathology. Lexicographic comparison (`-n_fails` first, then `fitness`) is immune: any 2-fail design beats any 3-fail design regardless of raw score. Within a homogeneous fail tier both schemes are identical (seeds 0 and 2 agree in serendipitous runs where scalar also stays in the 2-fail tier). **Decision: lexicographic. `0.5^n` stays in the fitness scalar (inner loop unchanged). Outer search uses `(-n_fails, fitness)` as comparison key.** ### 4.10 Deceptive level-fix valley and compound operators (measured 2026-06-14/15) **Context:** programme-house, Phase 3 native fitness + Phase 4 lex search, seed `warmstart-2f4.dom` (best Phase-3 result, 2 fails at score 0.032). Goal: reach ≤ 1 fail, beating the Perl optimiser (2–3 fails). **The deceptive valley.** The 2-fail state has l1 (living room, min 27 m², required level 0) on level 1. The obvious repair is `level_fix`: swap l1 with a leaf on level 0. But every single-step `level_fix` move creates 5+ new fails because the displaced room (t3, the WC) is dropped into an arbitrary slot that violates adjacency, size, and access constraints simultaneously. The lex comparator (`-n_fails, fitness`) correctly rejects these — but the result is that the 2-fail state appears completely surrounded by ≥ 5-fail states, and the search stalls. This is a textbook deceptive valley: the fitness gradient points away from the global optimum. **Compound operator.** `mutate_level_compound_fix` (added `operators.py`) escapes the valley by doing two things atomically: 1. Move l1 to level 0 by swapping it with the *largest* leaf there (the circulation C node, because C is generic and can absorb the swap without producing a new structural failure). 2. Re-insert the displaced t3 by dividing the sibling of that C node (so t3 lands adjacent to C, satisfying the adjacency requirement). The new split gets `division=[0.25,0.25]` (giving t3 ≈ 3.4 m², barely in range) and `rotation=0` (t3 on the left, adjacent to the C sibling). **The `warm_x0` initialization bug.** The compound operator sets specific ratios on a newly-created split node. But `driver.py` was initialising the NM inner loop from `parent.ratios`, which has no entry for the new node (it was a leaf). `warm_x0` defaulted the new node to 0.5, giving t3 ≈ 6.8 m² — a size fail — so NM started at 3 fails instead of 1. Lex then always rejected the compound child; `level_compound_fix` was completely invisible to the outer search for ~12 000 evals (until `warm_x0` was fixed). The correct fix distinguishes genuinely-new split nodes from stale hidden nodes that become visible after structural mutations (e.g. `swap` can flip a `b.below` pointer, revealing pre-writeback division values from a different topology). Only use the child's explicit ratio for node `(li, path)` if the matching node in the parent was *not already divided*; everything else falls through to `parent.ratios` or defaults to 0.5. Fix in `driver.py` lines 259–267. **Results (50 000 evals each, pop 8, child_budget 80, 4 workers):** | seed | event | eval | fails | score | |------|-------|------|-------|-------| | warmstart-2f4 | seed | 200 | 2 | 0.032 | | warmstart-2f4 | `level_compound_fix` fires | 12 280 | 1 | 0.000122 | | warmstart-2f4 | `level_retype 0/ll<->1/l` | 17 880 | 1 | 0.00497 | | warmstart-2f4 | final | 50 040 | **1** | **0.00518** | | compound3-raw | seed (1-fail hand-built) | 200 | 1 | 0.000118 | | compound3-raw | `level_retype 0/ll<->1/l` | 18 360 | 1 | 0.00383 | | compound3-raw | final | 50 040 | **1** | **0.00523** | Perl optimiser reference: **2–3 fails**. **The two-C topology breakthrough.** After `level_compound_fix` fires, the topology is: level 0 = `ll(l1), lr(t2), rl(C), rrl(t3), rrr(O)` — but now l1 is at level 0 (correct) and t3 is adjacent to rl(C) (staircase). However l1 is occupying ll, and rl(C) is the staircase core — so t3-adj-C is satisfied via rl, but there is no second C to satisfy staircase independently. Score ≈ 0.000157 (1 fail). At eval ≈ 18 000, `level_retype 0/ll<->1/l` (swap the type of ll on level 0 with l on level 1) creates a TWO-C configuration at level 0: `ll(C), lr(t2), rl(C), rrl(t3), rrr(O)`, with l1 moving to level 1. The score jumps 25× to ≈ 0.005. Why two C nodes work: - `ll(C)` (bottom-left, 23 m²) satisfies t3-adj-C via geometric contact at the l/r zone boundary with `rrl(t3)`. - `rl(C)` (top-right, 8.5 m²) satisfies staircase adjacency via tree adjacency to `rrr(O)` (its right sibling when `r.rotation=3`). Both constraints are simultaneously met because binary-tree sibling adjacency and cross-zone geometric adjacency provide *independent* paths. **Why 0 fails is geometrically impossible on this programme + plot.** l1 needs min 27 m² at level 0. The only space large enough is `ll` (≈ 23 m², the entire left half of level 0). Putting l1 at `ll` removes the t3-adj-C provider. The alternative — dividing `ll` into `lll(l1)+llr(C)` — gives `llr` a proportion of ≈ 6:1 (width ≈ 0.73 m), failing both the proportion and width constraints. 0 fails is not achievable on this programme+plot with a binary slicing tree representation; 1 fail is the geometric optimum. --- ## 5. Validated architecture **Memetic search, full objective throughout:** ``` ┌─────────────────────── topology search (OUTER) ───────────────────────┐ │ genome = slicing topology + per-leaf type assignment + per-floor │ │ divide/undivide deltas (base floor is master) │ │ operators = high-locality topology moves (see §6) │ │ │ │ for each proposed topology: │ │ ┌──────────── geometry inner loop ────────────┐ │ │ │ optimise equal-offset cut ratios (1 DOF/cut) │ │ │ │ against the FULL fitness (derivative-free / │ │ │ │ gradient), to convergence │ │ │ └──────────────────────────────────────────────┘ │ │ score = best full-fitness over inner loop │ └──────────────────────────────────────────────────────────────────────────┘ fitness: NATIVE Python (fast), reshaped penalty ``` Key decisions, all evidence-backed: 1. **Geometry = inner optimisation against full fitness** (§4.5), *not* an area proxy (§4.2). Equal-offset cuts, one DOF per free branch (§4.3). 2. **Search owns topology only.** The base-floor tree is the primary genome; per-floor deltas are a small secondary genome (multi-storey constraint as a regulariser, via `Below`-inheritance). 3. **Prefer population/batch-evaluating optimisers** so the batched oracle is efficient (§4.6). A **native Python fitness** (faithful to Urb, validated against the oracle on the 35-file corpus) **gates topology search at scale** (§4.6 arithmetic); the oracle suffices for the inner loop and a small-scale topology-search proof only. 4. **Reshape the failure penalty** (§4.8) — additive/soft or multi-objective — so the search has a gradient and isn't dominated by flag-count. **Caution:** the `0.5^n` cliff is what *protects* the inner loop from trading into new failures (§4.5); reshaping must not lose that property. Candidate resolutions: keep the cliff inside the inner loop only, lexicographic ordering (failure count first, score second), or genuine multi-objective Pareto. Decide in Phase 4 with measurements. 5. **Representation upgrade (later):** canonical slicing encoding (normalized Polish expression / skewed slicing tree, Wong–Liu) for redundancy-free, high-locality topology moves; bottom-up shape feasibility checks. Defer until the inner loop + native fitness are in place. 6. **Lamarckian geometry inheritance.** A child topology's inner loop warm-starts from the parent's optimised ratios (cuts that survive the topology move keep their values; new cuts get heuristic defaults). This is the main cost lever for the memetic loop (§4.6) and a standard memetic design choice (Lamarckian vs Baldwinian — we write the optimised geometry back into the genome). Validate the warm-vs-cold speedup in Phase 1. What we are **not** doing: the bottom-up area-proxy solver; independent-offset cuts; non-slicing representations (sequence-pair/B*-tree — excluded by §2). --- ## 6. Component plan | component | status | notes | |---|---|---| | `dom.py` (I/O + linkage) | ✅ done | round-trips byte-perfect; keep | | `geometry.py` (port + cache) | ✅ done, validated | the trusted geometry kernel | | `programme.py` | ✅ done | extend as fitness needs grow | | `oracle.py` (Perl bridge) | ✅ done | throwaway; the validation reference | | `solver.py` (area proxy) | ⚠️ keep as artifact | falsified; do not build on it | | **geometry inner loop** | ❌ to build | full-objective ratio optimiser (DOF = free branches); batch/population so the oracle batches; warm-start support (§5.6) | | **topology genome + operators** | ❌ to build | base tree + per-floor deltas; high-locality moves | | **search driver** | ❌ to build | memetic EA / SA over topology; small-scale on oracle, full-scale needs native fitness | | **native fitness** | ❌ to build | **gates topology search at scale** (§4.6); port + validate vs oracle; scope is larger than the term list — see below | | **penalty reshaping** | ❌ to design | additive/soft or multi-objective; must preserve inner-loop cliff protection (§5.4) | | canonical encoding (Polish expr.) | ❌ later | representation upgrade once core lands | Urb fitness terms the native port must reproduce (all couple to geometry): **size, width, proportion, adjacency, access/inaccessible, crinkliness, perpendicular, level, staircase volume/count, public access, circulation & outside ratios, min internal area.** Source of truth: `/home/bruno/src/urb/lib/Urb/Dom/Fitness/ProgrammeDriven.pm` and the `Storey`/ `Building`/`Leaf`/`Base` submodules. **Port scope beyond the term list** (found by source review — budget for these): - **Daylight + occlusion subsystem — DESCOPED (decision 2026-06-12).** Occlusion is orthogonal to building a scalable optimiser. Instead of porting `Urb::Misc::Sun`/`Urb::Field::Occlusion`/CIESky, disable it in Urb behind an env flag (`quality_daylight` → 1 everywhere; `Crinkliness`/`Area_Outside` pins the `CIEsky_vertical` illumination factor to 1 — *simple crinkliness* = unweighted external wall area / floor area). The boundary-overlap geometry (`Dom->Walls`) stays in scope; the sky model does not. The native fitness ports simple crinkliness only; a Python occlusion subsystem is rebuilt post-Phase-5 once optimisation is fully native. **Flipping the flag changes every score** — re-baseline the corpus, the §4.5 table, and gate bars at one clean boundary, and run the Phase-2 urb-evolve benchmark under the same flag. - **The cost denominator.** Fitness is value/**cost**: per-leaf area costs, interior/exterior wall edge costs, boundary costs (Leaf.pm:194-251, Storey.pm:122-147). Cost couples to geometry too. - **Structural failures** not in the term list: "edge too long" (>8 m, two variants), "unsupported covered outside", "covered outside above ground", "level N not connected". - **Missing-space failure stacking** (ProgrammeDriven.pm:192-212): a missing space generates 2 base failures plus one per size/width/proportion/adjacency/ level requirement — up to ~7 failures. Penalty reshaping (Phase 4) must preserve this hierarchy or the search will happily drop rooms. - **Two-phase graph build**: adjacency/level/vertical checks run on the *unmerged* tree; graphs are rebuilt after `Merge_Divided` for storey processing (ProgrammeDriven.pm:83-103). Easy to get subtly wrong; the 35-file validation gate will catch it, but anticipate it. - **Known stub to decide on** (fidelity-vs-fix, §8.1): `has_vertical_connection` (ProgrammeDriven.pm:399-423) matches any leaf of the target type anywhere on the level below — no spatial-overlap check. A faithful port reproduces the bug; decide explicitly. --- ## 7. Phased roadmap - **Phase 0 — diagnostics** *(done)*: geometry port validated; proxy solver falsified; full-fitness geometry headroom validated; oracle throughput measured (~1 s/dom batched). - **Phase 1 — geometry inner loop (on batched oracle)**: full-objective ratio optimiser; use a population/batch optimiser so a generation scores in one oracle call. Reproduce/exceed the §4.5 gains. Integrate as `optimise(topology, x0=None) -> (geometry, fitness)`. Two cheap experiments belong here: (a) **warm-vs-cold start** — quantify the §5.6 speedup; (b) **optimiser bake-off** — DOF is only ≈ rooms−1, so batched multi-start pattern search may beat CMA-ES on simplicity; measure, don't commit blind. *Gate:* match §4.5 gains at materially lower oracle-call budget. - **Phase 2 — topology search, small-scale proof (on batched oracle)**: base-tree + per-floor-delta genome, high-locality operators, memetic driver wrapping the Phase-1 inner loop. **Explicitly small** (§4.6 arithmetic): tens of topologies, budgets counted in **oracle evaluations**, not generations. Compare against `urb-evolve` from the same seeds/programmes *at equal oracle-call budget* (urb-evolve has diversity injection/culling baked in, so generations are not comparable). *Gate:* memetic loop beats equal-budget urb-evolve. Scaling up waits for Phase 3. **Gate result (homemaker-py-way, 2026-06-13, `URB_NO_OCCLUSION=1`, budget 2000):** `experiments/benchmark_vs_urbevolve.py`; urb-evolve scores unchanged, memetic scores corrected (patterns.config missing from re-score cwd in first run, fixed in same session). | seed | system | best@1000 | final@2000 | fails | |------|--------|-----------|------------|-------| | init.dom | memetic | 8.84e-10 | 3.37e-09 | 18 | | init.dom | urb-evolve p16 | 9.10e-06 | 9.36e-05 | 6 | | init.dom | urb-evolve p128 | 4.83e-09 | 3.27e-05 | 6 | | c964435 | memetic | 7.65e-03 | **7.65e-03** | 2 | | c964435 | urb-evolve p16 | 4.00e-03 | 4.00e-03 | 3 | | c964435 | urb-evolve p128 | 4.00e-03 | 4.00e-03 | 3 | | 2f45907 | memetic | 2.13e-02 | **2.13e-02** | 2 | | 2f45907 | urb-evolve p16 | 1.30e-02 | 1.30e-02 | 2 | | 2f45907 | urb-evolve p128 | 1.30e-02 | 1.30e-02 | 2 | **Verdict: 2/3 seeds → REVIEW.** - *Seeded designs (c964435, 2f45907)*: memetic beats urb-evolve by 1.91× and 1.63×; topology search adds value over the inner-loop-only reference (crossover finds a better topology at eval 372 for c964435). - *Blank-slate (init.dom)*: memetic stalls at 18 fails after 2000 evals; urb-evolve reaches 6 fails. The `0.5^n` cliff means each fail adds ~2× penalty; 12-fail gap = ×4096. Root cause: single-seed topology mutation chain builds structure one room at a time; urb-evolve's random-population initialisation explores broader topology diversity upfront. **Not a regression** — this is a scope gap: blank-slate construction is harder than seeded improvement, and addressed separately (random multi-start bootstrap, or Phase 4 penalty reshaping which flattens the fail cliff). - The memetic loop is confirmed correct and competitive on the realistic use case (seeded designs). Phase 3 (native fitness) unblocks scaled runs where this gap will also narrow. - **Phase 3 — native Python fitness** (**gates scaled topology search**): first disable occlusion/daylight in Urb behind an env flag and re-baseline (§6 descope note); then port Urb's programme-driven fitness — the §6 "port scope beyond the term list" items (simple crinkliness, cost denominator, structural failures, failure stacking, two-phase graph build). Validate score + failure set against the *flagged* oracle across the 35-file corpus (float tolerance, identical failure sets). Swap behind the same interface; retire the oracle. Then re-run Phase 2 at scale. **Gate result (homemaker-py-ccw, 2026-06-13, `URB_NO_OCCLUSION=1`, budget 20000):** `experiments/run_search_scaled.py`; native fitness only, no oracle. pop_size=16, child_budget=80, seed_budget=300. 71.8 evals/s, 279.8s elapsed. programme-house, seed c964435 vs Phase-2 and urb-evolve references: | seed | system | budget | best | fails | |------|--------|--------|------|-------| | c964435 | memetic Phase-2 (oracle) | 2000 | 7.65e-03 | 2 | | c964435 | urb-evolve p16 | — | 4.00e-03 | 3 | | c964435 | urb-evolve p128 | — | 4.00e-03 | 3 | | c964435 | **memetic Phase-3 (native)** | **20000** | **1.04e-02** | **2** | **Verdict: PASS.** - Best 1.04e-02 beats Phase-2 oracle run (7.65e-03) by **1.36×** and urb-evolve p128 (4.00e-03) by **2.60×**; both at 2 fails. - Winning topology found at eval 10357 via `rotate 1/ll` — unreachable within the Phase-2 budget of 2000. - Population diverse: 16 members, all at 2 fails (top 15), range 5.99e-03–1.04e-02. - Throughput 71.8 evals/s vs ~0.5 evals/s for the batched oracle (≈140× speedup). - harbor-house (16 rooms, oracle-impossible): run attempted, results below. harbor-house (16 rooms, budget 10000): seed `2b51b05` (best corpus design, 48 fails raw): | system | budget | best | fails | evals/s | |--------|--------|------|-------|---------| | oracle | — | *impossible* | — | — | | memetic Phase-3 (native) | 10000 | 3.73e-18 | 49 | 15.8 | Search found 3.73e-18 vs seed inner-loop baseline 8.73e-19 (4.3× lift). 638 topologies in 633s. 49-fail landscape: still many fails, but topology search is finding structure (best 3 population members all at 49 fails). The 16-room programme is qualitatively beyond the oracle's capability — this run is only possible with native fitness. - **Phase 4 — penalty reshaping** *(done, homemaker-py-yg5, 2026-06-14)*: **Decision: lexicographic outer-search comparison** (see §4.9). Inner loop unchanged — still uses raw `0.5^n` fitness scalar (cliff protection preserved, §5.4). Outer search compares individuals by `(-n_fails, fitness)`: fewer fails always beats more fails; within a tier, compare by score. Implemented in `driver.search(use_lex=True)`. `_CHILD_INNER_KW` stale `sigmas` entry also removed (NM default has no `sigmas` parameter). - **Phase 5 — representation upgrade**: canonical slicing encoding (Polish expression) + bottom-up shape feasibility; scale to larger programmes. Each phase has a concrete go/no-go gate; do not advance on faith. --- ## 8. Risks & open questions (decisions for the next session) 1. **Native-fitness fidelity vs simplification.** Port Urb's fitness exactly (maximise comparability) or take the opportunity to clean up known issues (the `0.5^n` cliff, the t3 width-default contradiction below, the `has_vertical_connection` no-overlap stub — §6)? Recommend: *port faithfully first* (bugs included), validate, then reshape in Phase 4. 2. **Programme contradictions exist.** e.g. t3 (3 m² WC) inherits the 4 m `width_inside` default (Fitness/Base.pm:60) — geometrically impossible; the original "passes" only by failing `size` instead. *Confirmed in source.* Need a sane width default scaled to area, or per-room widths. 3. **Inner-loop optimiser choice — RESOLVED (homemaker-py-d0s, 2026-06-13).** Bake-off over 3 files × 4 methods × 3 seeds at budget 200 (`experiments/bakeoff_innerloop.py`), cold-start, `URB_NO_OCCLUSION=1`: | method | x@40 | x@80 | x@200 | s/eval | oracle calls | fails+ | |-------------|------|------|-------|--------|--------------|--------| | Nelder-Mead | 1.45 | 1.50 | 1.56 | 2.05 | 200 | 0 | | CMA-ES | 1.09 | 1.32 | 1.41 | 1.69 | 18 | 0 | | compass | 0.71 | 0.92 | 1.48 | 1.69 | 12 | 3 | | compass-ms | 0.71 | 0.92 | 0.92 | 1.44 | 13 | 4 | **Decision: keep CMA-ES (already the default) for the Perl oracle era.** Nelder-Mead wins quality per eval (+x0.15 at @200) but is inherently sequential — 200 Perl invocations vs 18 for CMA (§4.6 batching matters). Compass stalls on narrow-valley landscapes (2f45907: x0.62 vs x1.30) and introduces fail regressions 3/9 runs. Multi-start compass wastes budget on phase splits. **Phase 3+ note:** once native fitness replaces the oracle, oracle-call count disappears. Revisit Nelder-Mead then — its quality advantage is real. Gradient-based (autograd through native fitness) is also an option. 4. **Search algorithm for topology.** Memetic GA (keep crossover — now meaningful, since a subtree = a contiguous region) vs simulated annealing (the floorplanning workhorse with M1/M2/M3 moves on Polish expressions). 5. **Penalty reshaping vs inner-loop protection — RESOLVED (homemaker-py-yg5, 2026-06-14).** Lexicographic outer-search comparison (§4.9). Inner loop unchanged. 6. **Other continuous DOF are out of scope for Phase 1 — deliberately.** Floor-to-floor height is an Urb mutation (Mutate.pm:279-291, bounded 2.7–3.6 m) and feeds cost and stair fit; stair riser/width similar. Cut ratios dominate. Revisit (+1 DOF per storey) if Phase 2 plateaus. 7. **End-state confirmed: 100% Python**; Perl oracle is scaffold only. --- ## 9. How to reproduce (for the next session) ```bash cd /home/bruno/src/homemaker-layout # deps: pyyaml numpy scipy (shapely networkx for later phases) # geometry port vs Urb (must be identical): for d in /home/bruno/src/urb/examples/programme-house/*.dom; do diff <(perl -I/home/bruno/src/urb/lib experiments/dump_areas.pl "$d") \ <(python3 experiments/dump_areas.py "$d") || echo "MISMATCH $d" done python3 experiments/resolve_ratios.py # proxy solver (falsified) python3 experiments/sweep_failtypes.py # failure-type histogram python3 experiments/optimize_fullfitness.py 200 # full-fitness headroom (validated) ``` Oracle invocation (see `oracle.py`): `cwd` = the `.dom`'s directory (so `patterns.config` is found), `perl -I/lib /bin/urb-fitness.pl `, env `DEBUG=1` to defeat the skip-if-newer cache; reads `.score` and `.fails`. --- ## 10. Key gotchas discovered (carry forward) - **Wall inset:** the `.dom` plot is the *outer* boundary; Urb insets the root by `wall_outer` on load (`Urb::Dom::_deserialise`, Dom.pm:458) and offsets back out on save. `geometry.offset_quad` mirrors it; `dom.py` stashes raw corners in `node_file`. Skipping this makes all areas ~14% too large. - **Multi-storey `Below`-inheritance:** an upper quad's coordinates come from the matching quad below; a cut is "owned" by the *lowest* storey where its path is divided (`solver.free_branches` selects these). Walls stack for free. - **Geometry must be cached** — the pull-based recursion is exponential in depth otherwise (`geometry._cache`, cleared on `dom.load` and after each solver mutation). - **Equal-offset cuts** (`a == b`) ⇒ perpendicular walls, 1 DOF/cut. Independent offsets are wrong. - **`0.5^n` cliff** dominates fitness; it punishes new failures catastrophically (good for the inner loop, brutal for search gradient). - **Oracle ≈ 1 s/dom batched** (1.65 s single; assessment-dominated, startup ~0.65 s amortises across a batch). Submit many `.dom`s per call and prefer population optimisers; native fitness is a later speed/scale win, not a gate. --- ## 11. Phase 6 — topology-search quality for full / multi-storey programmes **Epic:** `homemaker-py-c4c`. **Status:** scoped 2026-06-17, pre-implementation. This section is the experiment ledger for the epic; each subsection is stubbed now and **filled in by the session that runs the experiment** (record the command, the numbers, and a one-line verdict, in the style of §4). ### 11.0 Diagnosis (why this phase exists) The delivered speedups landed in the two layers that were **never the bottleneck**. The native fitness (~140× over the oracle, §7 Phase 3) and the geometry inner loop (~1.6×, §4.5/§4.7) both operate *within a fixed topology*: the inner loop polishes geometry **inside a failure tier** and, by design, the `0.5^n` cliff stops it ever changing the failure count (§4.5: 0-fail-change across the headroom table). But final design quality is dominated by **failure count**, which is almost entirely a **topology** property. So faster fitness and better geometry do not move the number an architect would notice. Topology search on full programmes is the weakness: - **blank-slate programme-house** (`init.dom`): memetic stalls at **18 fails**; urb-evolve reaches **6** (§7 Phase 2 verdict). - **harbor-house** (16 rooms): `out1.dom` = **74 fails**, `generated.dom` = **130 fails**, both at ~machine-epsilon score; failures dominated by **`missing`-room stacking** (each missing room stacks critical + size + width + adjacency + level, §6). **Smoking gun:** `operators.mutate_divide` (operators.py:71) types each new leaf **at random** from `programme-codes + C + O`. Nothing makes the required programme spaces a constructive invariant, so on a large programme required rooms simply go missing → catastrophic `0.5^n` stacking, and the search is a random walk over type assignments with a flat-and-catastrophic gradient in the high-fail regime. **Causal frame for the fixes.** The base-floor tree is the *master* genome; upper storeys are divide/undivide deltas (`Below`-inheritance); the programme partitions rooms by required level (harbor: **10 on L0, 4 on L1, 2 free**). So construction and search should follow the genome's dependency order — credible base floor first, upper floors as deltas, with each floor's required-room set known from the programme. **Do not hard-freeze the base** when adding floors: that recreates the §4.2 partial-objective trap at the topology level (a base optimised purely as a ground floor can be a bad *substrate* — the vertical core must stay aligned and load-bearing walls must stack). ### 11.1 Premise experiment: single-storey harbor (`homemaker-py-c4c.1`) *Stub.* Strip harbor to its 10 level-0 rooms as a single-storey programme; run the current search from a bare plot (and from a bootstrap population). Records the construction-vs-coupling verdict that gates the staging work (§11.3). - *Expectation / decision rule:* near-zero fails ⇒ bottleneck is multi-storey *coupling* (staging is the lever); still stalls (esp. `missing`) ⇒ per-floor *construction* itself is the bottleneck (§11.2 required first). - *Result:* TODO — command, best fails/score, fail histogram, verdict. ### 11.2 Programme-aware construction + missing-room repair (`homemaker-py-c4c.2`) *Stub.* Constructive seeder that instantiates each required space (count/level/type) + `mutate_place_missing` repair operator. Highest-leverage fix for the §11.0 diagnosis. - *Gate:* `missing`-type failures collapse to ~0 across the harbor population; net-fail improvement vs the 74-fail `out1.dom` baseline; no regression on the seeded programme-house 1-fail optimum (§4.10). - *Result:* TODO — before/after fail histograms, numbers, verdict. ### 11.3 Staged per-floor search (`homemaker-py-c4c.3`) *Stub.* Stage 1: base floor over the level-0 room set (one tree, no deltas) + reserved core + substrate-readiness term. Stage 2: upper floors as deltas seeded with their required room sets, base kept mutable at low probability. Gated by §11.1 premise. - *Gate:* staged beats single-stage on harbor at equal native-fitness budget; reserved-core + readiness shown to prevent the bungalow trap (stage 2 does not carve a core from scratch); no programme-house regression. - *Result:* TODO. ### 11.4 Graded high-fail objective (`homemaker-py-c4c.4`) *Stub.* Extends Phase 4 (§4.9). Lexicographic-by-total-count gives ~zero signal when every candidate sits at ~49–74 fails. Add partial credit (proximity per unsatisfied constraint and/or count of *distinct* unsatisfied requirements) as a secondary key beneath fail-count, preserving the inner-loop cliff (§5.4) and the missing-space hierarchy (§6). - *Gate:* measured escape from a high-fail plateau the current lex comparator cannot escape at equal budget; inner-loop 0/9-regression check (§4.9) still clean. - *Result:* TODO. ### 11.5 Topology diversity: structural niching + restarts (`homemaker-py-c4c.5`) *Stub.* Replace the fitness-scalar dedup (driver.py:174) with a topology signature so niching is by *structure*, not score; add crowding/restarts/islands to match urb-evolve's upfront diversity on blank slate. - *Gate:* blank-slate programme-house reaches ≤ 6 fails at equal budget; distinct topology-signature count over time quantified before/after. - *Result:* TODO. (Capstone `homemaker-py-9gp` canonical Polish encoding is the principled long-term signature — `(a|b)|c == a|(b|c)` collapse — and lands after §11.2.)