DESIGN.md §38: the plateau is an objective-gradient problem, not a search problem

Diagnostic investigation of why search stalls in local minima. Adds
experiments/diag_exposure_frontage.py (frontage/exposure/value reports,
no search run required) and records the findings as §38.

Core mechanism: quality_uncrinkliness returns a hard 0.0 for any leaf with
no daylit wall, and since leaf quality is a product feeding
value += quality * rate * area, every buried room contributes exactly zero
value while still costing. 45-56% of interior leaves are in this state under
the default construction stack.

Consequences measured, not inferred:
- Deleting a buried O leaf improves the score x85, a buried C leaf x62.
  Nothing pins circulation or outside space, so the search is rewarded by two
  orders of magnitude for deleting the circulation spine. This retro-explains
  §18, §21/§22 and the level-not-connected fails surviving >1M evals.
- Closed-form frontage bound: every interior leaf needs exposed wall
  L >= A/(1.6202*h). harbor-house supplies 54m against 148m needed (2.7x
  short), maple-court 56 vs 162; health-centre and programme-house are
  feasible. The corpus plateau is predicted by frontage deficit alone.
- Crinkliness is tiered SOFT but 60-100% of its fails are zero-exposure,
  which is topological, so §37.1's tiered comparator is mis-informed about
  the largest fail category.
- The missing-space cascade emits one extra fail per declared size/width/
  proportion key, so under 0.5^n a missing room is weighted 4x differently
  depending on patterns.config verbosity.

Filed as homemaker-py-ssz, hxi, tdp, gvb, 1i8 (plus bdf for the pre-existing
fresh-clone test failures found en route).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB

DESIGN.md

@@ -4834,3 +4834,161 @@ connectivity/level-placement/geometry problem, not an adjacency-graph one —
 future effort on that plateau (`homemaker-py-2g7.7`'s LLM repair operator, or
 a level-connectivity-targeted operator) is better aimed than a graph-dual
 construction pass would have been.
+
+## 38. The plateau is an objective-gradient problem, not a search problem (`homemaker-py-ssz`/`hxi`/`tdp`/`gvb`/`1i8`) — measured 2026-08-25
+
+Independent review of why the search "finds solutions that are clearly not the
+best and gets stuck in local minima", prompted by the §37 scoreboard: *every*
+fail-count win of Phases 6–8 was a construction/objective-honesty lever and
+*every* search-machinery lever (§11.4 grade, §11.5 niching/restarts, §11.8
+tournament-k, §14 islands, §16 annealing, §29/§30 beam, §27 bubble, §34
+autodiff, §37.7 CP-SAT) was null or negative. That pattern is itself the
+finding: eight independent attempts to improve the *search* all failed, which
+is what you would expect if the search is working correctly and the
+**objective's gradient points away from good buildings**.
+
+Reproduce everything below with `experiments/diag_exposure_frontage.py`
+(`frontage` / `exposure` / `value` reports; no search run required).
+
+### 38.1 Zero-exposure leaves score a hard quality of 0 (`homemaker-py-ssz`)
+
+`fitness.quality_uncrinkliness` computes `crink = area_outside / area` and
+returns a hard `0.0` when `area_outside == 0` — a leaf with no daylit wall
+(no non-`private`/`fortified` external edge, no adjacent uncovered outside
+leaf). This is the mathematically consistent limit of the formula
+(`1/crink → ∞`, and `gaussian(∞, …) → 0`), so it is a **faithful port, not a
+porting bug** — but its consequences were never traced:
+
+- `evaluate_leaf` **multiplies** factors into `quality`, and `process_storey`
+  accumulates `value += quality * rate * area`. A buried leaf therefore
+  contributes **exactly zero value** while still adding cost.
+- So the objective cannot distinguish a buried room that is perfectly sized
+  from one that is absurd. Both score zero. The only thing the objective can
+  still see about a buried room is that it costs money.
+
+Measured share of interior/covered leaves that are zero-exposure, under the
+driver's real default stack (`leaf_sharing`, `depth_balanced`,
+`interior_outside`, `collapse_insearch`), 3 constructed seeds each:
+
+| programme | zero-exposure | wrong-ratio | ok |
+|---|---|---|---|
+| harbor-house | **36 (46%)** | 24 | 18 |
+| health-centre | **35 (45%)** | 0 | 43 |
+| maple-court | **83 (56%)** | 44 | 22 |
+
+On a converged run (`homemaker-evolve init.dom --budget 20000 --seed 1`,
+harbor-house, 57 fails) **14 of 17 crinkliness fails are zero-exposure**, and
+roughly 470 m² of the 721 m² ground-floor plate sits at zero value.
+
+### 38.2 Buried circulation and outside space are negative-value (`homemaker-py-hxi`)
+
+Programme rooms are pinned in place by the missing-space fail cascade — but
+nothing pins circulation (`C`) or outside (`O`) leaves, which carry no
+`count:` requirement. Deleting a buried one is therefore a pure win.
+Measured on a constructed harbor-house seed (`value` report):
+
+| deleted leaf | score change | fail change |
+|---|---|---|
+| buried `O`, 45.8 m² | **×85.6 BETTER** | 92 → 85 |
+| buried `C`, 46.6 m² | **×61.6 BETTER** | 92 → 86 |
+| buried `k1`, 30.3 m² | ×0.00 worse | 92 → 107 |
+| buried `da1`, 61.7 m² | ×0.00 worse | 92 → 107 |
+
+**The search is rewarded, by roughly two orders of magnitude, for deleting the
+circulation spine.** Observed live: in the 20 000-eval harbor-house run above,
+`undivide`/`core_undivide` appear 16 times in the improvement log.
+
+This retro-explains three prior results as one mechanism, and suggests two of
+them were measuring a broken gradient rather than a bad idea:
+
+- **§18 graded circulation-connectivity — NEGATIVE.** A secondary comparator
+  key cannot beat a ×60 primary-scalar gradient pulling the other way.
+- **§21/§22 `bridge_circulation` — mixed/null.** The operator inserts exactly
+  the corridor leaves the objective then pays to delete.
+- **The 3M-eval run's `level 0/1 not connected` hard fails surviving >1M
+  evals.** Not a stubborn search; a correctly-followed gradient.
+
+### 38.3 The binding constraint is a frontage budget (`homemaker-py-tdp`)
+
+Closed form, no search needed. Crinkliness fails when `1/crink > 1.6202`
+(solving `gaussian(x, 1, 5/6, 1.1/3) = FAIL_THRESHOLD`), and `crink = L·h/A`,
+so every interior leaf needs exposed wall `L ≥ A/(1.6202·h)` — per storey,
+`A_storey/4.86` metres at `h = 3`. `area_outside` skips `private` and
+`fortified` perimeter edges, and harbor-house/maple-court mark **half their
+plot perimeter `private`**:
+
+| programme | daylit frontage | needed per built storey | verdict | observed floor |
+|---|---|---|---|---|
+| harbor-house | 54 m | 148 m | **2.7× short** | plateaus 30–40 fails |
+| maple-court | 56 m | 162 m | **2.9× short** | plateaus 74–84 fails |
+| health-centre | 43 m | 41 m | feasible | §32 clean null |
+| programme-house | 24 m | 12 m | 2× surplus | **1 fail** (12k evals) |
+
+**The corpus fail-count plateau is predicted by frontage deficit alone.** The
+two programmes that are frontage-short are exactly the two that plateau; the
+two with surplus are the two that effectively solve. Causal check
+(`experiments/diag_exposure_frontage.py`, 6 seeds): relabelling harbor's two
+`private` edges as open — identical geometry, identical programme, perimeter
+labels only — cuts zero-exposure leaves **52% → 19%** and seeder crinkliness
+fails 16.5 → 12.0.
+
+The deficit *is* closable: ~108 m² per storey (≈15% of the plate) given over
+to ~3 m courtyard slots would satisfy harbor's budget while still leaving
+1226 m² of floor against an 835 m² programme demand. **But that is precisely
+the move the objective punishes en route** — a small new `O` leaf is itself
+buried, hence zero-value, hence worth ×85 to delete. The payoff only arrives
+once a slot is wide enough and long enough to serve many rooms at once. Every
+intermediate step is punished; the reward is behind a coordinated multi-leaf
+move. That is the valley, and no amount of population diversity crosses it
+when the gradient opposes you the whole way — which is why §11.5, §14, §16 and
+§37.10-style diversity levers were always going to be null here.
+
+### 38.4 Crinkliness is mis-tiered as SOFT (`homemaker-py-gvb`)
+
+`fitness._SOFT_FAIL_MARKERS` lists `" crinkliness"` as SOFT, defined in §37.1
+as "a continuous per-leaf shape metric the inner-loop ratio solve can improve
+without changing the tree". **False for the zero-exposure case**: no ratio
+assignment can give a buried leaf a wall, so by this document's own definition
+it is HARD. Zero-exposure share of crinkliness fails: harbor-house 60%,
+maple-court 65%, health-centre 100%, and 82% (14/17) on the converged run.
+Since crinkliness is the single largest fail category (48% of the residual,
+§13.11), **the §37.1 tiered comparator is mis-informed about the largest block
+of fails it sorts** — `n_soft` is not the polish-budget signal it was designed
+to be. Fix: emit a distinct fail string for the zero-exposure case (which also
+makes the condition visible in `.fails` output, where today it is
+indistinguishable from an ordinary shape miss) and tier it HARD.
+
+### 38.5 The missing-space cascade is weighted by YAML verbosity (`homemaker-py-1i8`)
+
+`graph.check_space_counts` emits, per missing room instance, 2 base fails
+(`missing required space: X` + `(critical)`) plus one `would need <check>`
+placeholder for **each of `size`/`width`/`proportion` the programme happens to
+declare** — `has_size` is literally `"size" in c` from the YAML. So a missing
+room costs 3–5 fails depending only on how many optional keys the author
+typed, and under `value *= 0.5 ** len(failures)` that is a **4× difference in
+fitness weight between two single rooms**. In programme-house, missing `b1`
+(declares all three) = 5 fails = 1/32; missing `t2` (declares `size` only) =
+3 fails = 1/8. The tiered comparator inherits it: `n_hard` is dominated by
+these cascades, so the primary search key is weighted by config verbosity.
+
+### 38.6 Consequences for the Phase 9 plan
+
+§37 track 1 ("no ground truth … the residual taxonomy may be miscalibrated
+rather than unmet") was aimed at the right target, and §38.3 supplies a cheap
+way to test it that does **not** need `2g7.1`'s traced human plans: the
+frontage bound is a pre-flight feasibility check computable from a plot and a
+programme alone. Two of the four corpus programmes fail it by ~3×, which means
+a share of the residual those runs are being judged on **is not reachable at
+all** — and any A/B measured against that residual has been measuring, in
+part, an unsatisfiable constraint.
+
+Tracks 2 and 3 (cheaper evaluation, exact sub-solvers) remain sound but are
+orthogonal: making an evaluation 97× faster, or a labelling exact, does not
+change which direction the objective points. Recommended ordering is now
+`ssz` → `hxi`/`gvb` (restore a value gradient for interior space, re-tier),
+then `tdp` (ship the pre-flight bound and re-baseline the corpus), and only
+then resume `2g7.9`/`2g7.10`. In particular `2g7.7` (LLM repair operator at
+stagnation) is worth deferring until after `ssz`: an LLM asked to propose a
+valley-crossing multi-edit against an objective that pays ×85 to delete the
+corridor it just inserted will have its work reverted by the next selection
+step.

experiments/diag_exposure_frontage.py

@@ -0,0 +1,237 @@
+"""Exposure / frontage diagnostic for the crinkliness residual.
+
+Evidence for `homemaker-py-ssz` / `hxi` / `tdp` (DESIGN.md §38). Three reports,
+none of which needs a search run:
+
+1. ``exposure``  — decompose crinkliness failures into ZERO-EXPOSURE (the leaf
+   has no daylit wall at all, so ``quality_uncrinkliness`` returns a hard 0.0
+   and the leaf's whole quality product collapses to zero) vs. wrong-ratio
+   (exposed, but off the uncrinkliness target). Run on constructed seeds under
+   the driver's real default stack, or on a ``.dom`` from disk.
+
+2. ``value``     — measure what a buried leaf is worth to the objective, by
+   deleting one (undividing its parent) and re-scoring. Circulation/outside
+   leaves carry no missing-space requirement, so nothing offsets their removal.
+
+3. ``frontage``  — the closed-form feasibility bound. Crinkliness fails when
+   ``1/crink > X`` with ``crink = L*h/A``, so every interior leaf needs exposed
+   wall ``L >= A/(X*h)``; per storey that is ``A_storey/(X*h)`` metres. Compare
+   against the daylit plot perimeter (``area_outside`` skips ``private`` and
+   ``fortified`` edges) to get the deficit.
+
+Usage::
+
+    python experiments/diag_exposure_frontage.py frontage
+    python experiments/diag_exposure_frontage.py exposure examples/harbor-house
+    python experiments/diag_exposure_frontage.py exposure --dom out.dom examples/harbor-house
+    python experiments/diag_exposure_frontage.py value examples/harbor-house
+"""
+
+from __future__ import annotations
+
+import argparse
+import copy
+import math
+from pathlib import Path
+
+import numpy as np
+import yaml
+
+from homemaker_layout import dom as dom_mod
+from homemaker_layout import driver, fitness, geometry
+from homemaker_layout import graph as graph_mod
+from homemaker_layout import operators, programme
+
+CORPUS = ["examples/harbor-house", "examples/maple-court",
+          "examples/health-centre", "examples/programme-house"]
+
+
+def fail_bounds() -> tuple[float, float]:
+    """Solve ``gaussian(x, 1, 5/6, 1.1/3) == FAIL_THRESHOLD`` for x.
+
+    Returns ``(buried_above, exposed_below)``: a leaf fails crinkliness when
+    ``1/crink`` exceeds the first (too buried) or falls below the second (too
+    exposed). Derived from the real constants, never hard-coded.
+    """
+    b, c = fitness.CONF_DEFAULTS["uncrinkliness"]
+    k = math.sqrt(-2 * c * c * math.log(fitness.FAIL_THRESHOLD) / math.log(fitness._E))
+    return b + k, b - k
+
+
+def _fit(progdir: str) -> fitness.Fitness:
+    """Evaluator configured exactly as ``driver.search`` builds it by default."""
+    ov = driver._overrides_for(leaf_sharing=True, superpose=False, max_share=None,
+                               conn_grade=False, collapse_insearch=True,
+                               multi_use=False)
+    conf, cost = fitness.load_config(progdir, overrides=ov)
+    return fitness.Fitness(conf, cost)
+
+
+def _seed(progdir: str, seed: int) -> dom_mod.Node:
+    """One constructed seed under ``driver.search``'s own default arguments."""
+    reqs = programme.load_programme_dir(progdir)
+    return operators.constructive_topology(
+        dom_mod.load(f"{progdir}/init.dom"), reqs, np.random.default_rng(seed),
+        sorted(reqs) + ["C", "O"],
+        min_storeys=programme.storey_minimum(progdir),
+        adjacency_aware=True, proportion_aware=True, circ_divisor=3,
+        leaf_sharing=True, leaf_share_factor=3, depth_balanced=True,
+        interior_outside=True, outside_divisor=3)
+
+
+def _exposure_rows(fit: fitness.Fitness, root: dom_mod.Node) -> list[tuple]:
+    """Per interior/covered leaf: (level, id, type, area, exposed_area, quality).
+
+    Reproduces the state ``_evaluate_full`` reaches at storey processing —
+    in-search collapse, preprocess, merge — so the numbers match a real eval.
+    """
+    tree = copy.deepcopy(root)
+    geometry.clear_cache()
+    dom_mod.canonicalize_shares(tree)
+    if fit._collapse_insearch:
+        fit.collapse_global(tree, adjacency=True, objective="threshold",
+                            preserve_public_access=True, iters=3)
+    fit.preprocess_building(tree)
+    dom_mod.merge_divided(tree)
+    geometry.clear_cache()
+    graphs = graph_mod.build_graphs(tree, fit.conf("door_width") or 1.2)
+
+    rows = []
+    for li, lvl in enumerate(dom_mod.levels(tree)):
+        for leaf in lvl.leaves():
+            if dom_mod.is_outside(leaf) and not dom_mod.is_covered(leaf):
+                continue  # exempt by quality_uncrinkliness
+            rows.append((li, leaf.id, leaf.type, geometry.area(leaf),
+                         fit.area_outside(leaf, graphs[li], {}),
+                         fit.quality_uncrinkliness(leaf, graphs[li], {})))
+    return rows
+
+
+def report_exposure(progdir: str, seeds: range, dom_path: str | None) -> None:
+    fit = _fit(progdir)
+    roots = ([dom_mod.load(dom_path)] if dom_path
+             else [_seed(progdir, s) for s in seeds])
+    ok = zero = ratio = 0
+    n_fails = n_crink = 0
+    for root in roots:
+        _, fails = fit.score_with_fails(copy.deepcopy(root))
+        n_fails += len(fails)
+        n_crink += sum("crinkliness" in f for f in fails)
+        for _, _, _, _, exposed, q in _exposure_rows(fit, root):
+            if exposed == 0:
+                zero += 1
+            elif q < fitness.FAIL_THRESHOLD:
+                ratio += 1
+            else:
+                ok += 1
+    total = ok + zero + ratio
+    label = dom_path or f"{len(roots)} constructed seeds"
+    print(f"=== {progdir}  ({label})")
+    print(f"  mean fails/design {n_fails / len(roots):.1f}, of which crinkliness "
+          f"{n_crink / len(roots):.1f}")
+    print(f"  interior leaves: ok={ok}  ZERO-EXPOSURE={zero}  wrong-ratio={ratio}")
+    if total:
+        print(f"  -> {100 * zero / total:.0f}% of interior leaves have NO daylit wall: "
+              f"hard quality=0, unreachable by any ratio assignment")
+
+
+def report_value(progdir: str, seed: int, limit: int) -> None:
+    """What is a buried leaf worth? Delete one and re-score."""
+    fit = _fit(progdir)
+    root = _seed(progdir, seed)
+    base_score, base_fails = fit.score_with_fails(copy.deepcopy(root))
+    print(f"=== {progdir} seed {seed}: baseline score {base_score:.4g}, "
+          f"{len(base_fails)} fails")
+
+    buried = [(li, lid, typ) for li, lid, typ, _, exposed, _
+              in _exposure_rows(fit, root) if exposed == 0]
+    print(f"  {len(buried)} buried leaves; deleting each (undivide its parent):")
+
+    tried = 0
+    for li, lid, typ in buried:
+        if tried >= limit:
+            break
+        cand = copy.deepcopy(root)
+        lvls = dom_mod.levels(cand)
+        if li >= len(lvls):
+            continue
+        node = lvls[li].by_id(lid)
+        if node is None or node.parent is None:
+            continue
+        parent = node.parent
+        if parent.below is not None and parent.below.divided:
+            continue  # inherited cut, not owned here
+        sibling = parent.right if parent.left is node else parent.left
+        if sibling is None or sibling.divided:
+            continue
+        parent.division = None
+        parent.left = parent.right = None
+        parent.type = sibling.type
+        dom_mod.link(cand)
+        geometry.clear_cache()
+        score, fails = fit.score_with_fails(copy.deepcopy(cand))
+        tried += 1
+        verdict = "BETTER" if score > base_score else "worse"
+        print(f"    delete {lid:<8s} type={typ:<5s}: score x{score / base_score:>8.2f} "
+              f"({verdict}), fails {len(base_fails)} -> {len(fails)}")
+
+
+def report_frontage(progdirs: list[str]) -> None:
+    x_buried, x_exposed = fail_bounds()
+    print(f"crinkliness fails when 1/crink > {x_buried:.4f} (buried) "
+          f"or < {x_exposed:.4f} (over-exposed)")
+    print(f"=> every interior leaf needs exposed wall L >= A / ({x_buried:.4f} * h)\n")
+
+    for progdir in progdirs:
+        seed = yaml.safe_load(open(f"{progdir}/init.dom"))
+        corners, per = seed["node"], (seed.get("perimeter") or {})
+        height = seed.get("height") or 3.0
+        n = len(corners)
+        edges = [math.hypot(corners[(i + 1) % n][0] - corners[i][0],
+                            corners[(i + 1) % n][1] - corners[i][1]) for i in range(n)]
+        daylit = sum(e for k, e in zip("abcd", edges)
+                     if (per.get(k) or "").lower() not in ("private", "fortified"))
+        area = abs(sum(corners[i][0] * corners[(i + 1) % n][1]
+                       - corners[(i + 1) % n][0] * corners[i][1]
+                       for i in range(n))) / 2
+        reqs = programme.load_programme_dir(progdir)
+        n_storeys = max(programme.n_storeys_required(reqs),
+                        programme.storey_minimum(progdir))
+        demand = sum(r.size * r.count for r in reqs.values())
+        needed = area / (x_buried * height)
+
+        print(f"=== {Path(progdir).name}")
+        print(f"  plot {area:.0f} m2, perimeter {sum(edges):.0f} m, "
+              f"{n_storeys} storeys, h={height}")
+        print(f"  perimeter {per} -> daylit frontage {daylit:.0f} m")
+        print(f"  a fully built storey needs {needed:.0f} m exposed wall; "
+              f"plot supplies {daylit:.0f} m "
+              f"-> {needed / max(daylit, 1e-9):.1f}x short"
+              if needed > daylit else
+              f"  a fully built storey needs {needed:.0f} m exposed wall; "
+              f"plot supplies {daylit:.0f} m -> FEASIBLE")
+        print(f"  programme demands {demand:.0f} m2 over {n_storeys} storeys "
+              f"({demand / n_storeys:.0f} m2/storey of {area:.0f} m2 plot)\n")
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("report", choices=("exposure", "value", "frontage"))
+    ap.add_argument("progdir", nargs="?", default=None)
+    ap.add_argument("--dom", default=None, help="score this .dom instead of seeds")
+    ap.add_argument("--seeds", type=int, default=3)
+    ap.add_argument("--limit", type=int, default=5, help="deletions to try (value)")
+    args = ap.parse_args()
+
+    if args.report == "frontage":
+        report_frontage([args.progdir] if args.progdir else CORPUS)
+    elif args.report == "exposure":
+        for d in ([args.progdir] if args.progdir else CORPUS):
+            report_exposure(d, range(args.seeds), args.dom)
+    else:
+        report_value(args.progdir or CORPUS[0], seed=0, limit=args.limit)
+
+
+if __name__ == "__main__":
+    main()