homemaker-py-91f: residual diagnostic on the current full default stack

Re-ran the §13.1/§13.2-style per-leaf fail-breakdown diagnostic on real
driver.search_staged runs (budget 20000, seeds 0-2, harbor-house and
maple-court) under the current full default stack (leaf-sharing x3,
depth-balanced, interior-O, share-aware edge cap) -- never decomposed by
category since those defaults were flipped on.

Finding: crinkliness (48%) and size (20.6%) now dominate the residual on
both programmes (~69% combined); construction-completeness fails
(missing space, adjacency, level, connectivity) are down to a small
tail (<=6% each). This revises erc.1's old recommendation to deprioritise
compactness-cuts in favour of leaf-sharing -- leaf-sharing is now fully
deployed and crinkliness is proportionally more dominant than ever, so
DESIGN.md §13.11 recommends reopening a compactness/crinkliness-targeted
construction lever as the next concrete step.

Also files two bugs found while validating the methodology: dumping and
reloading a .dom under leaf_sharing+collapse_insearch does not reproduce
the search's own in-process fail count (homemaker-py-iio), and
run_staged_search.py's own sanity rescore omits the collapse_insearch
override (homemaker-py-7ua). experiments/run_and_capture_91f.py sidesteps
this by capturing the true in-process fails list instead of rescoring
from disk; experiments/diag_residual_91f.py tallies fail categories from
those sidecars.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01R8agJBT2ZpmF3ErW7wi2wY

DESIGN.md

@@ -2140,6 +2140,87 @@ Default-OFF parity holds: `overrides=None` leaves `load_config` byte-identical a
 `_share_rooms` is never reached. Smoke-checked end-to-end on harbor-house (sharing
 on 37 fails vs `--no-leaf-sharing` 95 at budget 160). 233 tests pass.
 
+### 13.11 Residual diagnostic on the current full default construction stack (`homemaker-py-91f`) — DONE
+
+The §13.1/§13.2 (`erc.1`/`erc.2`) per-leaf diagnostics predate the depth-balanced
++ leaf-sharing synergy flip (`erc.7`) and the share-aware edge cap flip (`rq2`/
+`x3b`) — the current §13.9 floor (harbor 31.0, maple 74.0) had never been
+decomposed by fail category. Unlike `erc.1` (which scores a single constructed
+seed at target geometry, a cheap proxy), this reads the actual best individual
+from a REAL `driver.search_staged` run — budget 20000, seeds 0/1/2, harbor-house
+and maple-court, the full default stack (`leaf_sharing`/`leaf_share_factor=3`,
+`depth_balanced`, `interior_outside`/`outside_divisor=3`, `share_edge_cap`
+default-on under sharing) — the actual reported floor, not a proxy.
+
+**Methodology note — a scoring pitfall found along the way.** The obvious
+approach (dump each run's best to `.dom`, reload, rescore with matching conf)
+gives a WRONG, but stable and easy-to-miss, fail count once `collapse_insearch`
+is doing real relabelling work: on harbor-house seed 0 the search itself
+reported 37 fails, and `copy.deepcopy(r.best.root)` rescored immediately
+in-process reproduces 37 exactly, but `dom.dump` + `dom.load` + rescore of the
+*same* topology gives a stable 53 — 15 extra `missing`/`adjacency`/`level`
+fails for a level-0 `count: 3` code that collapse-relabelling satisfies in the
+live tree but that is not present as a literal leaf type once round-tripped.
+Root cause not yet found (hash-seed randomness and float round-trip loss are
+both ruled out); filed as `homemaker-py-iio` (P2). A narrower, separate bug —
+`run_staged_search.py`'s own final sanity rescore omits the `collapse_insearch`
+override entirely, so its own "MISMATCH" line cannot be trusted whenever
+leaf-sharing is on — is filed as `homemaker-py-7ua` (P3). This diagnostic
+sidesteps both: `experiments/run_and_capture_91f.py` scores
+`copy.deepcopy(r.best.root)` immediately after `search_staged` returns, and
+writes the fails list to a `*.fails.json` sidecar (verified `rescore_match` on
+all 6 runs); `experiments/diag_residual_91f.py` tallies fail categories from
+those sidecars, never rescoring a `.dom` from disk.
+
+**Result (mean fails/seed; category % of all fails, combined):**
+
+| programme | seeds (fails) | mean | vs §13.9 cited floor |
+|---|---|---|---|
+| harbor-house | 37, 33, 30 | 33.3 | 31.0 |
+| maple-court | 82, 84, 78 | 81.3 | 74.0 |
+
+(Both a bit above the cited floor, as expected — a single staged run per seed
+here vs. whatever selection produced the cited numbers; same order of
+magnitude, good sanity check that the stack is wired correctly.)
+
+| category | combined n | % |
+|---|---|---|
+| **crinkliness** | 165 | **48.0%** |
+| **size** | 71 | **20.6%** |
+| adjacency (not adjacent) | 20 | 5.8% |
+| proportion | 13 | 3.8% |
+| access | 12 | 3.5% |
+| edge too long (outside) | 12 | 3.5% |
+| missing (adjacency/level/vertical cascade) | 12 | 3.5% |
+| circulation not connected | 9 | 2.6% |
+| edge too long (wall) | 9 | 2.6% |
+| missing required space | 6 | 1.7% |
+| (remaining: too-many-spaces, covered-outside, stairs, width, public-access) | 12 | 3.4% |
+
+Per-programme shares are consistent (crinkliness 43%/size 21% on harbor-house,
+crinkliness 50%/size 20.5% on maple-court) — this is not an artefact of one
+programme.
+
+**VERDICT — shape-intrinsic fails (crinkliness + size ≈ 69% of the residual)
+now completely dominate; construction-completeness fails (missing space,
+adjacency, level, vertical connectivity — the failure modes the §11–§13 series
+of construction levers targeted) are now a small tail, ≤6% each.** This
+revises the `erc.1` recommendation. `erc.1` (§13.1) found per-leaf crinkliness
+FLAT vs. slicing density and concluded the floor was intrinsic to leaf COUNT,
+prioritising leaf-sharing (`erc.3`) over compactness-aware cuts (`erc.5`,
+deprioritised: "cuts are already squarest ... little headroom at fixed count").
+Leaf-sharing (plus depth-balancing, interior-O, and the edge cap) is now fully
+deployed as the default stack, and crinkliness is not just still present but
+*more* dominant proportionally than in any earlier per-category breakdown in
+this document (cf. §7's 27/85 and §9's 346/939-ish shares) — the "reduce leaf
+count" avenue has been substantially exploited by the current stack, yet the
+per-leaf shape tax persists and is now, by a wide margin, the single largest
+lever available. **Recommendation: reopen `erc.5`-style compactness-aware
+cutting (or a crinkliness-targeted construction/mutation lever specifically,
+since crinkliness outweighs size ~2.3:1) as the next concrete construction
+lever** — the same diagnostic-first logic that turned §13.7's edge-too-long
+finding directly into `hph`.
+
 ## 14. Island model: multi-run recombination (`homemaker-py-psk`) — DONE (null)
 
 **Lever (user-proposed).** Perl Urb ran the search many times and kept the best,

experiments/diag_residual_91f.py

@@ -0,0 +1,140 @@
+#!/usr/bin/env python3
+"""Diagnostic (homemaker-py-91f): per-leaf fail-category breakdown of the
+CURRENT full default construction stack's residual, on harbor-house and
+maple-court.
+
+The last such per-leaf diagnostic (diag_leaf_shapefail.py / erc.1-erc.2)
+predates the depth-balanced+leaf-sharing synergy default flip (erc.7) and the
+share-aware edge cap default flip (homemaker-py-rq2/x3b) -- the current §13.9
+floor (harbor 31.0, maple 74.0) has never been decomposed by fail category.
+
+Reads the ``*.fails.json`` sidecars written by ``run_and_capture_91f.py``
+(a REAL staged search, budget 20000, seeds 0/1/2, full default stack:
+leaf_sharing/leaf_share_factor=3/depth_balanced/interior_outside/
+outside_divisor=3/share_edge_cap) -- the actual reported floor, not a proxy.
+
+IMPORTANT: this does NOT rescore the .dom files from disk. homemaker-py-iio
+found that reloading a dumped .dom under leaf_sharing+collapse_insearch and
+rescoring it does not reliably reproduce the search's own in-process
+n_fails (collapse_global's cell-relabelling converges differently after a
+dump/reload round trip, for reasons not yet root-caused). The .fails.json
+sidecar captures the TRUE fails list straight off the in-process
+``driver.search_staged`` result, which IS verified to reproduce the
+search's own reported n_fails exactly (see run_and_capture_91f.py's
+rescore_match field, true on all runs). Read-only: does not change
+behaviour or run any search itself.
+
+Usage:
+  python3 experiments/diag_residual_91f.py <dir-with-{prog}_s{seed}.fails.json files>
+"""
+
+from __future__ import annotations
+
+import json
+import re
+import sys
+from collections import Counter
+from pathlib import Path
+
+ROOT = Path(__file__).resolve().parents[1]
+PROGRAMMES = ["harbor-house", "maple-court"]
+SEEDS = (0, 1, 2)
+
+SHAPE_SUFFIXES = ("perpendicular", "proportion", "size", "width", "crinkliness",
+                  "access")
+
+
+def categorize(f: str) -> str:
+    if "too many spaces" in f:
+        return "too many spaces (over-provided)"
+    if "too few stairs" in f:
+        return "too few stairs"
+    if "too many stairs" in f:
+        return "too many stairs"
+    if "missing required space" in f:
+        return "missing required space"
+    if re.match(r"^missing \S+: would need", f):
+        return "missing (cascade: adjacency/level/vertical/quality)"
+    if " not adjacent to " in f:
+        return "adjacency (not adjacent)"
+    if "on wrong level" in f:
+        return "wrong level"
+    if "not connected to" in f and "below" in f:
+        return "vertical connectivity"
+    if "outside edge too long" in f:
+        return "edge too long (outside)"
+    if "edge too long" in f:
+        return "edge too long (wall)"
+    if "unsupported covered outside" in f:
+        return "covered outside (unsupported)"
+    if "covered outside above ground" in f:
+        return "covered outside (above ground)"
+    if "not connected" in f:
+        return "circulation not connected"
+    if "no outside space" in f:
+        return "level: no outside space"
+    if "no outside public access" in f:
+        return "no outside public access"
+    if "staircase volume" in f:
+        return "staircase volume"
+    if "storey limit" in f:
+        return "storey limit"
+    if "storey minimum" in f:
+        return "storey minimum"
+    for suf in SHAPE_SUFFIXES:
+        if f.endswith(" " + suf):
+            return suf
+    return f"other: {f[:40]}"
+
+
+def main() -> int:
+    if len(sys.argv) < 2:
+        print("usage: diag_residual_91f.py "
+              "<dir-with-{prog}_s{seed}.fails.json files>", file=sys.stderr)
+        return 1
+    indir = Path(sys.argv[1])
+
+    grand = Counter()
+    for name in PROGRAMMES:
+        tally = Counter()
+        n_fails_total = 0
+        n_seeds = 0
+        for s in SEEDS:
+            jsonfile = indir / f"{name}_s{s}.fails.json"
+            if not jsonfile.exists():
+                print(f"  (skip {jsonfile.name}: not found yet)", file=sys.stderr)
+                continue
+            data = json.loads(jsonfile.read_text())
+            if not data.get("rescore_match", False):
+                print(f"  WARNING: {jsonfile.name} rescore_match=False -- "
+                      f"in-process rescore did not match search's own "
+                      f"n_fails, investigate before trusting this seed",
+                      file=sys.stderr)
+            fails = data["fails"]
+            n_seeds += 1
+            n_fails_total += len(fails)
+            for f in fails:
+                tally[categorize(f)] += 1
+
+        if n_seeds == 0:
+            continue
+        print(f"=== {name}  ({n_seeds} seed(s), {n_fails_total/n_seeds:.1f} "
+              f"fails/seed avg) ===")
+        total = sum(tally.values())
+        for cat, n in tally.most_common():
+            pct = 100.0 * n / total if total else 0.0
+            print(f"  {n:4d}  ({pct:4.1f}%)  {cat}")
+        print()
+        grand.update(tally)
+
+    if grand:
+        print("=== COMBINED (both programmes, all seeds) ===")
+        total = sum(grand.values())
+        for cat, n in grand.most_common():
+            pct = 100.0 * n / total if total else 0.0
+            print(f"  {n:4d}  ({pct:4.1f}%)  {cat}")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

experiments/run_and_capture_91f.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""homemaker-py-91f support script: run the §13.9 full-default-stack staged
+search and capture the TRUE in-process fail list (not a post-hoc rescore).
+
+Investigation finding (see bd issue filed alongside 91f): reloading a
+dumped .dom and rescoring it with matching leaf_sharing/collapse_insearch
+conf does NOT reliably reproduce driver.search_staged's own reported
+n_fails once collapse_insearch's cell-relabelling is doing real work --
+scoring `copy.deepcopy(r.best.root)` immediately in-process (before any
+dom.dump/dom.load round trip) DOES reliably reproduce it (verified: exact
+match across 5 repeats and against the live search's own log). So this
+script never rescores from disk -- it captures the fails list right off
+the in-memory search result, and only dumps the .dom for reference.
+
+Usage:
+  python3 experiments/run_and_capture_91f.py <programme> <seed> <outdir>
+"""
+from __future__ import annotations
+
+import copy
+import json
+import sys
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
+from homemaker_layout import dom, driver, fitness  # noqa: E402
+
+ROOT = Path(__file__).resolve().parents[1]
+OVERRIDES = {"leaf_sharing": True, "share_edge_cap": True, "collapse_insearch": True}
+
+
+def main() -> int:
+    prog = sys.argv[1]
+    seed = int(sys.argv[2])
+    outdir = Path(sys.argv[3])
+    outdir.mkdir(parents=True, exist_ok=True)
+
+    pdir = ROOT / "examples" / prog
+    seed_root = dom.load(str(pdir / "init.dom"))
+
+    t0 = time.perf_counter()
+    r = driver.search_staged(
+        seed_root, pdir, budget=20000, pop_size=16, child_budget=80,
+        seed_budget=300, stage1_frac=0.4, base_p=0.15, p_crossover=0.2,
+        seed=seed, n_workers=1,
+        leaf_sharing=True, leaf_share_factor=3,
+        depth_balanced=True, interior_outside=True, outside_divisor=3,
+    )
+    elapsed = time.perf_counter() - t0
+
+    conf, cost = fitness.load_config(pdir, overrides=OVERRIDES)
+    fit = fitness.Fitness(conf, cost)
+    score, fails = fit.score_with_fails(copy.deepcopy(r.best.root))
+
+    match = (len(fails) == r.best.n_fails)
+    print(f"{prog} seed={seed}: elapsed={elapsed:.1f}s search_n_fails="
+          f"{r.best.n_fails} in_process_rescore_n_fails={len(fails)} "
+          f"match={match}", file=sys.stderr)
+
+    dom.dump(r.best.root, str(outdir / f"{prog}_s{seed}.dom"))
+    with open(outdir / f"{prog}_s{seed}.fails.json", "w") as fh:
+        json.dump({
+            "programme": prog, "seed": seed, "elapsed_s": elapsed,
+            "search_n_fails": r.best.n_fails, "search_fitness": r.best.fitness,
+            "rescore_n_fails": len(fails), "rescore_match": match,
+            "fails": fails,
+        }, fh, indent=2)
+
+    return 0 if match else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())