ssz: 61% of reported crinkliness fails are not defects; two corrections
Correction 1: 38.8's A/B scores every arm under stock urb, which is only sound if stock is ground truth -- and stock is what this section shows is miscalibrated. Scoring the repair under the objective it repairs penalises it for repairing. That result is not evidence against usage_daylight. The measurement that matters needs no search: are the emitted failures true? audit_crinkliness_truth.py classifies every stock crinkliness fail by the leaf's declared usage. 164 of 271 (61%) are reported against spaces that want no daylight -- stores, WCs, plant, corridors, covered courtyards -- and it holds on evolved artefacts (maple generated.dom: 64%), not just constructed seeds. Under value *= 0.5 ** len(failures) each one halves the fitness of a design that did nothing wrong. Correction 2: usage: is the wrong key, and usage_daylight as written mis-keys it. 39.7 established usage: as an ACCESS-requirement class. "Needs no special access" and "needs no window" are different questions, and the corpus separates them: usage: none is Waiting Room, Reception, Reception Office, Entrance Foyer -- a waiting room plainly wants daylight and DAYLIGHT_USAGES exempts it. The usage: bedroom half is right, but by luck of where the access axis happened to fall. The audit is robust to that error (61% -> 57% if none is reclassified), so the finding stands; the design does not. Daylight needs its own declared per-space axis, decided by the programme author as usage: was. usage_daylight stays default off and is not the shipping fix. The compact-side clip is the right shape for the factor, so the mechanism is retained pending a daylight: attribute to key it on. Higher-power A/B stopped: it was chasing a score against a flawed objective with a mis-keyed fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
This commit is contained in:
parent
6405ca1a59
commit
bb27a5e548
3 changed files with 194 additions and 0 deletions
55
DESIGN.md
55
DESIGN.md
|
|
@ -5229,6 +5229,61 @@ A/B: the daylight requirement is applied to rooms that architecturally do not
|
|||
want daylight, in two thirds of the buried population, and §38.6's contrary
|
||||
null was an artefact of three modes that never touched those leaves.
|
||||
|
||||
### 38.9 Two corrections to §38.8, and the measurement that matters (`homemaker-py-ssz`)
|
||||
|
||||
**Correction 1 — the A/B yardstick above is wrong.** §38.8 scores every arm
|
||||
under stock `urb`, reasoning that a permissive mode must not be allowed to win
|
||||
by deleting a fail category. That is sound only if stock is ground truth, and
|
||||
stock is exactly what this section shows is miscalibrated. Scoring the repair
|
||||
under the objective it repairs penalises it for repairing: stock counts a
|
||||
windowless broom cupboard as a failure, and the repair's whole purpose is to
|
||||
stop counting it. *Does the fix score well on the broken yardstick* is not a
|
||||
question worth answering, and the §38.8 A/B result should not be read as
|
||||
evidence against `usage_daylight`.
|
||||
|
||||
**The measurement that does matter** asks whether the emitted failures are
|
||||
*true*, and needs no search at all (`experiments/audit_crinkliness_truth.py`).
|
||||
Of the `crinkliness` failures the stock objective reports, classified by the
|
||||
leaf's declared usage:
|
||||
|
||||
| layout | crinkliness fails | on spaces that want no daylight |
|
||||
|---|---|---|
|
||||
| harbor-house, 3 constructed seeds | 67 | 41 (61%) |
|
||||
| maple-court, 3 constructed seeds | 112 | 68 (61%) |
|
||||
| health-centre, 3 constructed seeds | 20 | 10 (50%) |
|
||||
| harbor-house `generated.dom` (evolved) | 5 | 2 (40%) |
|
||||
| maple-court `generated.dom` (evolved) | 67 | 43 (64%) |
|
||||
| **overall** | **271** | **164 (61%)** |
|
||||
|
||||
**61% of the crinkliness failures the objective reports are not defects**, and
|
||||
it holds on evolved artefacts, not just seeds. Under
|
||||
`value *= 0.5 ** len(failures)` every one of them halves the fitness of a design
|
||||
that has done nothing wrong — a design is punished for putting the store in the
|
||||
middle of the plan, which is what a competent architect does. That is a
|
||||
correctness fault, and it is not contingent on any A/B.
|
||||
|
||||
**Correction 2 — `usage:` is the wrong key, and `usage_daylight` as written in
|
||||
§38.8 mis-keys it.** §39.7 established `usage:` as an *access-requirement*
|
||||
class. "Needs no special access" and "needs no window" are different questions,
|
||||
and the corpus separates them plainly:
|
||||
|
||||
- `usage: none` is **Waiting Room**, **Reception**, **Reception Office**,
|
||||
**Entrance Foyer** — a waiting room is a space people sit in for long
|
||||
stretches and plainly wants daylight, yet `DAYLIGHT_USAGES` exempts it;
|
||||
- `usage: bedroom` is where the GP consulting rooms, counselling rooms and
|
||||
staff offices live — all of which do want daylight, so that half is right,
|
||||
but it is right by luck of how the access axis happened to fall.
|
||||
|
||||
The audit is robust to the error (reclassifying `none` as wanting daylight
|
||||
moves the headline 61% → 57%), so the finding stands; the *design* does not.
|
||||
Daylight needs its own declared axis, per space, decided by the programme
|
||||
author exactly as `usage:` was — not derived from a different question that
|
||||
happens to correlate.
|
||||
|
||||
`usage_daylight` therefore stays default off and is **not** the shipping fix.
|
||||
It is retained as the mechanism — the compact-side clip is the right shape for
|
||||
the factor — pending a `daylight:` attribute to key it on.
|
||||
|
||||
|
||||
## 39. Config audit: requirements that actively fight the engine (`homemaker-py-ju3`) — measured 2026-08-25
|
||||
|
||||
|
|
|
|||
132
experiments/audit_crinkliness_truth.py
Normal file
132
experiments/audit_crinkliness_truth.py
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
"""Are the crinkliness failures the objective emits real defects? (`homemaker-py-ssz`)
|
||||
|
||||
Not an A/B. This asks a correctness question the search cannot answer: of the
|
||||
`crinkliness` failures the STOCK objective reports, how many are on a space
|
||||
that architecturally wants daylight at all?
|
||||
|
||||
A `crinkliness` fail says "this leaf has too little exposed wall for its area".
|
||||
For a bedroom or a living room that is a real defect. For a broom cupboard, a
|
||||
WC, a plant room, an internal corridor or a covered courtyard it is not -- those
|
||||
are ordinary buried architecture, and the fail is an artefact of applying one
|
||||
daylight requirement to every space regardless of use (DESIGN.md §38.8).
|
||||
|
||||
Every fail is classified by the leaf's DECLARED `usage:` (§39.7), so nothing
|
||||
here rests on how a code is spelled.
|
||||
|
||||
Usage::
|
||||
|
||||
python experiments/audit_crinkliness_truth.py
|
||||
python experiments/audit_crinkliness_truth.py --seeds 5
|
||||
python experiments/audit_crinkliness_truth.py --dom examples/harbor-house/generated.dom
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import copy
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from homemaker_layout import dom as dom_mod
|
||||
from homemaker_layout import driver, fitness, geometry
|
||||
from homemaker_layout import graph as graph_mod
|
||||
from homemaker_layout import operators, programme
|
||||
|
||||
CORPUS = ["examples/harbor-house", "examples/maple-court", "examples/health-centre"]
|
||||
|
||||
|
||||
def stock_fitness(progdir: str) -> fitness.Fitness:
|
||||
"""Stock objective -- `crinkliness_mode` left at its "urb" default."""
|
||||
ov = dict(driver._overrides_for(
|
||||
leaf_sharing=True, superpose=False, max_share=None, conn_grade=False,
|
||||
collapse_insearch=True, multi_use=False) or {})
|
||||
conf, cost = fitness.load_config(progdir, overrides=ov)
|
||||
return fitness.Fitness(conf, cost)
|
||||
|
||||
|
||||
def constructed(progdir: str, s: int) -> dom_mod.Node:
|
||||
reqs = programme.load_programme_dir(progdir)
|
||||
return operators.constructive_topology(
|
||||
dom_mod.load(f"{progdir}/init.dom"), reqs, np.random.default_rng(s),
|
||||
sorted(reqs) + ["C", "O"], min_storeys=programme.storey_minimum(progdir),
|
||||
adjacency_aware=True, proportion_aware=True, circ_divisor=3,
|
||||
leaf_sharing=True, leaf_share_factor=3, depth_balanced=True,
|
||||
interior_outside=True, outside_divisor=3)
|
||||
|
||||
|
||||
def audit(fit: fitness.Fitness, root: dom_mod.Node) -> collections.Counter:
|
||||
"""usage -> count, over the leaves that emit a stock `crinkliness` fail."""
|
||||
tree = copy.deepcopy(root)
|
||||
geometry.clear_cache()
|
||||
dom_mod.canonicalize_shares(tree)
|
||||
fit.preprocess_building(tree)
|
||||
dom_mod.merge_divided(tree)
|
||||
geometry.clear_cache()
|
||||
graphs = graph_mod.build_graphs(tree, fit.conf("door_width") or 1.2)
|
||||
|
||||
out: collections.Counter = collections.Counter()
|
||||
for li, lvl in enumerate(dom_mod.levels(tree)):
|
||||
groups = geometry.boundary_groups(lvl)
|
||||
for leaf in lvl.leaves():
|
||||
if dom_mod.is_outside(leaf) and not dom_mod.is_covered(leaf):
|
||||
continue
|
||||
if fit.quality_uncrinkliness(leaf, graphs[li], groups) >= fitness.FAIL_THRESHOLD:
|
||||
continue # not a failure
|
||||
out[fit.usage_of(leaf) or f"<generic {leaf.type}>"] += 1
|
||||
return out
|
||||
|
||||
|
||||
def report(label: str, tally: collections.Counter) -> tuple[int, int]:
|
||||
total = sum(tally.values())
|
||||
real = sum(n for u, n in tally.items() if u in programme.DAYLIGHT_USAGES)
|
||||
print(f"=== {label}: {total} crinkliness fails")
|
||||
if not total:
|
||||
print(" none\n")
|
||||
return 0, 0
|
||||
for usage, n in tally.most_common():
|
||||
verdict = ("REAL DEFECT" if usage in programme.DAYLIGHT_USAGES
|
||||
else "not a defect -- no daylight wanted")
|
||||
print(f" {usage:<18}{n:>4} {verdict}")
|
||||
print(f" -> {total - real}/{total} ({100 * (total - real) / total:.0f}%) "
|
||||
f"are reported against spaces that do not want daylight\n")
|
||||
return real, total
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--seeds", type=int, default=3)
|
||||
ap.add_argument("--corpus", nargs="+", default=CORPUS)
|
||||
ap.add_argument("--dom", nargs="*", default=[],
|
||||
help="also audit these evolved .dom files (programme dir inferred)")
|
||||
args = ap.parse_args()
|
||||
|
||||
print("Stock objective. Every fail classified by the leaf's declared usage.\n")
|
||||
grand_real = grand_total = 0
|
||||
|
||||
for progdir in args.corpus:
|
||||
fit = stock_fitness(progdir)
|
||||
tally: collections.Counter = collections.Counter()
|
||||
for s in range(args.seeds):
|
||||
tally += audit(fit, constructed(progdir, s))
|
||||
r, t = report(f"{Path(progdir).name} ({args.seeds} constructed seeds)", tally)
|
||||
grand_real += r
|
||||
grand_total += t
|
||||
|
||||
for dom_path in args.dom:
|
||||
progdir = str(Path(dom_path).parent)
|
||||
fit = stock_fitness(progdir)
|
||||
r, t = report(f"{dom_path} (evolved)", audit(fit, dom_mod.load(dom_path)))
|
||||
grand_real += r
|
||||
grand_total += t
|
||||
|
||||
if grand_total:
|
||||
print(f"OVERALL: {grand_total - grand_real}/{grand_total} "
|
||||
f"({100 * (grand_total - grand_real) / grand_total:.0f}%) of the "
|
||||
f"crinkliness failures the objective reports are not defects.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
7
experiments/results/ab_ssz_power.csv
Normal file
7
experiments/results/ab_ssz_power.csv
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
programme,seed,mode,urb_hard,urb_soft,urb_total,own_hard,own_soft,own_total,elapsed_s
|
||||
harbor-house,0,urb,16,39,55,16,39,55,76.4
|
||||
harbor-house,1,urb,22,45,67,22,45,67,80.8
|
||||
harbor-house,2,urb,22,40,62,22,40,62,76.6
|
||||
harbor-house,3,urb,15,48,63,15,48,63,75.2
|
||||
harbor-house,4,urb,17,42,59,17,42,59,83.9
|
||||
harbor-house,5,urb,29,42,71,29,42,71,81.7
|
||||
|
Loading…
Add table
Reference in a new issue