From 6405ca1a5924328bdf68767c9ecf6ea775ab1aed Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 26 Aug 2026 17:57:11 +0000 Subject: [PATCH] ssz: record the A/B result honestly -- not a pass at n=3 Fixed-budget A/B (3000 evals, 3 seeds, harbor + maple, every arm scored under stock urb so the permissive modes cannot win by deleting a fail category). usage_daylight's paired hard-fail deltas are harbor [0,-1,-10] and maple [0,+2,-10]. The means (-3.7, -2.7) flatter it: the whole signal is seed 2 in both programmes, and seed 2 is the seed where stock itself does worst. Two seeds in three are flat or slightly worse. On that seed soft rises as much as hard falls (harbor -10h/+9s, maple -10h/+15s), so totals go 62->61 on harbor and 120->125 on maple. Because the scoring is stock, that is a genuine trade of hard failures for soft ones, not a relabelling -- progress under the tiered comparator, where n_hard is primary, but a fail against this issue's acceptance criterion ("without inflating soft"). Which yardstick is right is now the live question, and it is gvb's question as much as ssz's. usage_daylight stays default off; undecided, not refuted. Higher-power run (urb vs usage_daylight, 10 seeds) is running. The diagnostic half stands independent of the search A/B: the objective demands daylight of two thirds of the buried population that does not want it, and 38.6's contrary null was an artefact of three modes that never touched those leaves. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB --- DESIGN.md | 38 +++++++++++++++++++++++++++ experiments/results/ab_ssz_search.csv | 24 +++++++++++++++++ 2 files changed, 62 insertions(+) diff --git a/DESIGN.md b/DESIGN.md index 7ccd300..0b03789 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -5192,6 +5192,44 @@ steer the search to a better building?* The arm's own-objective count is reported alongside solely to show the size of the definitional discount. A mode passes on the stock column. `experiments/ab_ssz_search.py`. +**The result: NOT A PASS at this budget, and n=3 cannot decide it.** Budget +3000, 3 seeds, paired per-seed deltas against `urb` (negative = fewer fails): + +| mode | harbor hard Δ | mean | maple hard Δ | mean | +|---|---|---|---|---| +| `floor` | [0, −1, −2] | −1.0 | [+5, −2, −1] | +0.7 | +| `compact_ok` | [0, −2, **−12**] | −4.7 | [0, +2, +1] | +1.0 | +| `exempt_circulation` | [+2, −3, −1] | −0.7 | [0, +2, +1] | +1.0 | +| `usage_daylight` | [0, −1, **−10**] | −3.7 | [0, +2, **−10**] | −2.7 | + +The means flatter every mode. **The whole signal is seed 2**, in both +programmes, and seed 2 is the seed where stock itself does worst (harbor 22 +hard against 16/22; maple 51 against 30/23). Two seeds in three are flat or +slightly worse. What this says is "on a bad run, the permissive modes do less +badly" — which is not nothing, but it is not the claim. + +And on that seed the soft count rises by as much as the hard count falls: +harbor seed 2 −10 hard / +9 soft, maple seed 2 −10 hard / +15 soft. **Totals**: +harbor 62 → 61, maple 120 → **125**. Because every arm is scored under stock +`urb`, this is a real change of layout and not a relabelling — the search +genuinely traded hard failures for soft ones. Under the project's tiered +comparator, where `n_hard` is the primary key, that trade is progress. Under +`ssz`'s acceptance criterion — *lowers hard without inflating soft* — it is +not. The criterion is stricter than the comparator the search actually uses; +which of the two is the right yardstick is now the live question, and it is +`homemaker-py-gvb`'s question as much as this one. + +`usage_daylight` stays **default off** pending a higher-powered run +(`urb` vs `usage_daylight` only, more seeds, both programmes). Nothing here +justifies shipping it as a default, and nothing here refutes it either: at +n=3 with one dominant seed, the honest reading is *undecided*. + +**What is decided** is the diagnostic half, which does not depend on the search +A/B: the daylight requirement is applied to rooms that architecturally do not +want daylight, in two thirds of the buried population, and §38.6's contrary +null was an artefact of three modes that never touched those leaves. + + ## 39. Config audit: requirements that actively fight the engine (`homemaker-py-ju3`) — measured 2026-08-25 The corpus `patterns.config` targets and `costs.config` values were estimated diff --git a/experiments/results/ab_ssz_search.csv b/experiments/results/ab_ssz_search.csv index 62833c4..8f6124d 100644 --- a/experiments/results/ab_ssz_search.csv +++ b/experiments/results/ab_ssz_search.csv @@ -5,3 +5,27 @@ harbor-house,2,urb,22,40,62,22,40,62,79.8 harbor-house,0,floor,16,38,54,16,38,54,78.9 harbor-house,1,floor,21,46,67,21,46,67,82.9 harbor-house,2,floor,20,41,61,20,41,61,79.0 +harbor-house,0,compact_ok,16,39,55,16,22,38,78.7 +harbor-house,1,compact_ok,20,44,64,20,20,40,82.6 +harbor-house,2,compact_ok,10,55,65,10,34,44,81.4 +harbor-house,0,exempt_circulation,18,37,55,18,34,52,77.2 +harbor-house,1,exempt_circulation,19,49,68,19,42,61,81.0 +harbor-house,2,exempt_circulation,21,42,63,21,37,58,79.0 +harbor-house,0,usage_daylight,16,39,55,16,30,46,79.9 +harbor-house,1,usage_daylight,21,51,72,21,30,51,82.3 +harbor-house,2,usage_daylight,12,49,61,12,35,47,81.8 +maple-court,0,urb,30,77,107,30,77,107,108.5 +maple-court,1,urb,23,83,106,23,83,106,109.0 +maple-court,2,urb,51,69,120,51,69,120,110.1 +maple-court,0,floor,35,74,109,35,74,109,109.1 +maple-court,1,floor,21,81,102,21,81,102,107.9 +maple-court,2,floor,50,69,119,50,69,119,110.3 +maple-court,0,compact_ok,30,78,108,30,47,77,109.8 +maple-court,1,compact_ok,25,82,107,25,43,68,108.6 +maple-court,2,compact_ok,52,70,122,52,33,85,105.9 +maple-court,0,exempt_circulation,30,72,102,30,66,96,105.1 +maple-court,1,exempt_circulation,25,81,106,25,70,95,106.6 +maple-court,2,exempt_circulation,52,70,122,52,61,113,112.6 +maple-court,0,usage_daylight,30,69,99,30,52,82,110.3 +maple-court,1,usage_daylight,25,73,98,25,56,81,107.9 +maple-court,2,usage_daylight,41,84,125,41,59,100,113.2