diff --git a/DESIGN.md b/DESIGN.md index 3168cf8..89cbd63 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -7597,6 +7597,34 @@ programmes where those reached two: | room | 0.34 | **0.66** | | circulation | 0.02 | **0.07** | +**A/B verdict: directionally positive, not resolvable at this budget.** Same +protocol as §39.13/§39.14 — harbor and maple, three 500 k plateau starts × two +RNG seeds, 8000 evals, all arms scored under stock (valid here by +construction, since no aggregation can move a fail). Run against the +*pre-§39.19* defaults, so `stock` really is the old objective: + +``` +harbor geomean vs stock N=6 39.17 -> 39.00 1W/0L/5T diff +0.167 MDD 0.428 +harbor geomean+daylight vs stock N=6 39.17 -> 39.00 1W/0L/5T diff +0.167 MDD 0.428 +maple geomean vs stock N=6 60.50 -> 60.17 1W/1L/4T diff +0.333 MDD 1.434 +maple geomean+daylight vs stock N=6 60.50 -> 60.00 1W/0L/5T diff +0.500 MDD 1.285 +``` + +Every margin is below what N=6 can resolve; N≈26–67 would be needed. Unlike +§39.13's ramp — which was an *exact* null, 0W/0L/6T with sd=0 and identical +trajectories — the arms here genuinely diverge and every one of the four lands +on the favourable side of zero. That is worth exactly as much as it sounds: +four coin-flips landing the same way. It is not evidence, and it is recorded +here so that a later run cannot cite it as if it were. + +**The honest summary of this section's three A/Bs**: none of §39.13, §39.14 or +§39.18 produced a measurable search improvement at an 8000-eval plateau budget. +All three were shipped on correctness grounds — an unrankable tail, a +double-charge, an exemption that bought quality — and §39.19 then required two +of them outright. If the objective changes in §39 pay off in search terms, this +corpus has not yet shown it, and the powered protocol in +`experiments/ab_9gj_crinkliness.py` is what would. + **What it does not fix, deliberately.** A terrace still out-earns a room 4:1. That residue is the rates, not the aggregation: `value_supported` is 300, the same as `value_inside`, while an upper outside leaf costs 110 against a room's diff --git a/experiments/results/ab_ecx.csv b/experiments/results/ab_ecx.csv new file mode 100644 index 0000000..3ea4532 --- /dev/null +++ b/experiments/results/ab_ecx.csv @@ -0,0 +1,37 @@ +programme,start,seed,tail,hard,soft,total,score,elapsed_s +harbor-house,coldstart-500000-s0.dom,0,stock,8,25,33,1.85176240526207e-11,406.1 +harbor-house,coldstart-500000-s0.dom,0,geomean,8,25,33,1.3331368428113424e-11,408.8 +harbor-house,coldstart-500000-s0.dom,0,geomean+daylight,8,25,33,1.3331474034308951e-11,486.4 +harbor-house,coldstart-500000-s2.dom,0,stock,12,30,42,2.0876504874272412e-14,434.0 +harbor-house,coldstart-500000-s2.dom,0,geomean,12,29,41,4.905855471687939e-14,360.9 +harbor-house,coldstart-500000-s2.dom,0,geomean+daylight,12,30,42,2.0270861698600025e-14,356.8 +maple-court,coldstart-500000-s1.dom,0,stock,17,56,73,3.549637383437798e-25,438.8 +maple-court,coldstart-500000-s1.dom,0,geomean,17,56,73,3.4927712102261773e-25,451.9 +maple-court,coldstart-500000-s1.dom,0,geomean+daylight,17,56,73,3.490922156298438e-25,473.2 +harbor-house,coldstart-500000-s0.dom,1,stock,8,24,32,3.749216704165142e-11,414.2 +harbor-house,coldstart-500000-s0.dom,1,geomean,8,24,32,2.7696350036240933e-11,426.3 +harbor-house,coldstart-500000-s0.dom,1,geomean+daylight,8,24,32,2.6179276610885845e-11,496.2 +harbor-house,coldstart-500000-s2.dom,1,stock,12,30,42,2.0876504874272412e-14,446.5 +harbor-house,coldstart-500000-s2.dom,1,geomean,12,30,42,1.987950485765416e-14,365.1 +harbor-house,coldstart-500000-s2.dom,1,geomean+daylight,12,29,41,3.0629342889555544e-14,354.0 +maple-court,coldstart-500000-s1.dom,1,stock,17,56,73,3.5495919599065193e-25,438.9 +maple-court,coldstart-500000-s1.dom,1,geomean,17,56,73,3.5024500858428627e-25,451.5 +maple-court,coldstart-500000-s1.dom,1,geomean+daylight,17,56,73,3.491084124317304e-25,447.6 +harbor-house,coldstart-500000-s1.dom,0,stock,7,36,43,1.4676873187056738e-15,368.6 +harbor-house,coldstart-500000-s1.dom,0,geomean,7,36,43,1.4665664078808208e-15,372.1 +harbor-house,coldstart-500000-s1.dom,0,geomean+daylight,7,36,43,1.2492266264690038e-15,462.0 +maple-court,coldstart-500000-s0.dom,0,stock,18,35,53,9.987013209179406e-18,546.9 +maple-court,coldstart-500000-s0.dom,0,geomean,18,35,53,9.60106526960439e-18,462.1 +maple-court,coldstart-500000-s0.dom,0,geomean+daylight,18,35,53,9.763084974756856e-18,460.8 +maple-court,coldstart-500000-s2.dom,0,stock,12,43,55,2.9415325131166865e-18,479.3 +maple-court,coldstart-500000-s2.dom,0,geomean,12,44,56,1.496080486892727e-18,481.9 +maple-court,coldstart-500000-s2.dom,0,geomean+daylight,11,44,55,3.1185722410760797e-18,479.8 +harbor-house,coldstart-500000-s1.dom,1,stock,7,36,43,1.4676873187056738e-15,381.1 +harbor-house,coldstart-500000-s1.dom,1,geomean,7,36,43,1.4465496601047535e-15,377.1 +harbor-house,coldstart-500000-s1.dom,1,geomean+daylight,7,36,43,1.2455994055175617e-15,465.2 +maple-court,coldstart-500000-s0.dom,1,stock,17,37,54,5.878387691540865e-18,547.7 +maple-court,coldstart-500000-s0.dom,1,geomean,18,36,54,5.1384167541013474e-18,487.6 +maple-court,coldstart-500000-s0.dom,1,geomean+daylight,17,37,54,5.537746091126898e-18,503.0 +maple-court,coldstart-500000-s2.dom,1,stock,12,43,55,2.931601349417379e-18,506.2 +maple-court,coldstart-500000-s2.dom,1,geomean,12,40,52,2.12145776572736e-17,493.1 +maple-court,coldstart-500000-s2.dom,1,geomean+daylight,12,40,52,2.288105258404665e-17,493.3