Bake-off: CMA-ES confirmed as inner-loop optimiser (homemaker-py-d0s)
4-way comparison (NM / CMA-ES / compass / compass-ms) over 3 corpus files × 3 seeds at budget 200, cold-start, URB_NO_OCCLUSION=1. CMA-ES wins on batch-efficiency (18 oracle calls vs 200 for NM, 12x speedup on Perl startup amortisation per §4.6) with acceptable quality (x1.41 @200 vs NM's x1.56). Compass stalls on narrow-valley landscapes and introduces fail regressions. NM flagged as Phase 3+ candidate once native fitness removes oracle call overhead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
c01a8a0887
commit
bc61f8cb73
3 changed files with 1012 additions and 5 deletions
26
DESIGN.md
26
DESIGN.md
|
|
@ -374,11 +374,27 @@ Each phase has a concrete go/no-go gate; do not advance on faith.
|
||||||
`width_inside` default (Fitness/Base.pm:60) — geometrically impossible; the
|
`width_inside` default (Fitness/Base.pm:60) — geometrically impossible; the
|
||||||
original "passes" only by failing `size` instead. *Confirmed in source.*
|
original "passes" only by failing `size` instead. *Confirmed in source.*
|
||||||
Need a sane width default scaled to area, or per-room widths.
|
Need a sane width default scaled to area, or per-room widths.
|
||||||
3. **Inner-loop optimiser choice.** Nelder-Mead worked for diagnostics; DOF is
|
3. **Inner-loop optimiser choice — RESOLVED (homemaker-py-d0s, 2026-06-13).**
|
||||||
small (≈ rooms−1, 6–7 on the corpus), so CMA-ES may be overkill — batched
|
Bake-off over 3 files × 4 methods × 3 seeds at budget 200
|
||||||
multi-start pattern search parallelises across the oracle and is simpler.
|
(`experiments/bakeoff_innerloop.py`), cold-start, `URB_NO_OCCLUSION=1`:
|
||||||
Resolve via the Phase 1 bake-off, not upfront. Gradient-based becomes an
|
|
||||||
option once native fitness is differentiable-ish.
|
| method | x@40 | x@80 | x@200 | s/eval | oracle calls | fails+ |
|
||||||
|
|-------------|------|------|-------|--------|--------------|--------|
|
||||||
|
| Nelder-Mead | 1.45 | 1.50 | 1.56 | 2.05 | 200 | 0 |
|
||||||
|
| CMA-ES | 1.09 | 1.32 | 1.41 | 1.69 | 18 | 0 |
|
||||||
|
| compass | 0.71 | 0.92 | 1.48 | 1.69 | 12 | 3 |
|
||||||
|
| compass-ms | 0.71 | 0.92 | 0.92 | 1.44 | 13 | 4 |
|
||||||
|
|
||||||
|
**Decision: keep CMA-ES (already the default) for the Perl oracle era.**
|
||||||
|
Nelder-Mead wins quality per eval (+x0.15 at @200) but is inherently
|
||||||
|
sequential — 200 Perl invocations vs 18 for CMA (§4.6 batching matters).
|
||||||
|
Compass stalls on narrow-valley landscapes (2f45907: x0.62 vs x1.30) and
|
||||||
|
introduces fail regressions 3/9 runs. Multi-start compass wastes budget
|
||||||
|
on phase splits.
|
||||||
|
|
||||||
|
**Phase 3+ note:** once native fitness replaces the oracle, oracle-call count
|
||||||
|
disappears. Revisit Nelder-Mead then — its quality advantage is real.
|
||||||
|
Gradient-based (autograd through native fitness) is also an option.
|
||||||
4. **Search algorithm for topology.** Memetic GA (keep crossover — now
|
4. **Search algorithm for topology.** Memetic GA (keep crossover — now
|
||||||
meaningful, since a subtree = a contiguous region) vs simulated annealing
|
meaningful, since a subtree = a contiguous region) vs simulated annealing
|
||||||
(the floorplanning workhorse with M1/M2/M3 moves on Polish expressions).
|
(the floorplanning workhorse with M1/M2/M3 moves on Polish expressions).
|
||||||
|
|
|
||||||
767
experiments/bakeoff_innerloop.json
Normal file
767
experiments/bakeoff_innerloop.json
Normal file
|
|
@ -0,0 +1,767 @@
|
||||||
|
{
|
||||||
|
"budget": 200,
|
||||||
|
"checkpoints": [
|
||||||
|
40,
|
||||||
|
80,
|
||||||
|
120,
|
||||||
|
200
|
||||||
|
],
|
||||||
|
"runs": [
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.015336134791349723,
|
||||||
|
"80": 0.01625453592736873,
|
||||||
|
"120": 0.016813214603667533,
|
||||||
|
"200": 0.016970841073693777
|
||||||
|
},
|
||||||
|
"final_fitness": 0.016970841073693777,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 426.10284887799935
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.015336134791349723,
|
||||||
|
"80": 0.01625453592736873,
|
||||||
|
"120": 0.016813214603667533,
|
||||||
|
"200": 0.016970841073693777
|
||||||
|
},
|
||||||
|
"final_fitness": 0.016970841073693777,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 407.97355182700267
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.015336134791349723,
|
||||||
|
"80": 0.01625453592736873,
|
||||||
|
"120": 0.016813214603667533,
|
||||||
|
"200": 0.016970841073693777
|
||||||
|
},
|
||||||
|
"final_fitness": 0.016970841073693777,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 410.4154980900057
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.013602853938370742,
|
||||||
|
"80": 0.016553033822579822,
|
||||||
|
"120": 0.016553033822579822,
|
||||||
|
"200": 0.016553033822579822
|
||||||
|
},
|
||||||
|
"final_fitness": 0.02127655034590672,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 350.45485657599056
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.01095718684601756,
|
||||||
|
"80": 0.02048062509654076,
|
||||||
|
"120": 0.02048062509654076,
|
||||||
|
"200": 0.02048062509654076
|
||||||
|
},
|
||||||
|
"final_fitness": 0.02048062509654076,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 347.38566490099765
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.011174318189865662,
|
||||||
|
"80": 0.011174318189865662,
|
||||||
|
"120": 0.018967095390790795,
|
||||||
|
"200": 0.018967095390790795
|
||||||
|
},
|
||||||
|
"final_fitness": 0.018967095390790795,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 337.9982590199943
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006607488139901861,
|
||||||
|
"80": 0.006607488139901861,
|
||||||
|
"120": 0.007209518879898406,
|
||||||
|
"200": 0.008009790075052892
|
||||||
|
},
|
||||||
|
"final_fitness": 0.008278786574187515,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 221,
|
||||||
|
"n_oracle_calls": 11,
|
||||||
|
"wall_s": 368.2416331849963
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006607488139901861,
|
||||||
|
"80": 0.006607488139901861,
|
||||||
|
"120": 0.007170981596472924,
|
||||||
|
"200": 0.008175449506286243
|
||||||
|
},
|
||||||
|
"final_fitness": 0.008399547907246088,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 221,
|
||||||
|
"n_oracle_calls": 11,
|
||||||
|
"wall_s": 380.4755447469943
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006607488139901861,
|
||||||
|
"80": 0.006607488139901861,
|
||||||
|
"120": 0.006912734615831356,
|
||||||
|
"200": 0.008078395888024203
|
||||||
|
},
|
||||||
|
"final_fitness": 0.008260349591825748,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 221,
|
||||||
|
"n_oracle_calls": 11,
|
||||||
|
"wall_s": 369.9571806669992
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006607488139901861,
|
||||||
|
"80": 0.006607488139901861,
|
||||||
|
"120": 0.006607488139901861,
|
||||||
|
"200": 0.006607488139901861
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006607488139901861,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 202,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 329.4596275669901
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006607488139901861,
|
||||||
|
"80": 0.006607488139901861,
|
||||||
|
"120": 0.006607488139901861,
|
||||||
|
"200": 0.006607488139901861
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006607488139901861,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 202,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 331.6751732940029
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 7,
|
||||||
|
"orig_fitness": 0.01303535257266566,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.006460366122034485,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006607488139901861,
|
||||||
|
"80": 0.006607488139901861,
|
||||||
|
"120": 0.006607488139901861,
|
||||||
|
"200": 0.006607488139901861
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006607488139901861,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 202,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 353.22266581900476
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.013043378536981178,
|
||||||
|
"80": 0.013162194510538643,
|
||||||
|
"120": 0.013541863507298595,
|
||||||
|
"200": 0.013861258388999518
|
||||||
|
},
|
||||||
|
"final_fitness": 0.013861258388999518,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 533.6580855659995
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.013043378536981178,
|
||||||
|
"80": 0.013162194510538643,
|
||||||
|
"120": 0.013541863507298595,
|
||||||
|
"200": 0.013861258388999518
|
||||||
|
},
|
||||||
|
"final_fitness": 0.013861258388999518,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 534.7732095019892
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.013043378536981178,
|
||||||
|
"80": 0.013162194510538643,
|
||||||
|
"120": 0.013541863507298595,
|
||||||
|
"200": 0.013861258388999518
|
||||||
|
},
|
||||||
|
"final_fitness": 0.013861258388999518,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 529.9340490540053
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.008766176333159788,
|
||||||
|
"80": 0.011349351963634282,
|
||||||
|
"120": 0.01183788578257949,
|
||||||
|
"200": 0.01183788578257949
|
||||||
|
},
|
||||||
|
"final_fitness": 0.01183788578257949,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 478.9317944670038
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.011654770458049522,
|
||||||
|
"80": 0.011654770458049522,
|
||||||
|
"120": 0.011654770458049522,
|
||||||
|
"200": 0.011654770458049522
|
||||||
|
},
|
||||||
|
"final_fitness": 0.011654770458049522,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 486.4617367609899
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.007280771403204401,
|
||||||
|
"80": 0.010704424796621367,
|
||||||
|
"120": 0.011790159199403088,
|
||||||
|
"200": 0.011790159199403088
|
||||||
|
},
|
||||||
|
"final_fitness": 0.011790159199403088,
|
||||||
|
"final_n_fails": 1,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 486.73567426699447
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.004059651815091663,
|
||||||
|
"80": 0.01100524588060076,
|
||||||
|
"120": 0.013031039518511447,
|
||||||
|
"200": 0.014373318674033223
|
||||||
|
},
|
||||||
|
"final_fitness": 0.01473833833441545,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 211,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 502.3361668680009
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.00921749875086076,
|
||||||
|
"80": 0.01007215847138918,
|
||||||
|
"120": 0.010784398878431061,
|
||||||
|
"200": 0.013581520956957
|
||||||
|
},
|
||||||
|
"final_fitness": 0.014220594074265345,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 482.41149059700547
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.004120081557419885,
|
||||||
|
"80": 0.005109588901973775,
|
||||||
|
"120": 0.010453965023981411,
|
||||||
|
"200": 0.014012902439663506
|
||||||
|
},
|
||||||
|
"final_fitness": 0.014332429744919455,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 482.2521033469966
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.004059651815091663,
|
||||||
|
"80": 0.01100524588060076,
|
||||||
|
"120": 0.01100524588060076,
|
||||||
|
"200": 0.01100524588060076
|
||||||
|
},
|
||||||
|
"final_fitness": 0.01100524588060076,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 14,
|
||||||
|
"wall_s": 333.32965936099936
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.00921749875086076,
|
||||||
|
"80": 0.01007215847138918,
|
||||||
|
"120": 0.01007215847138918,
|
||||||
|
"200": 0.01007215847138918
|
||||||
|
},
|
||||||
|
"final_fitness": 0.01007215847138918,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 14,
|
||||||
|
"wall_s": 401.0064252880111
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "candidate-002.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.008075772316711586,
|
||||||
|
"orig_n_fails": 2,
|
||||||
|
"x0_fitness": 0.004059651815091663,
|
||||||
|
"x0_n_fails": 3,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.004120081557419885,
|
||||||
|
"80": 0.005109588901973775,
|
||||||
|
"120": 0.005109588901973775,
|
||||||
|
"200": 0.005109588901973775
|
||||||
|
},
|
||||||
|
"final_fitness": 0.005109588901973775,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 215,
|
||||||
|
"n_oracle_calls": 14,
|
||||||
|
"wall_s": 282.3956634509959
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006177486909077849,
|
||||||
|
"80": 0.006527788282834294,
|
||||||
|
"120": 0.0066662305157972855,
|
||||||
|
"200": 0.006693148459352215
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006693148459352215,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 287.4831781459943
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006177486909077849,
|
||||||
|
"80": 0.006527788282834294,
|
||||||
|
"120": 0.0066662305157972855,
|
||||||
|
"200": 0.006693148459352215
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006693148459352215,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 276.6431823440071
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "nm",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.006177486909077849,
|
||||||
|
"80": 0.006527788282834294,
|
||||||
|
"120": 0.0066662305157972855,
|
||||||
|
"200": 0.006693148459352213
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006693148459352213,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 200,
|
||||||
|
"n_oracle_calls": 200,
|
||||||
|
"wall_s": 275.51684825500706
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.00533036008277192,
|
||||||
|
"80": 0.00533036008277192,
|
||||||
|
"120": 0.00533036008277192,
|
||||||
|
"200": 0.00533036008277192
|
||||||
|
},
|
||||||
|
"final_fitness": 0.00533036008277192,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 225.46989931999997
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.004648328603302138,
|
||||||
|
"80": 0.005836221739161555,
|
||||||
|
"120": 0.005836221739161555,
|
||||||
|
"200": 0.005836221739161555
|
||||||
|
},
|
||||||
|
"final_fitness": 0.005836221739161555,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 227.0608354230062
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "cma",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.004653776166165647,
|
||||||
|
"80": 0.004808176606999774,
|
||||||
|
"120": 0.004808176606999774,
|
||||||
|
"200": 0.004808176606999774
|
||||||
|
},
|
||||||
|
"final_fitness": 0.004808176606999774,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 208,
|
||||||
|
"n_oracle_calls": 18,
|
||||||
|
"wall_s": 226.25115464300325
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.002909077000944207,
|
||||||
|
"80": 0.005636022639392401,
|
||||||
|
"120": 0.005637234383710328,
|
||||||
|
"200": 0.006368553115957026
|
||||||
|
},
|
||||||
|
"final_fitness": 0.006368553115957026,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 228.6235062899941
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.0038443413369024806,
|
||||||
|
"80": 0.004505393404296923,
|
||||||
|
"120": 0.005335631011587571,
|
||||||
|
"200": 0.007639377813299466
|
||||||
|
},
|
||||||
|
"final_fitness": 0.0077190451924048515,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 230.63287285399565
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "compass",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.003989234037259822,
|
||||||
|
"80": 0.003993271664133415,
|
||||||
|
"120": 0.0083599248344224,
|
||||||
|
"200": 0.011177097393828111
|
||||||
|
},
|
||||||
|
"final_fitness": 0.011177097393828111,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 12,
|
||||||
|
"wall_s": 228.19498966900574
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 0,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.002909077000944207,
|
||||||
|
"80": 0.005636022639392401,
|
||||||
|
"120": 0.005636022639392401,
|
||||||
|
"200": 0.005636022639392401
|
||||||
|
},
|
||||||
|
"final_fitness": 0.005636022639392401,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 213,
|
||||||
|
"n_oracle_calls": 14,
|
||||||
|
"wall_s": 222.05841742099437
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 1,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.0038443413369024806,
|
||||||
|
"80": 0.004505393404296923,
|
||||||
|
"120": 0.004505393404296923,
|
||||||
|
"200": 0.004505393404296923
|
||||||
|
},
|
||||||
|
"final_fitness": 0.004505393404296923,
|
||||||
|
"final_n_fails": 2,
|
||||||
|
"n_evals": 215,
|
||||||
|
"n_oracle_calls": 14,
|
||||||
|
"wall_s": 228.25147005799226
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"file": "c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
"method": "compass-ms",
|
||||||
|
"seed": 2,
|
||||||
|
"dof": 6,
|
||||||
|
"orig_fitness": 0.0039977449092860615,
|
||||||
|
"orig_n_fails": 3,
|
||||||
|
"x0_fitness": 0.0019865661983037492,
|
||||||
|
"x0_n_fails": 4,
|
||||||
|
"best_at": {
|
||||||
|
"40": 0.003989234037259822,
|
||||||
|
"80": 0.003993271664133415,
|
||||||
|
"120": 0.003993271664133415,
|
||||||
|
"200": 0.003993271664133415
|
||||||
|
},
|
||||||
|
"final_fitness": 0.003993271664133415,
|
||||||
|
"final_n_fails": 3,
|
||||||
|
"n_evals": 215,
|
||||||
|
"n_oracle_calls": 14,
|
||||||
|
"wall_s": 223.43627467899933
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
224
experiments/bakeoff_innerloop.py
Normal file
224
experiments/bakeoff_innerloop.py
Normal file
|
|
@ -0,0 +1,224 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Inner-loop optimiser bake-off at equal oracle budgets (homemaker-py-d0s).
|
||||||
|
|
||||||
|
DESIGN.md §7 Phase 1(b), §8.3. DOF is only ≈ rooms−1 (6–7 on the corpus), so
|
||||||
|
the question is fitness gained per oracle evaluation, not asymptotic power.
|
||||||
|
Candidates:
|
||||||
|
|
||||||
|
nm multi-start Nelder-Mead (scipy) — the §4.5 diagnostic optimiser.
|
||||||
|
Inherently sequential: ONE dom per oracle call, so the Perl
|
||||||
|
startup never amortises (§4.6).
|
||||||
|
cma multi-phase CMA-ES (innerloop.cma_search), one batched oracle
|
||||||
|
call per generation.
|
||||||
|
compass single-start batched compass search with pattern moves + random
|
||||||
|
augmentation (innerloop.compass_search).
|
||||||
|
compass-ms multi-start compass: budget split across restarts (x0 first,
|
||||||
|
then random starts), global best kept.
|
||||||
|
|
||||||
|
Protocol: cold start from each file's equal-offset projection (x_current),
|
||||||
|
one run per (method, file, seed), best-so-far recorded after every oracle
|
||||||
|
call. Fitness-at-budget-B is read off the trace (evals ≤ B), so methods are
|
||||||
|
compared at exactly equal budgets regardless of batch granularity; checkpoint
|
||||||
|
budgets bracket the driver's real operating points (child_budget=80 warm,
|
||||||
|
seed_budget=200 cold). Wall-clock and oracle-invocation counts are recorded
|
||||||
|
per run.
|
||||||
|
|
||||||
|
Runs under URB_NO_OCCLUSION=1 (set by this script — the gp2 re-baseline flag;
|
||||||
|
all benchmarks must use it).
|
||||||
|
|
||||||
|
Usage: python3 experiments/bakeoff_innerloop.py [budget] [out.json]
|
||||||
|
(defaults: budget 200, experiments/bakeoff_innerloop.json)
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
|
||||||
|
from homemaker import dom, innerloop, oracle # noqa: E402
|
||||||
|
|
||||||
|
URB = Path("/home/bruno/src/urb")
|
||||||
|
EX = URB / "examples" / "programme-house"
|
||||||
|
|
||||||
|
FILES = (
|
||||||
|
"2f45907abd9accac2a124d311732f749.dom",
|
||||||
|
"candidate-002.dom",
|
||||||
|
"c964435454c459f86c3ed9a5a7621132.dom",
|
||||||
|
)
|
||||||
|
SEEDS = (0, 1, 2)
|
||||||
|
CHECKPOINTS = (40, 80, 120, 200)
|
||||||
|
|
||||||
|
|
||||||
|
class TracingEvaluator(innerloop.OracleEvaluator):
|
||||||
|
"""Records (cumulative evals, batch-best fitness) after every oracle call."""
|
||||||
|
|
||||||
|
def __init__(self, *a, **kw):
|
||||||
|
super().__init__(*a, **kw)
|
||||||
|
self.trace: list[tuple[int, float]] = []
|
||||||
|
|
||||||
|
def evaluate(self, xs):
|
||||||
|
scores = super().evaluate(xs)
|
||||||
|
self.trace.append((self.n_evals, max(s.fitness for s in scores)))
|
||||||
|
return scores
|
||||||
|
|
||||||
|
def best_at(self, budget: int) -> float:
|
||||||
|
vals = [f for n, f in self.trace if n <= budget]
|
||||||
|
return max(vals) if vals else float("nan")
|
||||||
|
|
||||||
|
|
||||||
|
class _BudgetExhausted(Exception):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def nm_search(ev, x0, budget=200, seed=0):
|
||||||
|
"""Multi-start Nelder-Mead: x0 first, random restarts until the budget is
|
||||||
|
spent. Sequential — every evaluation is its own oracle invocation."""
|
||||||
|
from scipy.optimize import minimize
|
||||||
|
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
n = len(x0)
|
||||||
|
x = np.clip(np.asarray(x0, dtype=float), innerloop._EPS, 1 - innerloop._EPS)
|
||||||
|
s = ev.evaluate([x])[0]
|
||||||
|
best = innerloop.Result(
|
||||||
|
x=x.copy(), fitness=s.fitness, n_fails=s.n_fails, fail_lines=s.fail_lines,
|
||||||
|
x0_fitness=s.fitness, x0_n_fails=s.n_fails, n_evals=0, n_oracle_calls=0,
|
||||||
|
)
|
||||||
|
|
||||||
|
def f(xi):
|
||||||
|
if ev.n_evals >= budget:
|
||||||
|
raise _BudgetExhausted
|
||||||
|
sc = ev.evaluate([np.asarray(xi, dtype=float)])[0]
|
||||||
|
if sc.fitness > best.fitness:
|
||||||
|
best.x = np.asarray(xi, dtype=float).copy()
|
||||||
|
best.fitness = sc.fitness
|
||||||
|
best.n_fails = sc.n_fails
|
||||||
|
best.fail_lines = sc.fail_lines
|
||||||
|
return -sc.fitness
|
||||||
|
|
||||||
|
start = x.copy()
|
||||||
|
while ev.n_evals < budget:
|
||||||
|
try:
|
||||||
|
minimize(
|
||||||
|
f, start, method="Nelder-Mead",
|
||||||
|
bounds=[(innerloop._EPS, 1 - innerloop._EPS)] * n,
|
||||||
|
options={"maxfev": budget - ev.n_evals, "xatol": 1e-3, "fatol": 1e-10},
|
||||||
|
)
|
||||||
|
except _BudgetExhausted:
|
||||||
|
break
|
||||||
|
start = rng.uniform(0.1, 0.9, n) # restart
|
||||||
|
|
||||||
|
best.n_evals = ev.n_evals
|
||||||
|
best.n_oracle_calls = ev.n_oracle_calls
|
||||||
|
return best
|
||||||
|
|
||||||
|
|
||||||
|
def compass_ms_search(ev, x0, budget=200, seed=0, n_starts=3):
|
||||||
|
"""Multi-start compass: budget split evenly; first start is x0, the rest
|
||||||
|
random. compass_search counts against ev.n_evals, so phase budgets are
|
||||||
|
cumulative caps."""
|
||||||
|
rng = np.random.default_rng(seed)
|
||||||
|
n = len(x0)
|
||||||
|
best = None
|
||||||
|
for phase in range(n_starts):
|
||||||
|
phase_end = ev.n_evals + (budget - ev.n_evals) // (n_starts - phase)
|
||||||
|
start = np.asarray(x0, dtype=float) if phase == 0 else rng.uniform(0.1, 0.9, n)
|
||||||
|
r = innerloop.compass_search(ev, start, budget=phase_end, seed=seed + phase)
|
||||||
|
if best is None or r.fitness > best.fitness:
|
||||||
|
keep_x0 = best.x0_fitness if best is not None else r.x0_fitness
|
||||||
|
keep_x0f = best.x0_n_fails if best is not None else r.x0_n_fails
|
||||||
|
best = r
|
||||||
|
best.x0_fitness, best.x0_n_fails = keep_x0, keep_x0f
|
||||||
|
if ev.n_evals >= budget:
|
||||||
|
break
|
||||||
|
best.n_evals = ev.n_evals
|
||||||
|
best.n_oracle_calls = ev.n_oracle_calls
|
||||||
|
return best
|
||||||
|
|
||||||
|
|
||||||
|
METHODS = {
|
||||||
|
"nm": nm_search,
|
||||||
|
"cma": innerloop.cma_search,
|
||||||
|
"compass": innerloop.compass_search,
|
||||||
|
"compass-ms": compass_ms_search,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
budget = int(sys.argv[1]) if len(sys.argv) > 1 else 200
|
||||||
|
out_path = Path(sys.argv[2]) if len(sys.argv) > 2 else (
|
||||||
|
Path(__file__).parent / "bakeoff_innerloop.json")
|
||||||
|
os.environ["URB_NO_OCCLUSION"] = "1"
|
||||||
|
checkpoints = [c for c in CHECKPOINTS if c <= budget] or [budget]
|
||||||
|
if checkpoints[-1] != budget:
|
||||||
|
checkpoints.append(budget)
|
||||||
|
|
||||||
|
# Baselines: the UNMODIFIED originals (gains measured from there, not from
|
||||||
|
# the equal-offset projection — accept_innerloop.py convention).
|
||||||
|
orig: dict[str, oracle.Score] = {}
|
||||||
|
with tempfile.TemporaryDirectory(prefix="bakeoff_orig_") as tmp:
|
||||||
|
shutil.copy(EX / "patterns.config", tmp)
|
||||||
|
for name in FILES:
|
||||||
|
orig[name] = oracle.score(shutil.copy(EX / name, tmp), URB)
|
||||||
|
|
||||||
|
runs = []
|
||||||
|
for name in FILES:
|
||||||
|
for method in METHODS:
|
||||||
|
for seed in SEEDS:
|
||||||
|
root = dom.load(str(EX / name))
|
||||||
|
with TracingEvaluator(root, EX, URB) as ev:
|
||||||
|
x0 = ev.x_current
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
r = METHODS[method](ev, x0, budget=budget, seed=seed)
|
||||||
|
dt = time.perf_counter() - t0
|
||||||
|
run = {
|
||||||
|
"file": name, "method": method, "seed": seed,
|
||||||
|
"dof": len(x0),
|
||||||
|
"orig_fitness": orig[name].fitness,
|
||||||
|
"orig_n_fails": orig[name].n_fails,
|
||||||
|
"x0_fitness": r.x0_fitness, "x0_n_fails": r.x0_n_fails,
|
||||||
|
"best_at": {str(c): ev.best_at(c) for c in checkpoints},
|
||||||
|
"final_fitness": r.fitness, "final_n_fails": r.n_fails,
|
||||||
|
"n_evals": ev.n_evals, "n_oracle_calls": ev.n_oracle_calls,
|
||||||
|
"wall_s": dt,
|
||||||
|
}
|
||||||
|
runs.append(run)
|
||||||
|
gains = " ".join(
|
||||||
|
f"@{c}:x{run['best_at'][str(c)] / orig[name].fitness:.2f}"
|
||||||
|
for c in checkpoints)
|
||||||
|
print(
|
||||||
|
f"{name[:12]:12s} {method:10s} seed={seed} {gains} "
|
||||||
|
f"fails {orig[name].n_fails}->{r.n_fails} "
|
||||||
|
f"{ev.n_evals}ev/{ev.n_oracle_calls}calls {dt:.0f}s",
|
||||||
|
flush=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
out_path.write_text(json.dumps(
|
||||||
|
{"budget": budget, "checkpoints": checkpoints, "runs": runs}, indent=1))
|
||||||
|
print(f"\nwrote {out_path}")
|
||||||
|
|
||||||
|
# Summary: mean gain over original at each checkpoint, mean s/eval.
|
||||||
|
print(f"\n{'method':10s} " + "".join(f"{'x@' + str(c):>8s}" for c in checkpoints)
|
||||||
|
+ f"{'s/eval':>8s}{'calls':>7s}{'fails+':>7s}")
|
||||||
|
for method in METHODS:
|
||||||
|
rs = [r for r in runs if r["method"] == method]
|
||||||
|
cols = ""
|
||||||
|
for c in checkpoints:
|
||||||
|
g = np.mean([r["best_at"][str(c)] / r["orig_fitness"] for r in rs])
|
||||||
|
cols += f"{g:8.2f}"
|
||||||
|
spe = np.mean([r["wall_s"] / r["n_evals"] for r in rs])
|
||||||
|
calls = np.mean([r["n_oracle_calls"] for r in rs])
|
||||||
|
newf = sum(r["final_n_fails"] > r["orig_n_fails"] for r in rs)
|
||||||
|
print(f"{method:10s} {cols}{spe:8.2f}{calls:7.0f}{newf:7d}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
Loading…
Add table
Reference in a new issue