e5eb397b52
All four 500k runs died about 10 minutes in when the container was
reclaimed. No SIGTERM fired, so no .dom was written and 0 of 12 runs
completed. My plan committed results per finished run, which protected
nothing because no run reached its commit point. The bad assumption was
reading "reclaimed after inactivity" as CPU inactivity; it is conversation
inactivity, and background compute does not hold the box open.
Progress reached before the loss (from the tracked logs): harbor 24,960
evals / 40 fails, maple 14,880 / 79, health-centre 25,920 / 33,
programme-house 138,800 / 2.
The underlying gap is not environmental: a search's only output lands at
the very end or on SIGTERM, so ANY abrupt loss -- reclaimed container, OOM,
power cut -- takes the whole run with it. On a 3M-eval search that is 2.4
days of compute with no recoverable artefact.
- driver.search gains checkpoint=/checkpoint_every=: the current best is
handed to a callback at most every N evals. Rate-limited by evals, not
improvements, which come in bursts early. A failing checkpoint is logged
and swallowed -- losing a checkpoint is bad, losing the search because a
checkpoint failed is worse.
- homemaker-evolve --checkpoint-every N writes <out>.dom.checkpoint via
mkstemp + os.replace, so a crash can never catch it half-written. It is
deliberately NOT the output path: a checkpoint is a leaf-sharing run's
internal best, dishonest under the canonical scorer until the finish
stage unfolds it (homemaker-py-3l6), and must not be mistaken for the
finished article.
- Verified the written checkpoint re-loads as a valid .dom.
Default off, so behaviour is unchanged without the flag.
Lint at parity (46); tests 372 passed (3 new), same 2 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
63 lines
3.4 KiB
Text
63 lines
3.4 KiB
Text
seed : /home/user/homemaker-layout/examples/programme-house/init.dom
|
|
programme : programme-house
|
|
budget : 500000
|
|
pop : 16
|
|
child_budget : 80
|
|
workers : 1
|
|
rng seed : 0
|
|
leaf sharing : True (factor=3)
|
|
superpose : False
|
|
multi_use : False
|
|
conn grade : False
|
|
use tiers : False
|
|
bridge circulation : False
|
|
ruin recreate : False
|
|
collapse in-search : True
|
|
shapecurve warmstart : False
|
|
shapecurve prune : False
|
|
output : /home/user/homemaker-layout/examples/programme-house/coldstart-500000-s0.dom
|
|
[ 80 evals] best 9.96032e-11 (fails 17) via construct/0
|
|
[ 160 evals] best 5.18243e-09 (fails 16) via construct/1
|
|
[ 320 evals] best 9.65534e-06 (fails 7) via construct/3
|
|
[ 1920 evals] best 1.96478e-05 (fails 7) via place_missing noop
|
|
[ 2480 evals] best 1.9751e-05 (fails 7) via place_missing noop
|
|
[ 2880 evals] best 1.99208e-05 (fails 7) via place_missing noop
|
|
[ 3120 evals] best 2.00298e-05 (fails 7) via core_undivide noop
|
|
[ 4640 evals] best 7.04607e-05 (fails 5) via crossover rl<->ll
|
|
[ 6000 evals] best 7.06394e-05 (fails 5) via level_fix noop
|
|
[ 7280 evals] best 7.07633e-05 (fails 5) via level_retype 0/rrl<->1/rl
|
|
[ 7680 evals] best 7.11015e-05 (fails 5) via core_undivide noop
|
|
[ 7840 evals] best 7.11352e-05 (fails 5) via core_divide noop
|
|
[ 8400 evals] best 0.000111875 (fails 5) via undivide 0/rr
|
|
[ 8480 evals] best 0.000114961 (fails 5) via core_undivide noop
|
|
[ 10640 evals] best 8.54011e-05 (fails 4) via swap 0/l
|
|
[ 11440 evals] best 8.56844e-05 (fails 4) via crossover rlr<->ll
|
|
[ 11600 evals] best 8.57076e-05 (fails 4) via level_retype 1/rl<->0/llr
|
|
[ 13120 evals] best 8.58391e-05 (fails 4) via core_undivide noop
|
|
[ 13440 evals] best 8.73822e-05 (fails 4) via undivide 0/rl
|
|
[ 15760 evals] best 0.000112799 (fails 4) via crossover ll<->ll
|
|
[ 16320 evals] best 0.00015754 (fails 3) via level_retype 1/lr<->0/rl
|
|
[ 16480 evals] best 0.00020643 (fails 3) via swap 0/ll
|
|
[ 18480 evals] best 0.000208867 (fails 3) via crossover lll<->llr
|
|
[ 19360 evals] best 2.54868e-05 (fails 2) via level_compound_fix l1: lvl1 → lvl0/rl
|
|
[ 19520 evals] best 2.54914e-05 (fails 2) via crossover rr<->rr
|
|
[ 19920 evals] best 2.54938e-05 (fails 2) via level_compound_fix noop
|
|
[ 22720 evals] best 0.000445739 (fails 2) via undivide 0/rl
|
|
[ 25760 evals] best 0.000446136 (fails 2) via crossover lll<->lll
|
|
[ 26080 evals] best 0.000446413 (fails 2) via core_undivide noop
|
|
[ 28800 evals] best 0.000446749 (fails 2) via retype 1/rrl->b2
|
|
[ 29440 evals] best 0.000447236 (fails 2) via level_retype 1/rrl<->0/rl
|
|
[ 29920 evals] best 0.000447713 (fails 2) via crossover r<->r
|
|
[ 30320 evals] best 0.00044781 (fails 2) via crossover rrr<->llr
|
|
[ 33520 evals] best 0.000447846 (fails 2) via retype 0/lr->t2
|
|
[ 33600 evals] best 0.000447864 (fails 2) via crossover root<->root
|
|
[ 92160 evals] best 0.00248135 (fails 2) via retype 1/ll->O
|
|
[102080 evals] best 0.00248174 (fails 2) via crossover rrr<->llr
|
|
[102880 evals] best 0.00248226 (fails 2) via crossover rrl<->rrl
|
|
[103120 evals] best 0.00248262 (fails 2) via retype 1/rl->t2
|
|
[103200 evals] best 0.00248301 (fails 2) via core_undivide noop
|
|
[110640 evals] best 0.0024834 (fails 2) via crossover ll<->ll
|
|
[111920 evals] best 0.00248376 (fails 2) via crossover root<->root
|
|
[114080 evals] best 0.00248391 (fails 2) via level_retype 1/rl<->0/lr
|
|
[120240 evals] best 0.00248398 (fails 2) via level_retype 1/rrr<->0/rrr
|
|
[138800 evals] best 0.00248406 (fails 2) via level_delete (2 storeys)
|