The 39.19 objective change broke three tests. They were the right ones to
break, and how they broke matters more than the change.
test_native_fitness_score_parity and test_native_fitness_fail_set_parity read a
cached .score/.fails beside each corpus .dom and assert the native Python
fitness agrees. They are the ONLY check that the native evaluator still agrees
with the Urb oracle it was ported from, and CLAUDE.md still describes oracle.py
and the Perl tool as kept for cross-validation.
But .gitignore lines 10-11 exclude *.dom.score and *.dom.fails, and
git log --diff-filter=A over those patterns finds zero files ever added on any
branch. No oracle cache has ever existed here, so on a clean checkout all 64
parametrised cases skip.
Worse than skipping is what happens when they do not. Nothing in a .score file
records who wrote it, so a .dom left in that directory by a search run -- with
a .score written by homemaker-fitness, the NATIVE scorer -- silently becomes a
parity fixture, and the test compares the native scorer with itself. That
passes by construction whatever the native scorer says.
Three such cases were live and green: the coldstart-500000-s*.dom artefacts
committed to examples/programme-house during 39.12 and scored natively this
session. They surfaced only because 39.19 made the native scorer disagree with
its own stale output; absent an objective change, a green "native matches
oracle" would have been reported indefinitely.
Stopgap: parametrisation restricted to the Perl corpus's MD5-named files so a
session artefact cannot become a fixture again; the skip message now says
parity is UNVERIFIED rather than reading like an optional missing cache; a
guard test asserts the restriction. All 64 cases skip honestly. Regenerating
the caches with the native scorer would not have been a fix -- it would have
re-cemented the self-comparison.
Filed as homemaker-py-118 (P1): regenerate fixtures from the Perl oracle,
narrow the ignore rules so fixture caches can be tracked, and find out whether
parity still holds -- it may not, since 39.14, 39.18 and 39.19 all changed the
native objective and the oracle has none of them. If parity is being abandoned
deliberately the tests should be deleted with a note. What must not survive is
a test that looks like a guarantee and is not one.
415 passed, 64 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MJ84Feep79Hhm3E4zZJmnB
47079997cf