Skip to content

refactor(harness): rename harness/scenarios to harness/trials - #563

Merged
Dong1017 merged 8 commits into
mainfrom
impl/558-harness-trials
Nov 21, 2026
Merged

Dong1017 merged 8 commits into
mainfrom
impl/558-harness-trials

Conversation

@Dong1017

@Dong1017 Dong1017 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Governing Issue and delivery slice

Refs #558

  • Accepted scope / delegation: task: rename harness/scenarios/ to match its trial-execution role #558 as accepted by the goal owner (Dong1017) on 2026-11-21: rename harness/scenarios/ to harness/trials/ with no forwarder, and carry the same release-boundary evidence as task: retire the infra/ and model/ packages into their owning modules #498 slice 5.
  • Base. docs: correct stale layout text and complete the directory map #559 merged; 10bb12b merges main 15f1703, so the diff against main is the rename alone.
  • What this PR delivers:
    • git mv src/inferrsi/harness/scenarios src/inferrsi/harness/trials: both files byte for byte.
    • The four importers (harness/runner.py, harness/project_service.py, harness/optimization_session.py with three imports, apps/legacy/run.py), the _runtime_identity() entry, eight test files and five docs use the new path.
    • Gate: RETIRED_MODULES adds inferrsi.harness.scenarios as a whole. New controls cover absolute and relative imports of the old module and a returning harness/scenarios/ file, and the not-retired control imports harness/trials.
    • tests/check_installed.py: the old names are absent from the wheel, the new ones present.
    • Release boundary: tests/fixtures/optimization-session/pre-trials-rename-session/ is a session that the runtime just before the rename started and interrupted with its own installed wheel, unedited. The checkout test is parametrized over both pre-change sessions. The installed-wheel check now reads every *-session directory through INFERRSI_PRE_CHANGE_SESSIONS, which replaces INFERRSI_PRE_INFRA_SESSION in the ready, host and release jobs.
    • CHANGELOG entry.
  • What it does not deliver: no behaviour change in execute_trial or collect_trial_checks; no other rename. The Harness root regrouping is deferred (task: rename harness/scenarios/ to match its trial-execution role #558 scope), and the apps/views/ split is a suggestion on task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371.

Knowledge authorship declaration

No path protected by docs/knowledge-model-policy.md changes. The declaration below is disclosure only.

{"schema_version":1,"head_sha":"10bb12bb56255daffd15b369988b6957e0d6e81c","human_only":false,"material_ai_authors":[{"tool":"Claude Code","model_id":"claude-opus-5-5","effort":"xhigh","role":"author"}]}

Changed artifacts and evidence

Requirement (#558) Change Evidence
Old path gone, no forwarder; gate rejects it with or without a file git mv; RETIRED_MODULES test_target_module_import_gate.py import and returning-file controls; check_installed.py
Every importer uses the new path 4 source files, 8 tests grep for harness.scenarios / harness/scenarios outside history (CHANGELOG, fixtures, retired-name tests) is empty
Docs name the new path and role architecture.md, contributing.md, module-layout.md, contracts/function-owners.md (the row now lists execute_trial/collect_trial_checks as Harness functions and leaves read_metric/validate_metric with Benchmark), moe-forward-benchmark.md text
Runtime identity: old sessions refused, files unchanged; old runtime recovers fixture + tests + paired-wheel record below
CHANGELOG entry with the resume rule text

Paired-wheel record on H200 (CPU only, Python 3.12; each wheel built from its commit's own tree into a clean venv). Old = 75b9872 (main 7a7ab84 plus #559's docs commit; same runtime identity as main), new = 997d92d (the rename commit before the rebase onto cce2159; same src/ as this head).

  1. Old wheel, from a neutral directory: start 0, diagnose 0, one round interrupted after its child completed, 130. 14 files, calls.log 2, plan.json names harness/scenarios/external.py, no host user name or /data path in the session.
  2. New wheel on a copy: status 2 and recover 2, "Optimization runtime changed, use its frozen runtime to resume"; the SHA-256 of all 14 files is unchanged.
  3. Old wheel on the same copy: recover 0, then status 0 with completed_rounds 1; calls.log still 2, so recovery used the receipt and did not run the candidate again.

The untouched session is the fixture; its 14 SHA-256 values match the H200 manifest. The fixture README records this and the slice-5 record side by side.

Checks and dependencies

  • Checks actually run (result and tested scope):
    • Local Windows: test_target_module_import_gate.py, test_trial_checks.py, test_legacy_run_split.py, test_architecture_closed_loop.py, test_benchmark_operations.py, test_moe_benchmark.py, test_package_root_inventory.py and test_knowledge_compatibility_shells.py: 269 passed, 2 skipped, 97 subtests. The two pre-change session tests, the pre-serving test and the runtime-identity test: 4 passed. Ruff passes on every changed file; the 35 F811 hits it reports elsewhere are on main too.
    • A/B on H200, full suite (228 test files) on the pre-rename base 75b9872 and the candidate 7cbac18, comparing every FAILED, ERROR and SUBFAILED line: 0 candidate-only failures. Base 433 failed / 5342 passed, candidate 432 failed / 5349 passed (the 7 extra passes are the new gate controls and the second pre-change session case). The one base-only failure is the known flaky test_dsh_plugin.py::test_remote_ssh_connection_and_scoped_operations. The other 432 fail the same way on both sides for host reasons: H200's git 2.34 rejects --no-lazy-fetch, and some tests need playwright, node or network. 7cbac18 is this branch's tree before later commits; it differs from this head only in tests/test_project_explanation.py (now main's Use the Harness digest in the interrupted recovery test #556 version), docs/contracts/function-owners.md and CHANGELOG.md.
  • Failed, skipped, timed-out or not-run checks (if applicable): installed-package and the host jobs run check_installed.py in CI; I did not run it locally.
  • Dependencies / shared interfaces / integration assumptions (if applicable): none. A session planned before this change is refused by design; resume it with the runtime that planned it. Open PRs that use inferrsi.harness.scenarios must move to the new path when they rebase. At the time of writing two drafts do: harness: add bounded VATS/GVO runtime and benchmark adapters #488 imports execute_trial from it in three files and also changes harness/scenarios/external.py, which now lives at harness/trials/external.py; Add an optional AISBench service benchmark adapter #326 imports it in one test.

Remaining work and Issue closure

Provenance

Written in a Claude Code session reported as claude-opus-5-5, effort xhigh. Each commit has DCO sign-off and an Agent-assisted-by line.

🤖 Generated with Claude Code

Removing infra and model left three sentences in module-layout.md that still state
Infra rules in the present tense. They now say what changed. The architecture
directory map gains the Apps delivery package, the Apps root compatibility exports and
the modules added since the map was last updated, so every package on disk is listed.

The contributor guide sent new models to harness/scenarios, whose one module runs any
external command as a trial and is the same for every model; it now names the serving
and Benchmark locations instead. The Harness, scenarios and views package docstrings
now describe what the packages hold, including the search evidence that views writes
for campaign import-search. No code changes.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
…ry test

#457 merged after #498 removed infra/, and its new recovery test still imported digest
from inferrsi.infra.artifacts, so full-check fails on main. The value it forges is
compared by harness.optimization, which takes digest from harness.artifact_primitives,
so the test now imports that copy.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
The package's one module, external.py, runs an external command as one trial and
retains its record (execute_trial, collect_trial_checks). It holds no scenario logic
and is the same for every model, so the old name pointed contributors to the wrong
place for new models.

The move uses git mv and keeps both files byte for byte. The four importers, the
runtime identity entry, the tests and the docs use the new path. There is no forwarder:
the gate retires inferrsi.harness.scenarios as a whole, with import and returning-file
controls, and the installed-wheel check requires the old names to be absent and the new
ones present. A session planned before this change is refused before any operation and
must be resumed with the runtime that planned it.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
…pre-change session

The fixture pre-trials-rename-session is a session that the runtime just before the
rename started and interrupted with its own installed wheel, unedited: a completed
execution receipt for an unsettled round, calls.log at one candidate run, and a plan
that names harness/scenarios/external.py. Its bytes match the H200 run that made it.

The checkout test is now parametrized over both pre-change sessions, and the
installed-wheel check reads every *-session directory under
tests/fixtures/optimization-session through INFERRSI_PRE_CHANGE_SESSIONS, which
replaces INFERRSI_PRE_INFRA_SESSION in the ready, host and release jobs. Both require
the receipt state, the refusal of status and recover, and no file change. The README
records the paired-wheel result for each session: the new wheel exits 2 and changes
nothing, and the old wheel recovers from the receipt without running the candidate.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 21, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

Independent review of 97952bf47e3d1f0e7e8a16e101c426bcbd438755 for #558. I isolated the rename from stacked #559 by comparing cce2159c97d7a76d82620f05c1b19179c0e87eb6 to this head, then checked the full PR against main@10ba7a55. This is the first review of #563.

Required before approval

  1. Function ownership, line 19 changes the path but still attributes read_metric / validate_metric to trials/external.py. That module defines execute_trial and collect_trial_checks; document those as the Harness functions and keep semantic metric validation under Benchmark. This is part of #558’s explicit ownership-doc acceptance.
  2. Restack on a base containing the canonical #556 import fix without carrying #559’s duplicate local import. In a three-way check of tests/test_project_explanation.py, main removes the local digest import and this stack replaces the same line, producing a conflict. The rename is not the cause, but the conflicting line is present in this PR as submitted.

Verified: both moved source blobs match the stack parent byte-for-byte; the old package is gone, import callers and the runtime-identity key use trials, and the retired-path gate and installed-wheel checks cover the new layout. I read all 14 fixture files and checked the completed zero-exit receipt, calls.log = 2, and that the old identity hash matches the pre-rename source. I did not run the paired wheels, full suite, or inspect CI; the old-wheel recovery result is the author’s record, not my rerun.

Non-blocking: qualify the changelog’s old-session refusal statement to plans covered by this runtime_identity check. Service-reuse dispatches before this comparison; the rename should not change its behavior.

#556 fixed the same stale import in tests/test_project_explanation.py with a
module-level import from harness.artifact_primitives. The merge keeps main's file, so
this branch no longer changes that test and its diff is the docs and docstrings only.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)
Signed-off-by: GUOGUO <xwdong1998@163.com>
The base branch merged main, which carries #556; this merge keeps main's tests/test_project_explanation.py, so the rename branch no longer changes that test.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)

Signed-off-by: GUOGUO <xwdong1998@163.com>

Copy link
Copy Markdown
Collaborator

Reviewed the rename-only range cce2159c97d7a76d82620f05c1b19179c0e87eb6 → 97952bf47e3d1f0e7e8a16e101c426bcbd438755, and checked the submitted stack against main@10ba7a5500bcd95949219b138f9ada5033b922b0. I agree with the outstanding NO PASS on this head and am not approving yet:

  1. docs/contracts/function-owners.md:19 only retargets the path to harness/trials/external, but still attributes read_metric / validate_metric (and execute_command) to that module. The moved module defines execute_trial and collect_trial_checks. task: rename harness/scenarios/ to match its trial-execution role #558 requires this ownership doc to name the new path and its actual role; please correct the symbols and keep semantic metric validation under Benchmark.
  2. The submitted stack still carries docs: correct stale layout text and complete the directory map #559's function-local digest import at tests/test_project_explanation.py:497, which conflicts with main@10ba7a55 where Use the Harness digest in the interrupted recovery test #556 moved that import to module level. mergeable_state is currently dirty. Please restack on a base that already contains the Use the Harness digest in the interrupted recovery test #556 import fix, without reintroducing the local import.

The package move, importer / runtime-identity updates, retired-path gate and installed-wheel name checks look consistent with #558. On this head I only see draft-lane lint-and-smoke and declarations; Ready / full-check has not run.

… refusal

Review of 97952bf: the function-owner row for harness/trials/external named read_metric and validate_metric, which live in benchmark/metrics.py. It now lists execute_trial and collect_trial_checks as Harness functions and keeps semantic metric validation with Benchmark. The CHANGELOG limits the refusal to benchmark and AB/BA sessions whose plan carries the runtime identity; service-reuse campaigns are dispatched before that check.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)

Signed-off-by: GUOGUO <xwdong1998@163.com>
zengchuang-hw
zengchuang-hw previously approved these changes Nov 21, 2026

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

Incremental review: 97952bf47e3d1f0e7e8a16e101c426bcbd438755 → e75dbd3b8b0e4a207c0b6450e599b1e56216ed85. Both blockers in the previous review are resolved. The ownership table now lists execute_trial and collect_trial_checks under Harness and locates semantic validation in Benchmark. The changelog correctly limits the compatibility break to benchmark and AB/BA plans that carry this runtime identity; service reuse is unchanged.

The head contains the updated #559 stack and main@10ba7a55. tests/test_project_explanation.py is byte-identical to that base and is absent from the PR diff, so the earlier import overlap is gone. I checked the exact three-file increment and verified the runtime, fixture and checkout/installed-wheel probes are unchanged versus the previous review. The 14-file fixture and completed receipt were checked previously. The paired-wheel recovery result is the author’s record; I did not rerun the wheels, full suite or inspect CI. No remaining code-review blockers. This approval is not a merge or Issue acceptance; #559 is still a separate stacked dependency.

@Dong1017

Copy link
Copy Markdown
Collaborator Author

Thanks. Pushed a4bc6b1 and e75dbd3; src/ is unchanged. No approval was on record, so nothing is voided.

  1. Function ownership. The row for harness/trials/external now lists execute_trial and collect_trial_checks as Harness functions (run one command through execute_command, retain the record, collect the trial checks). Semantic metric validation (read_metric / validate_metric) stays with Benchmark in benchmark/metrics.py, and execute_trial only applies the metric callable its caller injects.
  2. Use the Harness digest in the interrupted recovery test #556 conflict. docs: correct stale layout text and complete the directory map #559 merged main 10ba7a5 as caf2517 and kept main's tests/test_project_explanation.py; a4bc6b1 brings that into this branch. The file is now identical to main, so neither PR changes that test.
  3. Non-blocking, done. The CHANGELOG now limits the refusal to benchmark and AB/BA sessions whose plan carries the runtime identity, and says that service-reuse campaigns are dispatched before this check.

The H200 A/B was run on 7cbac18. That tree differs from this head only in those three files: the test is now main's version, and the other two are docs. The paired-wheel record used 997d92d, which has the same src/ as this head.

amy-why-3459
amy-why-3459 previously approved these changes Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed the increment 97952bf47e3d1f0e7e8a16e101c426bcbd438755 → e75dbd3b8b0e4a207c0b6450e599b1e56216ed85.

Both previous blockers are resolved: docs/contracts/function-owners.md:19 now names execute_trial / collect_trial_checks as Harness functions and keeps semantic read_metric / validate_metric under Benchmark in benchmark/metrics.py; the stack includes the updated #559 head caf2517c and main@10ba7a55, and tests/test_project_explanation.py is byte-identical to main (no longer in the PR-versus-main diff). src/ is unchanged in this increment. The changelog now limits the old-session refusal to benchmark and AB/BA plans that carry this runtime identity, and notes that service-reuse campaigns are dispatched before the check.

Ready full-check (3.10/3.11) was still in progress when I reviewed; the other Ready checks on this head had passed. I did not rerun the paired wheels or the full suite. This approval is for the rename follow-up on this head; #559 remains a stacked dependency. Accuracy/performance comparison evidence is not applicable (docs plus a test-import alignment with main).

Main gained #559 as a squash commit and #541's release workflow. The docs and the scenarios initializer conflicted only because this branch already carried #559's text with the rename on top; the merge keeps this branch's versions, and trials/__init__.py equals main's scenarios/__init__.py byte for byte. release.yml merged cleanly and still passes INFERRSI_PRE_CHANGE_SESSIONS to its one check_installed run.

Agent-assisted-by: Claude Code / claude-opus-5-5 [effort=xhigh] (author)

Signed-off-by: GUOGUO <xwdong1998@163.com>
@Dong1017
Dong1017 dismissed stale reviews from amy-why-3459 and zengchuang-hw via 10bb12b November 21, 2026 14:13
@Dong1017

Copy link
Copy Markdown
Collaborator Author

#559 merged, so I merged main (15f1703) as 10bb12b. This push voids the approvals on e75dbd3 (zengchuang-hw and amy-why-3459): the PR showed as conflicting and could not merge without it.

The conflicts were only in docs/architecture.md, docs/contributing.md and harness/scenarios/__init__.py. This branch already carried #559's text with the rename on top, and #559 landed as a squash commit, so Git saw the same edits twice. The merge keeps this branch's versions:

  • Against main, the two docs now differ only in the five scenarios → trials lines.
  • harness/trials/__init__.py equals main's harness/scenarios/__init__.py byte for byte.

#541's release.yml change merged cleanly, and its one check_installed.py run passes INFERRSI_PRE_CHANGE_SESSIONS. Since e75dbd3, src/ and the rename's own tests are unchanged; the only other changes are #541's two release tests from main.

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PASS

Incremental review: e75dbd3b8b0e4a207c0b6450e599b1e56216ed85 → 10bb12bb56255daffd15b369988b6957e0d6e81c. The prior review cleared the ownership-table and stacking issues. This head merges main@15f17037afb6a57a7a51348f73979f81f5e0721f after #559 landed; those fixes are still intact.

I checked both merge parents and compared the full file trees and the release workflow diff. The rename, import gate, saved-session fixture and regression tests are unchanged from the reviewed head. Five changed release files match main exactly. In the one overlapping file, .github/workflows/release.yml, the only difference from the new main is the already-reviewed fixture path passed to the isolated installed-wheel check. The weekly-release settings come from main, not this PR.

No new blockers or non-blocking findings. I did not rerun the installed-wheel or hardware tests on this merge commit or check CI.

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed the increment e75dbd3b8b0e4a207c0b6450e599b1e56216ed85 → 10bb12bb56255daffd15b369988b6957e0d6e81c.
Reviewed base 15f17037afb6a57a7a51348f73979f81f5e0721f and head 10bb12bb56255daffd15b369988b6957e0d6e81c.

This head merges main@15f17037 after #559 landed (and includes #541). The five release files added in this increment match that main exactly. In the one overlapping file, .github/workflows/release.yml, the only difference from main remains the already-reviewed INFERRSI_PRE_CHANGE_SESSIONS fixture path.

The previously cleared blockers remain intact: docs/contracts/function-owners.md still names execute_trial / collect_trial_checks as Harness functions and keeps semantic metric validation under Benchmark; the changelog still limits old-session refusal to benchmark and AB/BA plans that carry this runtime identity. Against main, docs/architecture.md and docs/contributing.md differ only in the scenarios → trials wording, and src/ is unchanged from the previously reviewed head.

lint-and-smoke, installed-package, and host-compatibility have passed on this head; Ready full-check was still running at review time.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Technical / third-party tip COMMENT (cursor[bot]; collaborator permission none — not write+ / dual-tip / merge-ready).

Reviewed base 15f17037afb6a57a7a51348f73979f81f5e0721f and head 10bb12bb56255daffd15b369988b6957e0d6e81c (Dong1017-authored #558 rename). Tip zengchuang-hw Approve present. Holding formal Approve until Ready full-check is green on this head (currently pending).

Clean-code / repo style (provisional): PASS

  • Byte-identical git mv of harness/scenarios → harness/trials (external.py / __init__.py digests match main).
  • No compatibility forwarder; RETIRED_MODULES + import-gate controls cover absolute/relative old imports and a returning harness/scenarios/ path.
  • Live importers/docs (runner, project_service, optimization_session, apps/legacy/run, architecture/module-layout/contributing/moe-forward) use harness.trials.
  • Remaining scenarios strings are intentional: frozen evidence pins, pre-change session fixtures, retirement tests, and historical CHANGELOG lines — not live API drift.
  • Release-boundary fixture pre-trials-rename-session/ + INFERRSI_PRE_CHANGE_SESSIONS wiring matches the #498/#558 evidence pattern.

Will re-check tip + Ready and submit formal LGTM Approve if unchanged and green.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 15f17037afb6a57a7a51348f73979f81f5e0721f and head 10bb12bb56255daffd15b369988b6957e0d6e81c.

Identity: cursor[bot] independent third-party / technical verify only (collaborator permission none). Not write+ / dual-tip / merge-ready under docs/agent-collaboration.md. Dong1017 is the author — this vote does not substitute for write+ independents (zeng + amy tip Approves already present).

Clean-code / repo style: PASS

  • Byte-identical git mv harness/scenarios → harness/trials (no forwarder).
  • RETIRED_MODULES + import-gate cover old absolute/relative imports and a returning harness/scenarios/ path.
  • Live importers/docs on harness.trials; leftover scenarios only in frozen evidence, pre-change fixtures, retirement tests, historical CHANGELOG.
  • Release-boundary pre-trials-rename-session/ + INFERRSI_PRE_CHANGE_SESSIONS matches the accepted #558/#498 pattern.

Verified: Ready required checks green on this tip (full-check 3.10/3.11, declarations, installed-package, lint). Prior tip COMMENT held only for Ready; tip unchanged. Not CI-green alone — rename tree and gate/docs were read earlier this round.

@Dong1017
Dong1017 merged commit af83696 into main Nov 21, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants