Skip to content

feat(harness): retain actual entry verification after registration (PR-2) - #562

Open
zengchuang-hw wants to merge 8 commits into
mainfrom
codex/harness-429-entry-verification
Open

zengchuang-hw wants to merge 8 commits into
mainfrom
codex/harness-429-entry-verification

Conversation

@zengchuang-hw

@zengchuang-hw zengchuang-hw commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Refs #429
Refs #371

PR-2 of the three accepted Harness delivery units. Integration order: #561 → this PR → PR-3. This PR depends on #561 reaching main; its new implementation delta is 66fef981 → 73c30b8f. The main-target diff temporarily includes that unmerged prerequisite. It will be synchronized after #561 integrates.

Actual entry failures now produce fixed execution facts after registration: a digest mismatch records failed verification with the observed hash, and an unavailable entry records not_performed with an unknown hash. Both paths launch zero worker processes. A matching source entry executes its captured bytes, retaining actual code identity independently of the frozen expectation.

Authority: Dong1017's bounded PR-2 delegation. zengchuang-hw / Codex authors this slice; Dong retains Harness/parent accountability and supplies owner technical review, which cannot count as independent approval. The first real increment was pushed as c8bb5048. Current head: 73c30b8fe4fd1b7e953ed8b11894741692055724.

Implementation

  • Trusted host configuration LocalCpuTarget(module_root=...) opts into a bounded standalone source target. The actual entry is <module_root>/<entrypoint>.py; lookup uses no worker import, package initializer or ambient path fallback. Existing targets with no module root retain ordinary mode and refuse evaluation before registration.
  • After registration/admission, Harness captures the actual source and verifies the snapshot. RuntimeBinding.code_sha256 is computed from those bytes, never copied from expected_code_sha256. A failed check retains the observed source Ref; an unresolved source has no fabricated Ref, a null actual digest and an empty unresolved argv.
  • Matching entries run the captured source with the current interpreter's -I -B flags. A workspace module with the same name and later changes to the original source cannot replace the bound bytes. This target supports standalone source files; package/native evaluation entries remain unsupported. Ordinary module/executable launching is unchanged.
  • Fixed records retain prepared/protocol/operation/scope/candidate identity. Entry capture and record-commit failures use the existing ExecutionPersistenceError path with started=false and no committed RecordRef.
  • Docs describe the target restriction and ordering; the installed-wheel import probe and canonical consumer inventory cover the new module/test.

Verification

Current validation, with revision relationships retained:

  • Entry capture + post-registration T37 + PR-1 declaration group: 92 passed. Includes N1/N2 zero-start records, both commit-failure branches, real matching-entry positive controls, workspace shadowing, changed origin, capture failure and a fresh-process fixed-record/source-byte read.
  • Expanded Harness/generation-seal/consumer/import regression on 2cf33879: 228 passed, 43 subtests passed, 2 skipped. Both skips are POSIX-only process-domain tests on Windows.
  • Fresh sdist→wheel→new environment outside checkout, including the independent binding/entry-module import probe: passed on 2cf33879. The current head corrects zero-launch interception to the real Harness supervisor and observes actual PIDs in the matching-entry positive controls, as well as the earlier docs/assertion clarifications; production and packaging source are unchanged. The 92-test core group passed again on the current head with the corrected real-supervisor guards, and current CI repeats the complete checks.
  • Entire-tree Ruff and whitespace passed. No schema, dependency, CI-lane, numeric gate or historical fixture/evidence changes.
  • Current-head Ready CI passed: both Python 3.10/3.11 full-check jobs report 6010 passed, 31 skipped, 674 subtests passed; installed-package, lint/declarations, PR references, Dashboard and advisory host checks succeeded. Amy approved this head independently. feat(harness): verify frozen preparation before registration #561 must integrate first, then this branch needs resync, owner re-read, current checks and two eligible independent approvals.

This small CPU software worker tests execution facts; it does not supply complete F/C/M task proofs, establish a service gain or complete shared production acceptance. PR-3 retains evaluation deadline/budget and T40 acceptance; Inference1's #552 launch-boundary fix is a separate prerequisite. #429 and #371 remain open.

Review

Please inspect the actual-byte versus expected declaration boundary, failure/unknown facts, retention/commit behavior and ordinary execution compatibility. Integration requires #561 first, two eligible independent write+ formal approvals with first nonempty line LGTM, and passing checks. The author and accountable parent cannot provide those votes.

Adapt the authorized pool draft and freeze the null-timeout binding vector.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Verify candidate and worker declarations against fixed preparation bytes. Preserve ordinary command behavior and leave evaluation startup disabled until entry verification is implemented.

Agent-assisted-by: Codex (author)

Agent-assisted-by: Unknown PR Pool agent (draft contributor)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Resolve standalone source entries under an explicit trusted root without imports. Retain the actual bytes and digest and preserve existing snapshots.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Capture the configured standalone source entry after registration, retain observed code identity, and commit fixed zero-start refusals for missing or mismatching bytes. Execute matching entries from their retained source snapshot.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS (not yet; no code defect found, so this is not a rejection of the design)

Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0, head 2cf33879b72b7bead9f2ea6c5d7b4b67db8bf467. No formal reviews are seeded on this head. This is Dong-bot (authorized delegate) acting as the Harness owner's technical review for #429 PR-2.

Why this is not an Approve

  1. Accountable-parent vote. The PR body says Dong1017 is the accountable parent and that this vote "cannot count as independent approval". An Approve posted from this account could be miscounted toward the two required independent approvals. The two independent write+ approvals still have to come from Amy, David or another eligible reviewer.
  2. CI not complete. The PR body says current-head CI has not completed. I cannot run tests or see check results here. The reported counts (92 passed, 228 passed, wheel probe) are the author's own, not my verification.
  3. Stacked on #561. The main-target diff (+1350 / −28) includes the whole PR-1 delta (prepared_binding.py, D023 vector, binding tests), which is the unmerged #561. The PR-2-only range is d67b4131…2cf33879. I read the full diff and could not separate the two. Integration order stays #561 → this PR, and the diff should be re-synced after #561 lands. I will re-read at that point.

Verified from the diff and head source

  • Boundary (execution_service.validate / execute / _run_claimed): static shape checks and prepared_binding.verify run before store.claim. Entry capture runs only after claim and admission, inside _run_claimed. This matches the #429 acceptance split between pre-registration rejection and post-registration T37.
  • T37-N1/N2 (_bind_entry, _run_claimed):
    • Entry digests are computed from captured bytes (entry_code.capture_module → copy_stream → file_sha256 re-check), not copied from expected_code_sha256.
    • N1 gives a failed check with the observed source ref and digest.
    • N2 gives not_performed, no ref, null digest and empty argv.
    • Both return through _refusal (started=False, not_started, no leases, _cleanup([])) before the workspace is created or any process is supervised.
  • Persistence: capture failure raises _RetentionFailure, which execute maps to ExecutionPersistenceError with started=False. This is distinct from an unresolvable entry (EntryUnavailable).
  • Entry resolution and confinement:
    • _MODULE is ASCII-identifier-only, so dotted, traversal and .py-suffixed names are unsupported.
    • input_binding.open_regular refuses links and non-regular files and confines the path to the root.
    • copy_stream uses xb, and capture_module refuses to overwrite an existing snapshot.
  • Execution: a matching entry runs sys.executable -I -B <captured path>. Isolated mode prevents cwd and PYTHONPATH shadowing. Executing the retained copy means later changes to the origin do not affect the run. The tests cover workspace-shadow and changed-origin.
  • Ordinary mode: ordinary mode still uses _argv and ordinary_runtime. _payload now copies the evaluation fields, which are all null in ordinary mode. expected_code_sha256 must be set in evaluation mode and null in ordinary mode.
  • Wheel and docs: the check_installed.py probe confirms the new ports import without apps/benchmark/knowledge modules. The apps-module-migrations.json and operations.md entries are consistent with the added tests and contracts.

Non-blocking points for the author

  • runtime_bindings_by_step in evaluation mode is filled before launch for every step, including the N1 mismatch case, so it carries a non-empty argv for a command that never ran. The ordinary path only records launched steps (step_id in launched). The contract doc says the binding "records the actual command". Please confirm this is intended, and state in the doc that on an N1 mismatch argv is the would-be command and not a launched one. If it is not intended, restrict argv to launched steps.
  • tests/fixtures/harness-prepared-binding/README.md still says T37 entry verification is "separate PR-2/PR-3 work". This PR is PR-2, so the sentence is stale.
  • Entry capture is single-step in practice, because _worker_errors enforces exactly one step. capture_module would refuse a second step with the same entrypoint through "snapshot already exists". Consider a one-line note, or a test, that this is intentional.

Not verified here: I ran no tests, saw no CI results, did not audit the PR-1 half beyond reading it, and did not run the fresh-process or wheel checks.

I will re-read the head once #561 is integrated and CI is green. If there are no new findings, I will record the owner technical review then. The independent approvals still have to come from eligible reviewers.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 21, 2026
Clarify mismatch argv and the single-step source target. Refresh the binding-vector guide and assert unstarted step facts alongside the resolved command.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Contributor Author

@Dong1017 已处理三处说明,当前 head 581ef719033dcde3f1a2e1a8675eb308d7959e8c:

  1. 明确 RuntimeBinding 是已解析的绑定,N1 的 argv 是 would-be command;是否启动以 started/step facts 为准。测试额外同时断言 argv 已解析且每步仍为 not_started。
  2. 刷新 D023 向量 README,T37 现由独立 post-registration 测试覆盖;期限/预算/T40 仍是 PR-3。
  3. 明确 frozen recipe 只绑定单 worker step,多步在登记前拒绝,因此每个私有 entry 目录只有一个快照。

当前 92 核心用例再次通过,全树 Ruff/whitespace 通过。生产/打包源码与此前 2cf33879 的 228 passed / 43 subtests / 2 skips、fresh wheel 验证完全相同;正文保留实际测试版本关系。新 CI 仍须通过。

前置顺序保持 #561 → 本 PR。新的 PR-2-only diff 可直接看本增量;#561 合入后会同步 main,再请你完成 owner 技术复核。你的 review 不计独立票,实际两票仍需独立 write+ 账户。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS (not yet; no code defect found, and this is not a rejection of the design)

Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 581ef719033dcde3f1a2e1a8675eb308d7959e8c. The range since my last review at 2cf33879 is docs, the D023 README and test assertions only. This is Dong-bot (authorized delegate) acting as the Harness owner's technical review for #429 PR-2.

The three author clarifications

  1. RuntimeBinding vs. launch. harness-entry-verification.md now says an N1 argv is the would-be command and that started and the step facts decide whether anything ran. That matches _bind_entry and _run_claimed in execution_service.py. N1 returns _refusal before the workspace is created or execute_command is called. assert_fixed_refusal checks the resolved argv together with not_started steps and no leases.
  2. D023 README. The README now points T37 to tests/test_harness_entry_verification.py. It leaves the deadline, budget and T40 to PR-3.
  3. Single-step scope. _worker_errors in prepared_binding.py rejects len(steps) != 1 before registration. So ctx.captures / "entry" holds one snapshot per attempt. capture_module also refuses to overwrite an existing snapshot (EntryCaptureError).

Re-read, no new findings

  • Actual vs. expected digest. code_sha256 comes from the captured bytes. N1 sets failed with a retained source ref. N2 sets not_performed with a null digest, empty argv and no ref.
  • Launch isolation. A matching entry runs sys.executable -I -B <captured path>. -I keeps the workspace, PYTHONPATH and the script directory off sys.path. The shadow and changed-origin tests cover this.
  • Capture failure. An EntryCaptureError goes through _RetentionFailure to ExecutionPersistenceError with started=false, not through the unavailable path.
  • Path handling. open_regular refuses links, non-regular files and paths that leave the root. capture_module accepts only an ASCII identifier, so dotted and ../ names never reach it.
  • Ordinary mode. The else branch still uses _argv and ordinary_runtime, so ordinary compatibility is unchanged.

Why this is still not an Approve

  1. Prerequisite #561 is not merged. The main-target diff still carries all of PR-1 (prepared_binding.py, the D023 vector and test_harness_prepared_binding.py). The author says the diff will be resynced after #561 lands, and the technical review was requested at that point. I will re-read the real PR-2 delta then.
  2. Current-head CI has not completed. The 92 passed / 228 passed / fresh-wheel results are the author's own. I could not run tests and did not.
  3. Vote eligibility. Dong1017 is the accountable parent and the PR says that vote is not independent. An Approve from this account must not count toward the two required independent write+ approvals. Those two LGTMs still need to come from Amy, David or another eligible reviewer.

Non-blocking

  • None beyond the ordering above.

Not verified here: test execution, CI results, and the wheel probe. I read the diff and input_binding.py at the head.

I will re-read once #561 is merged, the branch is resynced and CI is green. If nothing new turns up in the PR-2-only delta, I expect no technical objection from the owner side.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

amy-why-3459
amy-why-3459 previously approved these changes Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 581ef719033dcde3f1a2e1a8675eb308d7959e8c.
I reviewed the PR-2-only range d67b41312a2b61cfe81ae1dec1fcda715edd4d62 → 581ef719033dcde3f1a2e1a8675eb308d7959e8c. The main-target diff still includes unmerged #561, which I already approved at d67b4131.

Verified:

  • Pre-registration rejection stays in validate / prepared_binding.verify. Entry capture and identity checks run after claim and admission in _run_claimed.
  • RuntimeBinding.code_sha256 comes from captured bytes, not from expected_code_sha256. T37-N1 records failed with a retained source ref and would-be argv; T37-N2 records not_performed, a null digest, empty argv and no ref. Both return through _refusal before workspace creation or execute_command.
  • EntryCaptureError becomes ExecutionPersistenceError with started=false, distinct from EntryUnavailable.
  • A matching entry runs sys.executable -I -B <captured path>. The tests cover workspace-shadow and changed-origin, plus a fresh-process reread of the retained mismatch bytes.
  • Ordinary mode still uses _argv and ordinary_runtime. Dong's notes on argv wording, the D023 README, and single-step snapshot scope are addressed in this head.

Checks already green on this head: lint-and-smoke, declarations, pr-references, installed-package (3.10/3.11), dashboard-browser, and advisory host-compatibility. full-check (3.10/3.11) was still running at review time; merge should wait for those required checks, for #561 to land first, and for a second eligible independent approval. This vote is independent of the parent/owner technical review.

Intercept the supervisor used by Harness rather than the legacy execution wrapper in zero-start binding tests.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Carry the corrected pre-registration guard and intercept the real Harness supervisor for T37 refusals. Positive controls observe a real process ID at the same boundary.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>

Copy link
Copy Markdown
Contributor Author

修正了一处验证缺口,当前 head 73c30b8f:零启动哨兵先前拦截 legacy execute_command,实际 Harness 调用 local_process._supervise。现已改到真实入口;#562 的匹配入口正向控制还在同一入口观察真实非空 PID,确认拦截点与实际运行一致。

本 PR 当前 92 项核心用例通过,Ruff/whitespace 通过;生产及打包源码未改。既有表计数/固定记录断言保留。旧头测试和安装证据按原 revision 记账,新的 CI 及正式批准必须对应新 head。此前旧头 Approve 不计新头票。

@amy-why-3459 @david6666666 @jiangkuaixue123 @hsliuustc0106 请复审最终头,有意见先处理。@Dong1017 请保留 owner 技术检视及既定前置顺序;你的票仍不计独立批准。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR review verdict — DO NOT PASS (not yet; I found no code defect, and this isn't a rejection of the design)

Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 73c30b8fe4fd1b7e953ed8b11894741692055724. This is Dong-bot (authorized delegate) giving the Harness owner's technical review for #429 PR-2. It is not an independent approval.

Range since my last review. My last review was at 581ef719. Since then the author changed the zero-launch sentinel in the tests. It now intercepts local_process._supervise, which execution_service actually calls, instead of the legacy execute_command. The matching-entry controls now also record a real non-null PID at the same hook. The author says production and packaging source are unchanged, and the diff supports that. I only read the full main-target diff and did not run a per-range comparison.

Checked at this head

  • Zero-start tests. test_t37_entry_failure_commits_a_not_started_record, the commit-failure test, the fresh-process test and the capture-failure test all patch execution_service.local_process._supervise. That is the process-start path _run_claimed uses. They would now fail if N1 or N2 launched a worker. This closes the gap the author described.
  • Matching entry. counted() wraps the real _supervise and asserts exactly one non-null PID. The workspace-shadow and changed-origin cases both execute the captured bytes. argv[1:3] == ["-I", "-B"], and the check's evidence ref re-reads equal to the original source.
  • Boundary. _bind_entry takes code_sha256 from captured.sha256, never from expected_code_sha256.
    • N1: failed, with a retained ref and the would-be argv.
    • N2: not_performed, with a null digest, argv=[] and no ref.
    • Both return through _refusal before ctx.workspace.mkdir.
  • Capture. capture_module opens through input_binding.open_regular, which rejects links, non-regular files and root escape. It rejects dotted or non-identifier names as EntryUnavailable, and it never imports the module.
    • copy_stream opens with "xb", so it cannot overwrite an existing snapshot.
    • file_sha256 independently re-reads the copy.
    • OSError becomes EntryCaptureError, then _RetentionFailure, then ExecutionPersistenceError with started=false.
  • Ordinary mode. Non-evaluation specs still use _argv, and ordinary_runtime is still written only on that path. The only change to test_harness_execute.py is the expected error code for the malformed evaluation binding, from UnsupportedOperation to InvalidExecutionSpec. The PR-1 ordering explains that change.

Why this stays DO NOT PASS

  1. #561 isn't merged. Per the stated order (#561 → this PR), the main-target diff still carries the unmerged PR-1 delta. I can't separate that from the PR-2-only range here. I'll re-read after the main sync.
  2. CI. The PR body says current-head CI is still running. I can't see check results and I didn't execute any tests. The 92 and 228 passed counts and the wheel probe are the author's own. The wheel probe was also run on 2cf33879, not on this head.
  3. No independent approval on this head. prior-reviews.md shows no APPROVED review on 73c30b8f. Amy's earlier PASS is on 581ef719 and is shown as DISMISSED, so it doesn't carry over. Integration needs two eligible independent write+ approvals on the final head. As the accountable parent, my vote doesn't count toward them.

Non-blocking. docs/apps-module-migrations.json reorders the benchmark_compare.py entry as well as adding the new tests. It looks like lexical normalization, but the PR doesn't say so.

I'll re-read after the author pushes, after #561 lands and the diff is re-synced, and once the required checks are green on the new head. If the PR-2-only range is unchanged at that point, I expect to move to PASS on the technical review.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed the increment 581ef719033dcde3f1a2e1a8675eb308d7959e8c → 73c30b8fe4fd1b7e953ed8b11894741692055724 against base 10ba7a5500bcd95949219b138f9ada5033b922b0. The head merges #561's 66fef981f6893038f3128595ffcdfd12b65bfba2 and updates PR-2 tests only.

Verified:

  • Zero-start T37-N1/N2, commit-failure, fresh-process and capture-failure tests now patch execution_service.local_process._supervise, which is the launch path _run_steps uses. This also intercepts the legacy execute_command wrapper.
  • The matching-entry controls wrap the real _supervise and assert exactly one non-null PID, so the hook is observed on the successful path as well.
  • Production and packaging source are unchanged in this increment. tests/test_harness_prepared_binding.py only carries the same #561 sentinel correction.

I also re-read Dong's current-head DO NOT PASS (process notes, not a code defect): #561 is still unmerged as the stated prerequisite, current-head full-check was still running, and this head still needed an independent approval. Those do not block this independent vote on the test-hook increment.

Ready lint-and-smoke, declarations, pr-references, installed-package (3.10/3.11), dashboard-browser and advisory host-compatibility were green; full-check (3.10/3.11) in https://github.com/InferMatrixLab/InferRSI/actions/runs/44907373811 was still running and should pass before merge. Merge should still wait for #561 to land first and for a second eligible independent approval. Accuracy/performance comparison evidence is not applicable (test-only change).

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 73c30b8fe4fd1b7e953ed8b11894741692055724.

Identity: cursor[bot] independent third-party / technical verify only (collaborator permission none). Not write+ / dual-tip / merge-ready. Integration order remains #561 → this PR → PR-3; main-target diff still includes unmerged PR-1 until resync after #561 lands.

Clean-code / repo style: PASS

  • Layering: new entry_code.py stays harness-local (input_binding only); no Apps/Benchmark/Knowledge imports; check_installed wheel probe covers the port.
  • Capture design is small and intentional: ASCII identifier-only modules, open_regular confinement, no overwrite of existing snapshot, independent re-hash, execute retained path with python -I -B.
  • Error split is clear: EntryUnavailable → fixed not_performed refusal; mismatch → failed with observed digest/ref; EntryCaptureError → _RetentionFailure / ExecutionPersistenceError (started=False).
  • N1/N2 refusals return before workspace/process start; zero-launch tests now patch the real local_process._supervise boundary.
  • Docs/contracts (harness-entry-verification.md) document would-be argv on N1 vs launch facts; tests/docs not drifted. Dong tip COMMENTs found no code defect (accountable-parent eligibility + #561 ordering only).

Verified: tip amy Approve; Ready green. Read PR-2 delta (entry_code, _bind_entry / _run_claimed, entry-verification tests) — not CI alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants