Repository navigation
feat(benchmark): define the pinned CPU evidence smoke task - #569
zengchuang-hw wants to merge 4 commits into
Conversation
Keep the documentary fixture unchanged. Grade a captured answer against a fixed public byte-sum reference, and measure actual AB/BA operations independently of correctness, without performance thresholds. Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Dong1017
left a comment
There was a problem hiding this comment.
PR review verdict — DO NOT PASS
Reviewed base abc6de4ddd36a01c37187b64640e81015549bc50 and head 79800f8b1ee5cd32413ef8592c738e3e54256731 (full diff, 5 files, diffComplete: true). There is no formal review on this head and no outstanding CHANGES_REQUESTED. This review is the first independent look. Dong-bot (authorized delegate) is reviewing against the p1-cpu-smoke@1 scope accepted in #565 (comment 7064895332).
Blockers
-
No tests ship with the change. The diff touches only
docs/contracts/p1-cpu-smoke.md,benchmark/cpu_smoke.py,benchmark/preparation.pyandbenchmark/workers/{__init__,fixture_worker}.py.tests/already hastest_benchmark_preparation.pyandtest_harness_*, but nothing covers the new behavior. The doc's own acceptance sequence requires each of the following. None of them is exercised or pinned in the diff:- the real correct answer (C passed);
- a valid UTF-8 wrong answer (
response_mismatch); - invalid UTF-8;
- a missing candidate file (grader
None, which the doc says gives no C proof); - timing validity and order, using an injected
clock; protocol()→PreparationService.prepare(default_request())succeeding end-to-end with the worker and baseline closure bytes;- the unchanged
p1-fixture@1behavior after the_supported_taskchange inpreparation.py.
The
clock=seam inevaluate()exists but no test uses it. Please add unit tests for these paths. -
The pins are not in the reviewable diff. The doc says exact worker/source/input pins must be recorded in the PR before acceptance runs. The change has no
sha256or size forfixture_worker.pyand noINPUT_BYTESpin. If they are in the PR body, please state them against this head. Any later edit tofixture_worker.pychangescode_sha256, so the pin must name this head.
Checked and looks consistent
- Task registration in
preparation.py:_supported_taskis applied symmetrically in_protocoland_prepared.PreparationService._preparestill requiresrequest["task"] == protocol["task"], so a request cannot switch tasks. protocol()output: its shape agrees with_recipe,_requirements, the claim registry,sampling,feedback_visibilityandmetric_definitions == [].worker-codeandbaseline-codeare distinct roles over the same bytes, so the closure-role uniqueness check passes.PreparationServicebindsworker_role="worker-code"tocode_sha256._milliseconds: I traced1_000_000,1_500_000and10_000_000ns to"1","1.5"and"10". This matches_DECIMAL(no trailing zeros), andend <= startis treated as an incomplete sample, so there is no clamping or fill.evaluate(): the grader result is independent of sampling, and a read error only nulls the grader or samples.workers/has an__init__.py, sofindwithnamespaces = falsepackages it and the.pyis included without apackage-datachange.
Non-blocking
cpu_smoke.pyreaches into privatepreparation._require,_idand_CANDIDATE. A small public helper would be cleaner.main()catches onlyOSError. An unexpected exception writes no report. Confirm that is the intended closed-report behavior.
Not verified: I ran no tests or CI. I did not read the PR body. I did not check stack ordering against #566/#567/#568 or the final-head Amy and David approvals. I will re-read after the author pushes tests and the pins.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
Cover actual worker sampling, independent claims, retained reference failures and original-attempt replay without changing the pinned worker, input or criteria. Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
开始首轮任务用例前记录:测试计划已在 |
|
首轮失败记录和采样实现修订申请(不改验收门槛):
|
|
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5). Accepted: the worker's default clock may change from Accepted clauses
Conditions (non-blocking next steps for the author)
DetailsBasis: the first run shows C passed but samples=null. Your clock-info evidence points at the 0.015625 s tick of The diff at head I did not run any tests. I judged this from the thread text and the diff. Windows/macOS advisory CI stays non-blocking, as you noted. This decision replaces nothing earlier. It refines the #565 accepted p1-cpu-smoke@1 semantics (comment 7064895332) by changing the clock source only. |
Preserve the input, operations, order and validity rules. Keep the coarse-clock run inconclusive and freeze the new source before its fixed cases. Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
|
Revision 2 first run is preserved: 16 passed, 1 failed, stopping at the first real pipeline case. The three actual standalone correct/wrong/invalid-UTF-8 workers passed and emitted complete positive canonical samples. The real pipeline reached the fixed record with its four proof families, then Apps received |
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang-hw <zengchuang3@huawei.com>
|
@Dong1017 请复核当前 @amy-why-3459 @david6666666 请各自独立检视并按实际结论提交正式 review;有意见我先修复。当前为 Ready,临时 base 仅用于窄 diff review,不可向其合入。#566/#567/#568 实际合入 main 后再同步转 main、刷新最终头 CI 和两份独立审批。Dong 的 parent/owner 检视不计独立票。 |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS (owner/parent scope-and-diff review; not an independent vote)
Reviewed base abc6de4ddd36a01c37187b64640e81015549bc50 and head 2e203ae6f1810fe9e52f09d5c1b51f086465e93a (full diff, diffComplete: true). There is no formal review on this head and no outstanding CHANGES_REQUESTED. My earlier DO NOT PASS was on 79800f8b. Dong-bot (authorized delegate) is posting this. As the PR text says, it does not count toward the two required independent approvals from Amy and David.
Earlier blockers, resolved on this head
- Tests.
tests/test_benchmark_cpu_smoke.pyis new and covers the paths I listed:- the real standalone worker run via
-I -Bon correct, wrong and invalid-UTF-8 candidates; - four samples in A1/B1/B2/A2 order even when C fails, checked through the action trace
B A A B B A; - zero, decreasing and
Trueclock values givingsamples=Nonewith exactly 8 clock reads and no retry or fill; - a missing candidate giving
grader=None, samples=None; - integer ns→ms canonical conversion;
- the prepare closure for
worker-codeandbaseline-code, with bytes equal to the source; - unknown or mixed task/grader versions rejected as
UnsupportedRuleVersionby both the prepare port andvalidate_assessment; - five real prepare → execute/capture → assess → view → replay cases, including a publication failure and a missing candidate. Expected claim verdicts are PASS/PASS/PASS, PASS/FAIL/PASS, FAIL/FAIL/PASS and INCONCLUSIVE×3, with replay not re-launching or re-capturing.
- the real standalone worker run via
- Pins. The PR body now records the worker,
INPUT_BYTESand candidate SHA-256 values and sizes, with the retained failure history (revision 1samples=null; revision 2 first run 16 passed and 1 failed). - Registration gap.
_supported_taskinpreparation.pyis the single registration point.assessment.validate_assessmentnow calls it. Thep1-fixture@1 / p1-rules@1pair stays accepted, and mixed pairs are rejected.
Accepted clock revision. Only the default clock changes, to time.perf_counter_ns in fixture_worker.evaluate. Inputs, operations, order, the positive-value rule and the "no clamp, fill or retry" rule are unchanged. This matches my acceptance in comment 7065967644. The contract doc names perf_counter_ns.
Not verified here
- I ran no tests. The 25-case pass, the sdist/wheel install and the Ruff results come from the PR text, not from my own run.
- I could not recompute the worker SHA-256 (
0f17b906…, 2264 bytes) from the diff, so I could not confirm that hash and size. The author should keep the pin check tied to the packaged bytes. - The expanded frozen-fixture, old CLI, plugin and view compatibility group was still running when the PR text was written, and I saw no CI result for this head.
- The body's contract link points at
79800f8b, which predates the clock revision. The in-diff doc is current. Please repoint the link to the head.
Non-blocking next steps
- Append the final result of the expanded compatibility group to the PR.
- Do not merge into the temporary review base. After #566, #567 and #568 reach main, sync, retarget main, and refresh CI. Amy and David then need fresh independent approvals on the final head.
- The shared D023/t108-v4 production acceptance must be re-run on merged dependencies. This PASS is not a claim of model quality, performance gain or service acceptance.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
|
Validation finished on @amy-why-3459 @david6666666 @Dong1017 the complete development-head validation and preserved failures are now recorded. Native diff is the 10-file CPU task increment against #568; no code/head change accompanied this result update. Final integrated-main CI/review and shared acceptance still remain, and the temporary review base is not a merge destination. |
|
当前头 owner 复核已收到:6006846156 明确清除原测试/pin/版本登记阻塞;owner 票仍不计独立审批。两项非阻塞补充现已完成:扩大组 630 passed / 7 skipped / 3 existing tarfile warnings 已补入正文,contract 链接已改为当前 仍等待 Amy/David 的独立正式 review,后续按依赖上 main 后刷新最终头 CI/审批;没有向临时 review base 合入,也没有宣告共享生产验收完成。 |
Refs #565
Refs #371
Refs #434
Revision 2 pins frozen before its cases. Current source/test head:
2e203ae6f1810fe9e52f09d5c1b51f086465e93a(result-validator registration only; worker unchanged). The owner accepted only the default clock implementation change; inputs, operations, order and validity rules are unchanged. This Ready PR records the real source/input pins first, following the owner's accepted task spec and sequencing condition. Revision 1 atd6546460produced one failed correct-case test: C passed but samples=null; 13 separate non-acceptance unit tests passed. The raw report and XML are retained as insufficient M evidence. Revision 2's first run atbebd7bf4is retained: 16 passed, 1 failed, stopping at the first pipeline assessment. Real worker cases produced complete samples, but the B result validator omitted the accepted new task/grader pair and returned UnsupportedRuleVersion. The2e203ae6follow-up registers that exact pair and adds four unknown/mixed/assessor rejection regressions; worker/source bytes and inputs below are unchanged. Static Ruff/whitespace passed. After this source head was recorded, all 25 focused cases passed, including the five real prepare → execute/capture → assess/publish/replay paths. Fresh sdist → wheel → independent installed-package verification passed on the exact head. Expanded compatibility tests completed: 630 passed, 7 skipped, 3 existing standard-library tarfile deprecation warnings, no failures/errors (includes the 25 focused cases). No acceptance PASS is claimed.Bounded scope: zengchuang-hw / Codex implements the explicitly delegated B worker/profile slice, not the PR Pool. This adds
p1-cpu-smoke@1 / p1-cpu-smoke-rules@1, keepingp1-fixture@1behavior and frozen vectors unchanged. Dong1017 remains the accountable/acceptance owner, not an independent reviewer.Review-only stack: base
codex/apps-565-evidence-captureat #568abc6de4ddd36a01c37187b64640e81015549bc50; current source head2e203ae6f1810fe9e52f09d5c1b51f086465e93a. Worker/profile-only increment. Dependencies are #566, #567 and #568 and their existing predecessors/#552 condition. Do not merge into the temporary review base. After all three reach main: sync, retarget main, run final-head checks and obtain Amy/David's two fresh independent formal approvals before squash there.Frozen source and inputs
The following were calculated by reading/parsing source bytes only, before any worker or acceptance execution:
0f17b902c3b90b1a75321ecc2181019a5c3084ca2bffeee775ce86aabe019903cca2fa85fcc1ad89b23d709ecc809786766da48cc95b5f931826ec986f0f6597b6899661828d50335325d289b9cedccdb3dbcbf5b64f094e14896c734bd9c39b543df89fec85b1c280e5be7bc6a33e31203503cd6edb3084312de4db5a9b436ce4688624e5f1ad0629505e6768e3bb36244f2f3e33e751215afa820334a76ed3Revision 2 source anchor pins the implementation. Changes to source/input pins, if required, will be recorded before any new cases; old failures/results remain in the record. No timing value, threshold or criterion will be retuned after observing results.
Revision 1's worker
c0fce99fbae3a104f21a74ae88aad9b022e532cdb0090a36f375ece20442d005(2261 bytes), source anchor79800f8b, and failed correct-case report remain recorded; they are not replaced with a PASS. The new default istime.perf_counter_ns, which is monotonic integer-ns here. No workload/input or threshold changed.Implementation
Validation on recorded source head
25 focused tests passed on
2e203ae6: real standalone correct/wrong/invalid-UTF-8 worker execution; C-independent four-trial order and canonical positive timing; failed/non-positive clock and missing-input gaps without re-measurement; full resource closure; unsupported/mixed task/grader/assessor rejection; five actual prepare/capture/assess/publish/replay cases. Correct: F/C/M all PASS for this software evidence-smoke task; wrong: PASS/FAIL/PASS; invalid UTF-8: FAIL/FAIL/PASS; missing candidate: all INCONCLUSIVE. These are task-local claim results, not model-quality or service-gain acceptance.Publication failure preserves the already committed RecordRef and readable proof references. Replay of completed or refused attempts returns the original attempt/record without consulting the retired capture/reader or launching another worker. Standard manifest/result reads, fixed attempt/trace refs and the assessment match. Fresh sdist/wheel installation outside checkout passed, including the cold pure B profile/owned-resource probe and existing public Apps/plugin entrances. Entire-tree Ruff and whitespace passed. The expanded group completed on this exact head: 630 passed, 7 skipped, 3 existing stdlib tarfile deprecation warnings, no failures/errors. It includes those 25 focused cases, frozen P1 prepare/assess/proofs, real H binding/entry/capture/replay, Apps output/publication/fixed report/Dashboard/trace views, exact migration/import boundaries, Codex/DSH/Workswarm entrances, Cannbot boundaries/channels and MindIE context/project/runtime. Five skips need an approved external Cannbot checkout with a built index; two POSIX process-group tests are inapplicable on this Windows host. Those paths remain unverified locally.
The shared D023/t108-v4 production acceptance still requires merged dependencies and a re-run on them. Development-stack tests are reported separately. #371/#429/#430/#434 remain open. No approval is prefilled: Amy/David formal independent review and Dong owner re-review are now requested on the actual source head; final main integration requires fresh checks/reviews after dependencies.