Skip to content

feat(cli): a feedback report names why the last check failed - #6717

Open
miguel-heygen wants to merge 2 commits into
mainfrom
checklong/feedback-last-check
Open

miguel-heygen wants to merge 2 commits into
mainfrom
checklong/feedback-last-check

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What changed

hyperframes feedback already attaches join keys to a report (fid, tid, recent render ids). It now also attaches check=exit<code>:<error code>*<count>,... for the newest local check: the exit code and up to three error codes with how often each fired. Agents sometimes file a check failure with the error left as a placeholder, so the report named no reason and could not be looked up. check records that summary in the local config next to the recent-renders ring after every run. Codes and counts only: no messages, no paths, no project content. Each code is cut to 24 characters so the keys stay inside the report's 500-character env cap.

What I measured

feedback.telemetryJoinKeys.test.ts, feedback.test.ts, check.test.ts and checkBrowser.test.ts pass (233 tests). With feedback.ts, checkPipeline.ts and config.ts at main's versions the three new tests fail (156 pass, 3 fail). A full ring of render ids plus three long error codes stays under the cap.

What I did NOT exercise

A real check then a real feedback on one machine; the pipeline test reads the saved record and the key builder is tested on its own.

Why under 100 lines

A 25-line summariser, a small config record, a 4-line join key and one pipeline call, plus the tests.

fails without the fix: remembers the failed check's error codes so a later feedback can name them

@heygengenesis heygengenesis Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES · 1 blocker · 1 important · 0 nits · head adc5142

A check that throws never updates lastCheck, so feedback attaches the result of an older check. ↓ inline

Scope

Read all 6 files, plus check.ts (run/reportCheckFailure), writeConfig/withConfigLock, and the test-home setup. The test is isolated: the config cache means readConfig() reads back the in-process write. Codes are fixed rule ids, so no content leaks. The fails without the fix: test exists and fails on main. CI (CI, Windows, CodeQL) was still running at review time, so this verdict does not depend on it.

Comment thread packages/cli/src/utils/checkPipeline.ts Outdated
exitCode: checkExitCode(report),
runId: getRunId(),
});
recordLastCheck({ exitCode: checkExitCode(report), codes: failureCodes(report) });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blocker: this only runs when buildReport is reached. A check that throws (CheckDeadlineError, CheckStoppedError, project lock, a pipeline crash) goes through reportCheckFailure in commands/check.ts and exits 1, but lastCheck keeps the previous run. Example: check passes on project A, then check on project B times out, then feedback sends check=exit0. The report says the newest check passed, which is the opposite of the truth, in the same empty-error case this PR targets.

Fix: in reportCheckFailure, record { exitCode: 1, codes: { [usage ? error.code : "runtime_error"]: 1 } }, or clear lastCheck when run starts. Add a test where runPipeline throws.

important: also store an at timestamp, the way RecentRenderRecord does. Without one, triage can't tell a check from a minute ago from one last week on another project.

@github-actions

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 271 (base branch 271), smooth 129 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant