Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/ctrf-pr-report.hbs
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
### {{github.job}}: [Run #{{github.runNumber}}]({{github.buildUrl}})

| **Tests {{getCtrfEmoji "tests"}}** | **Passed {{getCtrfEmoji "passed"}}** | **Failed {{getCtrfEmoji "failed"}}** | **Skipped {{getCtrfEmoji "skipped"}}** | **Duration {{getCtrfEmoji "duration"}}** |
| --- | --- | --- | --- | --- |
| {{ctrf.summary.tests}} | {{ctrf.summary.passed}} | {{ctrf.summary.failed}} | {{ctrf.summary.skipped}} | {{formatDurationFromTimes ctrf.summary.start ctrf.summary.stop}} |

### {{#if ctrf.summary.failed}}{{getCtrfEmoji "failed"}} **Some tests failed!**{{else}}🎉 **All tests passed!**{{/if}}

{{#if ctrf.summary.failed}}
<table>
<thead>
<tr>
<th>Name</th>
<th>Failure Message</th>
</tr>
</thead>
<tbody>
{{#each (limitFailedTests ctrf.tests 5)}}
<tr>
<td>{{getCtrfEmoji "failed"}} {{name}}</td>
<td>{{#if message}}{{stripAnsi message}}{{else}}No failure message{{/if}}</td>
</tr>
{{/each}}
</tbody>
</table>
{{#if (gt ctrf.summary.failed 5)}}
<p><a href="{{github.buildUrl}}">See all failed tests here</a></p>
{{/if}}
{{/if}}
36 changes: 35 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ on:
jobs:
test:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v4

Expand All @@ -19,4 +22,35 @@ jobs:

- run: bun install --frozen-lockfile

- run: bun test
- name: Record test start time
id: test_start
run: echo "value=$(date +%s%3N)" >> "$GITHUB_OUTPUT"

- name: Run tests
run: bun test --reporter=junit --reporter-outfile=test-report.xml

- name: Record test stop time
id: test_stop
if: always()
run: echo "value=$(date +%s%3N)" >> "$GITHUB_OUTPUT"

- name: Convert test report to CTRF
if: always()
run: |
bunx junit-to-ctrf@0.0.14 test-report.xml -o ctrf/ctrf-report.json -t bun
jq '.results.summary.start = ${{ steps.test_start.outputs.value }} | .results.summary.stop = ${{ steps.test_stop.outputs.value }}' \
ctrf/ctrf-report.json > ctrf/ctrf-report.tmp.json
mv ctrf/ctrf-report.tmp.json ctrf/ctrf-report.json

- name: Publish test report
if: always() && github.event_name == 'pull_request'
uses: ctrf-io/github-test-reporter@e500b992f936420eb633c91644cf10d4d71df700 # v1.1.0
with:
report-path: ctrf/ctrf-report.json
custom-report: true
template-path: .github/ctrf-pr-report.hbs
pull-request: true
overwrite-comment: true
comment-tag: bun-test-report
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
## Context

`.github/workflows/test.yml` runs `bun test` on every push to `main` and every pull request (see `ci-test-workflow` spec). Results are currently only visible as raw console output in the Actions run log. Bun's test runner has a built-in `--reporter=junit --reporter-outfile=<path>` flag that emits a standard JUnit XML file, which is the common interchange format most PR-comment/reporting GitHub Actions consume — no custom parsing needs to be written in this repo.

The workflow currently runs with default (read-only) `GITHUB_TOKEN` permissions, which cannot post PR comments.

## Goals / Non-Goals

**Goals:**
- Post a single, updating PR comment with pass/fail/skip counts and failing test names whenever the `Test` workflow runs for a pull request.
- Keep the comment accurate across re-runs (edit in place, not one comment per push).
- Keep the change CI-only: no new package dependency, no change to `src/`.

**Non-Goals:**
- Historical trend reporting (e.g. flaky-test tracking across runs) — only the latest run's result matters.
- Coverage reporting in the comment — this change is about pass/fail visibility, not coverage (`--coverage` is a separate concern).
- Commenting on push-to-`main` or `workflow_dispatch` runs — there is no PR to comment on.

## Decisions

- **Report format: Bun's built-in JUnit reporter (`--reporter=junit --reporter-outfile=test-report.xml`).**
Alternative considered: parsing Bun's plain console/`--dots` output with a hand-written script. Rejected — JUnit XML is structured, and re-implementing a results parser to work around the console format is unnecessary work for a solved problem. Bun ships JUnit support natively, so no test-side dependency is added.

- **Comment posting: convert the JUnit report to CTRF (Common Test Report Format) via `junit-to-ctrf` (run through `bunx`, not added to `package.json`), then post it with `ctrf-io/github-test-reporter`, both pinned to a full commit SHA / exact version rather than a floating tag.**
Superseded an earlier draft of this decision that used `EnricoMi/publish-unit-test-result-action` directly against the JUnit file. Switched to the CTRF-based pair because the resulting comment (tests/passed/failed/skipped/pending/other/flaky/duration table, pass/fail banner, "this comment has been updated" footer) is the format the repo owner asked for by example, and `github-test-reporter`'s `overwrite-comment` + `comment-tag` inputs give the same upsert-in-place behavior without a hand-rolled Issues API call. Note: the action's `update-comment` input *appends* each run's report to the same comment rather than replacing it — `overwrite-comment` is the one that replaces the tagged comment's content in place, which is what "stays up to date" (one clean report, not an accumulating thread) means here.
Alternative considered: a hand-rolled `actions/github-script` step that reads the XML and calls the Issues API directly. Rejected as the primary approach — correctly finding-and-editing a prior comment, formatting counts/failure lists, and handling the "no comment yet" case is exactly what an existing, widely-used action already does, and hand-rolling it means maintaining parsing logic in-repo for no behavioral benefit.
Note: `github-test-reporter`'s `status-check` input (creating a GitHub Checks-API check run) defaults to `false` and is left off — an earlier iteration of this workflow enabled a similar option on the previous action and hit a `403 Forbidden` because check-run creation needs `checks: write`, which this job intentionally doesn't grant (see Risks).

- **Comment content: a custom Handlebars template (`.github/ctrf-pr-report.hbs`, wired via `custom-report: true` + `template-path`) instead of `github-test-reporter`'s built-in `pull-request-report`.**
The built-in `pull-request.hbs` template hardcodes an 8-column table (Tests/Passed/Failed/Skipped/Pending/Other/Flaky/Duration) with no input to drop columns. The repo owner only wants Tests/Passed/Failed/Skipped/Duration, so the built-in template can't be configured to match — a custom template (copied from and trimmed down from the built-in one, keeping the same failed-tests detail block) is the supported extension point for this (`docs/build-your-own-report.md`).

- **Duration: capture real wall-clock timestamps around the `bun test` step and patch them into the CTRF report's `results.summary.start`/`stop` (epoch milliseconds) via `jq`, rather than relying on `junit-to-ctrf`'s output.**
`junit-to-ctrf@0.0.14` unconditionally sets `summary.start`/`summary.stop` to `0` (confirmed in its `convert.ts` source — there's no CLI flag to populate them from the JUnit `time` attribute or otherwise), and `github-test-reporter`'s `formatDuration` helper renders "not captured" whenever both are `0`. Two extra steps (`test_start`/`test_stop`, using `date +%s%3N` — GNU `date`'s epoch-milliseconds idiom, available on the `ubuntu-latest` runner) record real timestamps as step outputs; the CTRF-conversion step then overwrites the zeroed summary fields with them before the report is published. `test_stop` uses `if: always()` so it still runs (and the report still gets a duration) when `bun test` itself fails.

- **Job permissions: scope `pull-requests: write` to the `test` job only, only for the steps that need it**, rather than widening default repo-level permissions. The install/build/test steps need no elevated permissions; only the comment-posting step does.

- **Trigger condition: run the report/comment steps only `if: github.event_name == 'pull_request'`**, and only after the test step, using `if: always()` on the report-generation and comment steps so a failing test run still produces a report (JUnit output is written regardless of pass/fail, and the comment should reflect failures too — see proposal's "posted regardless of pass/fail").

## Risks / Trade-offs

- [Third-party action/package compromise] → Pin `github-test-reporter` to a full commit SHA (not a tag) and `junit-to-ctrf` to an exact version (not a range) via `bunx`, reviewing each before adopting; this matches treating both as audited dependencies rather than floating references.
- [Job needs `checks: write` it doesn't have] → Leave `status-check` (and any other Checks-API-backed input) at its default `false`; the job only grants `pull-requests: write`, and a prior iteration of this workflow hit a `403 Forbidden` by enabling check-run creation without that permission.
- [`pull-requests: write` on a workflow that also runs `pull_request` on forked-repo PRs] → `pull_request` (not `pull_request_target`) is already the existing trigger, so the workflow runs with the forked repo's read-only token for cross-repo PRs; GitHub will not grant write permissions in that context regardless of the `permissions:` block, so the comment step will simply no-op/fail silently for fork PRs rather than expose a token to untrusted code. This is an accepted limitation, not a regression — same-repo PRs (the common case here) are unaffected.
- [Comment noise on force-pushed/rebased PRs] → Upsert-by-marker behavior (finding the prior bot comment by a hidden identifier and editing it) avoids duplicate comments; this is the action's default behavior and doesn't need custom logic.

## Open Questions

- None — if the chosen action's default comment format needs tightening (e.g. collapsing long failure lists), that's an implementation detail for `tasks.md`/the PR itself, not a blocking design decision.
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
## Why

The `Test` workflow (`.github/workflows/test.yml`) already runs `bun test` on every pull request, but a reviewer only learns whether it passed or failed by opening the Actions tab and reading raw log output. Surfacing a summary of the test results directly on the pull request removes that extra click and makes failures (and pass counts) visible where review already happens.

## What Changes

- Extend the `Test` workflow so that, on pull request runs, it generates a structured test report from `bun test` (via Bun's built-in `--reporter=junit --reporter-outfile=...`) instead of relying solely on console output.
- Add a step that posts (or updates) a single PR comment summarizing the test results: total/passed/failed/skipped counts and the names of any failing tests.
- The comment is upserted (edited in place on re-runs) rather than appended, so repeated pushes to the same PR don't accumulate duplicate comments.
- The report is posted regardless of whether the test run passed or failed, so the comment always reflects the latest run.
- Push-triggered and `workflow_dispatch` runs are unaffected — there is no pull request to comment on, so the report step is skipped for those triggers.

## Capabilities

### New Capabilities
- `pr-test-report-comment`: Posting/updating a single pull-request comment that summarizes the outcome of the `bun test` run (counts and failing test names) for that PR's latest workflow run.

### Modified Capabilities
- `ci-test-workflow`: The workflow gains a report-generation step (JUnit output from `bun test`) and a report-posting step that runs only on `pull_request` events, in addition to its existing trigger/install/test-execution behavior.

## Impact

- `.github/workflows/test.yml`: new steps to generate a JUnit report file and to upsert a PR comment from it; requires `pull-requests: write` permission for the job (currently unset, defaults to read).
- No changes to `src/`, package exports, or published package behavior — this is CI-only.
- No new runtime dependency for the package itself; any GitHub Action used to post the comment is a workflow-only dependency (pinned by SHA), not a package dependency.
- Does not require a changeset — CI-only change with no user-facing package effect.
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
## MODIFIED Requirements

### Requirement: Test execution determines pass/fail
The workflow SHALL run the project's test suite via `bun test`, produce a structured JUnit test report as part of that run, and report the run as failed if any test fails, or succeeded if all tests pass.

#### Scenario: All tests pass
- **WHEN** `bun test` completes with no failing tests
- **THEN** the workflow run is marked successful and a JUnit test report is produced

#### Scenario: A test fails
- **WHEN** `bun test` reports at least one failing test
- **THEN** the workflow run is marked failed and a JUnit test report is still produced, reflecting the failure
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
## ADDED Requirements

### Requirement: Test results are posted as a PR comment
When the `Test` workflow runs for a pull request, the workflow SHALL post a comment on that pull request summarizing the `bun test` run: total test count, passed count, failed count, skipped count, and the names of any failing tests.

#### Scenario: All tests pass on a pull request run
- **WHEN** the `Test` workflow runs for a pull request and `bun test` completes with no failing tests
- **THEN** a PR comment is posted showing the total/passed/skipped counts and zero failures

#### Scenario: Some tests fail on a pull request run
- **WHEN** the `Test` workflow runs for a pull request and `bun test` reports one or more failing tests
- **THEN** a PR comment is posted showing the total/passed/failed/skipped counts and the names of the failing tests, even though the workflow run itself is marked failed

### Requirement: Report comment is updated in place across re-runs
The workflow SHALL update its existing test report comment on a pull request rather than posting a new comment for every run.

#### Scenario: Pull request receives a second push
- **WHEN** the `Test` workflow has already posted a test report comment on a pull request and a new commit is pushed to that pull request, triggering another run
- **THEN** the existing test report comment is edited to reflect the latest run's results, and no additional comment is created

#### Scenario: First run on a pull request
- **WHEN** the `Test` workflow runs for a pull request that has no prior test report comment
- **THEN** a new test report comment is created

### Requirement: Report comment is scoped to pull request runs
The workflow SHALL only attempt to post or update a test report comment when triggered by a pull request event.

#### Scenario: Push to main
- **WHEN** the `Test` workflow runs because of a push to `main`
- **THEN** no test report comment step is attempted, since there is no pull request to comment on

#### Scenario: Manual workflow_dispatch run
- **WHEN** the `Test` workflow runs because of a manual `workflow_dispatch` trigger
- **THEN** no test report comment step is attempted, since there is no pull request to comment on
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
## 1. Generate a structured test report

- [x] 1.1 Update the `test` step in `.github/workflows/test.yml` to run `bun test --reporter=junit --reporter-outfile=test-report.xml`, with `if: always()` (or equivalent) so the report is written even when tests fail.

## 2. Post/update the PR comment

- [x] 2.1 Add `permissions: pull-requests: write` scoped to the `test` job in `.github/workflows/test.yml`.
- [x] 2.2 Add a report-posting step, pinned to a full commit SHA (e.g. `EnricoMi/publish-unit-test-result-action`), that reads `test-report.xml` and upserts a single PR comment with total/passed/failed/skipped counts and failing test names.
- [x] 2.3 Scope the report-posting step to run only `if: github.event_name == 'pull_request'` and only after the test step (`if: always()` combined with the pull_request check), so push/workflow_dispatch runs and always-run reporting both behave per the spec.

## 3. Verify

- [x] 3.1 Open a PR with a passing test suite and confirm a comment is created showing correct counts.
- [x] 3.2 Push a commit that introduces a failing test to the same PR and confirm the existing comment is updated (not duplicated) with the failure listed and correct counts.
- [x] 3.3 Confirm a push to `main` and a manual `workflow_dispatch` run do not attempt to post a comment.
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
schema: spec-driven
created: 2026-08-03
Loading
Loading