Skip to content

fix(devel): make the lab compose stack work end to end - #3797

Open
e00dan wants to merge 3 commits into
chainloop-dev:mainfrom
e00dan:fix/cas-lab-staging-dir
Open

e00dan wants to merge 3 commits into
chainloop-dev:mainfrom
e00dan:fix/cas-lab-staging-dir

Conversation

@e00dan

@e00dan e00dan commented Nov 8, 2026 •

Copy link
Copy Markdown

Summary

The lab stack (devel/compose.labs.yml, the README's "Self-host / Local stack") doesn't work on current releases. The CAS crashes on startup because of two regressions, and once it starts, attestation push with a real storage backend fails because the control plane can't reach the CAS.

  1. The public key path points at the private key. CAS_PUBLIC_KEY_PATH is set to /devkeys/cas.pem, which is the EC private key. The public key is cas.pub, which the CAS devel config already uses as its default. This was harmless while the key was parsed per request, but since perf(artifact-cas): parse the EC public key once instead of per request #3225 (v1.105.0) it is parsed at startup, and the CAS panics with failed to parse public key: x509: malformed tbs certificate.
  2. There is no writable staging dir. feat(artifact-cas): verify uploaded content against the declared digest #3365 (v1.110.0) made staging_dir required. The image is FROM scratch with USER 1001 and has no directory that user can write to, so the CAS panics before it reaches the key. The Helm chart mounts an emptyDir at /tmp-staging-fs; the compose file has nothing equivalent.
  3. The control plane dials itself instead of the CAS. With a non-inline CAS backend, the control plane uploads the attestation bundle to the CAS at cas_server.grpc.addr, which app/controlplane/configs/config.devel.yaml sets to 0.0.0.0:9001. Inside the control-plane container that address is the container itself, so attestation push fails with an internal error after the dial times out (creating the gRPC client: ... DeadlineExceeded). The value has no env placeholder, so compose couldn't override it. This one predates the two regressions; it only shows up with a non-inline backend.

Changes

All changes are dev-only. The production images and the Helm chart are unchanged.

  • devel/compose.labs.yml (CAS):
    • point CAS_PUBLIC_KEY_PATH at cas.pub
    • mount a tmpfs owned by the CAS user at Helm's staging path (/tmp-staging-fs:uid=1001,gid=1001)
    • set CAS_STAGING_DIR: /tmp-staging-fs
  • app/controlplane/configs/config.devel.yaml: cas_server.grpc.addr becomes ${CAS_SERVER_ADDR:0.0.0.0:9001}. The default is unchanged, so running the control plane on the host behaves as before.
  • devel/compose.labs.yml (control plane): CP_CAS_SERVER_ADDR: cas:8001. Like the Helm chart, this targets the CAS's shared endpoint rather than the deprecated gRPC-only listener on 9001.

This works with the released :latest images, so the lab stack works as soon as this is merged.

Testing

On Linux arm64 (Docker 29.9.3, rootless), with docker compose -f devel/compose.labs.yml up and the released artifact-cas:latest and control-plane:latest images:

  • The CAS loads /devkeys/cas.pub and listens on :8001 and :9001. The staging dir is a tmpfs owned by 1001.
  • End to end with the CLI, using an S3 backend (the optional RustFS service) as the org's default:
    • attestation init, then attestation add of a new 5 MB artifact: the CAS staged it, verified its digest and uploaded it to the backend.
    • attestation push: the control plane uploaded the signed bundle to the CAS over cas:8001. The run is success.
    • artifact download returned the 5 MB file with a matching SHA-256.

AI disclosure

Investigation and changes were done with Claude Code (Assisted-by: Claude Code on all commits). I reviewed and tested the changes myself.

🤖 Generated with Claude Code

The CAS in devel/compose.labs.yml crashes on startup because of two
regressions:

- CAS_PUBLIC_KEY_PATH points at cas.pem, which is the private key.
  Since chainloop-dev#3225 (v1.105.0) the key is parsed at startup, so the CAS panics
  with "failed to parse public key: x509: malformed tbs certificate".
- chainloop-dev#3365 (v1.110.0) made staging_dir required, but the scratch image has
  no directory that user 1001 can write to, so the CAS panics before it
  reaches the key.

Point the key path at cas.pub. Ship an empty /tmp-staging-fs owned by
1001 in the image, which is the path the Helm chart already mounts its
emptyDir on, and back it with a named volume in the lab compose.

Assisted-by: Claude Code
Signed-off-by: Daniel Kmak <4950658+e00dan@users.noreply.github.com>
@chainloop-platform

chainloop-platform Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

PR validation — ⚠️ 1 failing

Status Policy Material Messages
⚠️ Failed pr-min-approvals pr-info PR/MR #3797 has 0 approving reviews, 1 required.
✅ Passed pr-description-required pr-info -
✅ Passed pr-user-story-linked pr-info -

View attestation ↗

AI Session Checks — ⏭️ bypassed by label

AI Coding Session Check Bypassed

This PR carries the skip-ai-session label, so the AI coding session check was bypassed.

Learn more about Chainloop Trace.

Security Checks — ✅ 6 passing

✅ Nothing this change touches re-opens a recorded security fix.

View in Chainloop ↗ · How this works ↗
✅ Nothing this change touches re-opens a recorded security fix.

View in Chainloop ↗ · How this works ↗

✅ secret-scan

Status Policy Messages
✅ Passed secrets-detection -

✅ sast-scan

Status Policy Messages
✅ Passed owasp-top10-2025 -
✅ Passed cwe-top25 -
✅ Passed sast -
✅ Passed cwe-top26-40-cusp -

✅ iac-scan

Status Policy Messages
✅ Passed iac-misconfiguration -
Scans not applied (2)
Scan Reason
vulnerability-scan no manifest/lockfile changed
github-actions-scan no workflow files changed

View attestation ↗


Powered by Chainloop and Chainloop Trace

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 2 files

View guided diff | Re-trigger cubic

With a non-inline CAS backend, attestation push against the lab stack
fails with an internal error. The control plane uploads the attestation
bundle to the CAS at cas_server.grpc.addr, which config.devel.yaml sets
to 0.0.0.0:9001. Inside the control-plane container that is the
container itself, so the dial times out. The value has no env
placeholder, so compose could not override it.

Add a CAS_SERVER_ADDR placeholder that defaults to the current value,
so running the control plane on the host is unchanged, and set
CP_CAS_SERVER_ADDR to cas:8001 in the lab compose. Like the Helm chart,
this targets the CAS shared endpoint rather than the deprecated
gRPC-only listener.

Assisted-by: Claude Code
Signed-off-by: Daniel Kmak <4950658+e00dan@users.noreply.github.com>
@e00dan e00dan changed the title fix(cas): make the lab compose CAS start again fix(devel): make the lab compose stack work end to end. Nov 8, 2026
@jiparis
jiparis requested a review from a team November 9, 2026 11:40
@jiparis

jiparis commented Nov 9, 2026

Copy link
Copy Markdown
Member

Thanks for tracking this down. I reproduced all three problems against artifact-cas:latest.

One change request: please drop the Dockerfile.goreleaser change and fix the staging dir in compose only.

The production image doesn't need it: Helm mounts an emptyDir at /tmp-staging-fs, so released images already work. The image change is only there so the cas-staging named volume picks up 1001 ownership, which means changing the production image for a dev-only setup. It also means the lab stack stays broken until the next release.

A tmpfs mount with the right owner works with today's :latest image and needs no image change:

  cas:
    tmpfs:
      - /tmp-staging-fs:uid=1001,gid=1001
    environment:
      CAS_VAULT_ADDRESS: http://vault:8200
      CAS_PUBLIC_KEY_PATH: /devkeys/cas.pub
      CAS_STAGING_DIR: /tmp-staging-fs

With that, the cas-staging volume mount and the top-level volumes: block can go. I tested it with artifact-cas:latest: the CAS loads cas.pub and listens on :8001 and :9001. tmpfs is RAM-backed, but that's fine for a lab stack, and it matches Helm's per-pod emptyDir more closely. Add size=... if you want a cap.

The cas.pub fix and the CP_CAS_SERVER_ADDR / ${CAS_SERVER_ADDR:0.0.0.0:9001} change look good as they are.

Keep the production image unchanged, as requested in review. Revert
the Dockerfile change and mount a tmpfs owned by 1001 at
/tmp-staging-fs in the lab compose, replacing the named volume. This
works with the released artifact-cas:latest image, so the lab stack
works as soon as this lands rather than after the next release.

Assisted-by: Claude Code
Signed-off-by: Daniel Kmak <4950658+e00dan@users.noreply.github.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 2 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

View guided diff | Re-trigger cubic

Comment thread devel/compose.labs.yml Resolved
@e00dan e00dan changed the title fix(devel): make the lab compose stack work end to end. fix(devel): make the lab compose stack work end to end Nov 9, 2026
@e00dan

e00dan commented Nov 9, 2026

Copy link
Copy Markdown
Author

@jiparis

Feedback addressed. Please re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants