Skip to content

Harden import, validation, storage, outbound fetches, and deploy manifests (SC-06/07/13/14/15/16/19/22/23/24) - #82

Merged
aaronbrethorst merged 11 commits into
mainfrom
harden-import-validation
Nov 21, 2026
Merged

aaronbrethorst merged 11 commits into
mainfrom
harden-import-validation

Conversation

@aaronbrethorst

@aaronbrethorst aaronbrethorst commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Hardens the import path, content validation, storage permissions, outbound-fetch bounds, and deploy manifests per the pre-launch security audit. One commit per finding, each test-first.

Findings

ID Fix
SC-13 Document.Validate refuses alert, study, survey, and question ids, plus a survey response's survey_id, at or above the sequence floor. The floor is export.DefaultIDFloor = 1,000,000, which is the runbook's sequence bump --min and OBACloud's Sidecar::Cutover::ID_HEADROOM. A deployment that raised it passes import --id-floor N. Translations, responses, push registrations, and ghost bus reports are keyed by language, public id, or token, so no carried id applies to them. Alarms and Live Activities are not in the document. Before this, an alert id of MaxInt64 bricked alert creation in every region.
SC-07 Added alerts.ValidateURL, which accepts an empty url or an absolute http(s) URL with a host, at most 2048 bytes. Admin create and patch (400), CLI alert create and alert edit, and import all use it. An invalid legacy url fails import with an error naming the alert.
SC-15 Imported rider rows now meet the rider API's limits: 255-byte identifiers, 4096-byte tokens, a 255-byte test-device description, 1000-rune comments, lat ±90 and lon ±180, wait-duration choices, URL-safe public ids of 1–64 characters, and a captured snapshot that is a JSON object of at most 64 KiB. Each failure is a rejection that names the row. The one normalization: a non-test device's description is dropped, as the registration API does. The caps moved into the domain packages (pushreg, ghostbus, surveys), and a test pins httpapi's copies to them. The CLI reads at most 256 MiB.
SC-14 An id held by a sidecar-authored row is now an ErrImportConflict. Semantics are below.
SC-16 sqlite.Open creates a missing database at 0600 (SQLite gives -wal/-shm the database's mode) and narrows existing files. entrypoint.sh sets umask 077 before Litestream restores or replicates. Checked with the built image against an S3 stub: replication, wipe, restore all work, and every file is 0600 with .sidecar.db-litestream at 0700.
SC-22 A regions-directory entry whose obaBaseUrl, or non-empty sidecarBaseUrl, is not an absolute http(s) URL with a host is skipped like any other invalid entry. Plain http and loopback or private hosts stay allowed on purpose: local dev and the OBACloud contract harness use http://127.0.0.1, and the directory URL is chosen by the operator. The OBA client's transport caps every response body at 32 MiB (httpx.CapResponseBodies), and ErrResponseTooLarge survives the key-redacting error path.
SC-23 csvsafe.Cell skips leading whitespace, control, and format characters (NBSP, BOM, ZWSP) and matches = + - @ plus their full-width and small-form variants.
SC-24 golang.org/x/crypto v0.55.0 → v0.59.0. go mod tidy also bumps x/sync, x/sys, x/term, and x/text, and writes the go directive as 1.26.0. Added make vulncheck (govulncheck pinned at v1.9.0) and .github/workflows/govulncheck.yml, which runs weekly and on go.mod changes to main, with SHA-pinned actions. With Go 1.26.11: "No vulnerabilities found".
SC-06 Confirmed from gorush v1.22.0 source: log.hide_token defaults to true (config/config.go), and both the inline logs and the feedback webhook payload are built from the masked log entry (logx.GetLogPushEntry → notify.SendNotification → DispatchFeedback). render.yaml, render.staging.yaml, and compose.yaml now set GORUSH_LOG_HIDE_TOKEN=false (viper prefix GORUSH, .→_). A token containing * counts as masked: the webhook and SendBatch each log one ERROR naming the setting, and the webhook skips the delete. Downside: gorush's own logs now carry full tokens.
SC-19 Every FROM is now tag@sha256 (multi-arch index digests; golang:1.26-alpine is Go 1.26.11). gorush is pinned by digest, using the digest-only form for Render. The Litestream tarball is checked with sha256sum -c against per-arch digests from the v0.5.16 checksums.txt, cross-checked against downloads. deploy/manifests_test.go is the CI lint (make test). It skips only our own ghcr.io/onebusaway/sidecar:main, because that tag is how deploys happen. Built and verified on arm64; the litestream stage was also built for amd64.

Import semantics (SC-14)

Migration 00022 adds imported_at to alerts, studies, and surveys, set to the instant an import first wrote the row. When an import finds a row in this region (and under the same parent) already holding an exported id:

  1. imported_at set → same row → skipped, whatever its content now. An identical re-import does nothing. An edit made in OBACloud after the bulk run is not carried. This is the delta's existing insert-if-absent behavior, and OBACloud's cutover preflight (Sidecar::ContentComparison#import_conflicts) already reports that drift before any flip. Translations added since still land.
  2. imported_at NULL but created_at equals the export's → a row imported by a binary older than 00022 → skipped, and imported_at is stamped. OBACloud never changes created_at, and import carries it verbatim, so a bulk import on the old binary followed by a delta on this one keeps working with no backfill guess.
  3. Otherwise → authored on this sidecar → ErrImportConflict. The transaction rolls back, so nothing from the document is written and no translations, surveys, or questions get attached to the unrelated row.

Questions need no column. A question is only reached under a survey that already passed this check, and its survey_id ownership is compared as before. Rows keyed by public id or token (responses, ghost bus reports, push registrations) keep their existing checks: their keys are random, and a device re-registering directly is meant to be skipped.

Validation

  • make check green.
  • OBACloud contract suite (SIDECAR_CONTRACT=1 SIDECAR_SRC=<this branch> bundle exec rspec spec/contract, obacloud sidecar-migration @ 0544ae06): 15 examples, 0 failures, 1 pending. The full-cutover example skips on this branch because main has no alerts.pb alias yet (Serve alerts.pb, weather.json and vehicles.json (production cutover blocker) #80). Against a scratch merge of this branch + Serve alerts.pb, weather.json and vehicles.json (production cutover blocker) #80 it runs and passes: preflights, the real delta sidecar-admin import, and step 5. With that merge, only the "lacks the alias" example skips.
  • Real export documents from a live local cutover (regions 9001/9002/9003, bulk + deltas), imported into a scratch DB. Each was run --dry-run then for real, and re-runs were no-ops (0 added).
  • A read-only snapshot of a database imported by the old binary (migration 17) was upgraded to 00022. Re-running the deltas recognized the legacy rows by created_at and stamped them, and left the sidecar-authored alert 1000003 unmarked. An export claiming id 1000003 was refused, first by the floor, then (with --id-floor 2000000) as a conflict, with no translations attached.

Summary by CodeRabbit

  • New Features
    • Added an option to set an ID floor when importing data.
    • Imports now provide clearer validation for oversized documents, invalid URLs, and out-of-range or oversized values.
  • Bug Fixes
    • Alert creation and editing now reject invalid or excessively long URLs.
    • CSV exports better guard against formula triggers following invisible characters and whitespace.
    • Re-imports better distinguish previously imported records from conflicting records.
  • Security & Reliability
    • Database files receive owner-only permissions; external response sizes are limited.
    • Added scheduled vulnerability checks and verification for third-party image and download integrity.

An export could carry any positive id, and an alert id of MaxInt64 parked
the shared alerts sequence where SQLite has no id left to mint, breaking
alert creation in every region. Document.Validate now holds alert, study,
survey, and question ids (and a survey response's survey_id) below the
floor the cutover runbook bumps the sequences to (1,000,000, OBACloud's
ID_HEADROOM); `import --id-floor` matches a raised floor.
The alert url was stored and published verbatim, so javascript:, data:,
protocol-relative, and relative values reached the feed and the link
hrefs that render it. alerts.ValidateURL (empty, or absolute http(s) with
a host, at most 2048 bytes) now guards admin create and patch (400), CLI
alert create and edit, and import, where an invalid legacy url fails with
an error naming the alert to fix in OBACloud.
Import re-validated enum names, question content, answers, and windows,
but not the HTTP layer's limits, and the CLI read the document unbounded.
Imported survey responses, push registrations, and ghost bus reports now
meet the same caps the rider API applies (identifier and token lengths,
coordinate ranges, wait-duration choices, comment length, a URL-safe
public id, a captured snapshot that is a bounded JSON object), each
failure naming the row; a non-test device's description is normalized
away as the registration API does. The caps move into the domain
packages, with a test pinning the HTTP layer's copies to them, and the
CLI reads at most 256 MiB.
Import treated any existing row at an exported id in the same region as
the row being imported, so a locally authored alert at a colliding id was
skipped and the export's translations were attached to it. Alerts,
studies, and surveys now record imported_at (migration 00022), and a skip
requires provenance: a row an earlier import wrote is skipped whatever
its content (identical re-imports are no-ops; OBACloud-side edits after
the bulk run keep the documented insert-if-absent behavior), a row
imported before the column existed is recognized by its verbatim
created_at and stamped, and anything else is ErrImportConflict with the
whole document rolled back.
The SQLite file and its -wal/-shm siblings were created 0644 under the
default umask though they hold password hashes, key digests, OBA keys,
push tokens, and rider data. Open now creates a missing database at 0600
(SQLite gives -wal and -shm the database's own mode) and narrows existing
files, and the container entrypoint sets umask 077 before Litestream
restores or replicates, covering its page copies too. Litestream runs as
the same user, so replication is unaffected.
The regions directory's obaBaseUrl and sidecarBaseUrl were stored without
a scheme or host check, and OBA response bodies were read unbounded. A
directory entry whose obaBaseUrl (or non-empty sidecarBaseUrl) is not an
absolute http(s) URL with a host is now skipped like any other invalid
entry -- plain http and loopback/private hosts stay allowed for local
development and the OBACloud contract harness -- and the OBA client's
transport caps every response body at 32 MiB (httpx.CapResponseBodies),
surfacing ErrResponseTooLarge through the key-redacting error path.
csvsafe.Cell checked only the first byte, so "\n=1+1", " =1+1", a
no-break space before '=', and a full-width = all passed through, though
spreadsheets trim or skip those before deciding a cell is a formula. The
guard now skips leading whitespace, control, and format characters and
matches the ASCII triggers plus their full-width and small-form variants.
golang.org/x/crypto v0.55.0 carried GO-2026-6355/6354/5932 (unreachable
from this code). Bump to v0.59.0 (go mod tidy also lifts x/sync, x/sys,
x/term, x/text and writes the go directive as 1.26.0). Add `make
vulncheck` with a pinned govulncheck and a scheduled workflow that runs
it weekly and on go.mod changes to main, SHA-pinning its actions like the
existing workflows. With Go 1.26.11 it reports no vulnerabilities; with an
older local 1.26 patch it flags the standard library, which is the point.
gorush 1.22 defaults log.hide_token to true (config.go setDefaults) and
builds both its inline sync-mode logs and its feedback webhook payloads
from the masked log entries (logx.GetLogPushEntry, notify.SendNotification),
so the sidecar's exact token match never pruned anything. render.yaml,
render.staging.yaml, and compose.yaml now set GORUSH_LOG_HIDE_TOKEN=false
(viper env prefix GORUSH, '.' -> '_'), pinned by a manifest test. A token
containing '*' is treated as masked: the feedback webhook and SendBatch
each log one ERROR naming the setting, and the webhook skips the delete.
…nload

The Dockerfile's base images and the gorush service in render.yaml,
render.staging.yaml, and compose.yaml used mutable tags, and the build
piped the Litestream tarball from GitHub straight into tar. Every FROM is
now tag@sha256 (multi-arch index digests resolved with docker buildx
imagetools inspect; golang:1.26-alpine is Go 1.26.11), gorush is pinned by
digest (digest-only form for Render), and the Litestream tarball is
checked against the per-architecture SHA-256 from the v0.5.16 release's
checksums.txt (cross-checked against the downloaded files) before it is
unpacked. deploy/manifests_test.go is the CI lint: it fails on an
unpinned FROM or third-party image or an unverified download. The repo's
own ghcr.io/onebusaway/sidecar:main stays a tag; it is the deploy path.
@coderabbitai

coderabbitai Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Explain Change →

Caution

Review failed

The pull request is closed.

Walkthrough

This pull request adds import validation and provenance handling, alert and region URL checks, OBA response limits, CSV formula-trigger protection, masked-token handling, owner-only file permissions, image and download verification, and scheduled vulnerability scanning.

Changes: Import validation and provenance

Layer / File(s) Summary
Import data validation
internal/export/export.go, internal/ghostbus/ghostbus.go, internal/pushreg/pushreg.go, internal/surveys/answers.go, internal/store/sqlite/import_rows.go, internal/httpapi/import_limits_test.go, internal/export/export_test.go, internal/store/sqlite/import_test.go
Import validation adds a configurable ID floor, public-ID patterns, and rider-row limits for identifiers, tokens, snapshots, coordinates, comments, and wait durations. Tests cover these constraints and boundary values.
Import input and CLI options
cmd/sidecar-admin/import.go, cmd/sidecar-admin/import_test.go, README.md
The CLI caps JSON input at 256 MiB, rejects unknown fields and trailing content, and accepts a positive --id-floor option.
SQLite import writes and provenance
internal/store/sqlite/import.go, internal/store/sqlite/import_provenance_test.go, internal/store/sqlite/migrations/00022_import_provenance.sql, internal/store/sqlite/queries/import.sql, internal/store/sqlite/import_test.go, README.md
SQLite records import timestamps for alerts, studies, and surveys. Existing rows are classified as imported, legacy imported, or conflicting. Import tests cover re-imports, conflicts, and legacy rows.

Alert URL validation

Layer / File(s) Summary
URL rules and validation
internal/alerts/url_test.go, specification/specification.md, README.md
Tests and documentation specify the accepted alert URL forms and the 2048-byte limit.
CLI, API, and import enforcement
cmd/sidecar-admin/commands.go, cmd/sidecar-admin/alert_url_test.go, internal/httpapi/admin_alerts.go, internal/httpapi/admin_alerts_url_test.go, internal/store/sqlite/import.go, internal/store/sqlite/import_test.go
CLI and admin API create and patch operations reject invalid URLs before writing. Import validation also rejects invalid alert URLs.

Directory URLs and OBA response limits

Layer / File(s) Summary
Region directory URL checks
internal/regions/directory.go, internal/regions/directory_urls_test.go, README.md, specification/specification.md
Region fetching rejects entries with invalid OBA URLs or invalid non-empty sidecar URLs.
HTTP response body caps
internal/httpx/httpx.go, internal/httpx/httpx_test.go, internal/obaapi/obaapi.go, internal/obaapi/obaapi_test.go
A copied HTTP client can limit response bodies. The OBA client applies a 32 MiB cap and redacts oversized-response errors.

CSV formula-trigger protection

Layer / File(s) Summary
Cell trigger detection and tests
internal/csvsafe/csvsafe.go, internal/csvsafe/csvsafe_test.go, CLAUDE.md, README.md
CSV cell handling scans past leading whitespace, control, and format characters before checking ASCII, full-width, and small-form formula triggers.

Gorush masked-token handling

Layer / File(s) Summary
Masked-token logging and feedback handling
internal/push/gorush.go, internal/push/masked.go, internal/push/masked_test.go, internal/httpapi/feedback.go, internal/httpapi/feedback_masked_test.go, cmd/sidecar/main.go
The push sender and feedback handler log a masked-token warning once. The feedback handler returns HTTP 200 without pruning by the masked token.
Gorush token logging configuration
compose.yaml, render.yaml, render.staging.yaml, deploy/manifests_test.go, README.md
Compose and Render set GORUSH_LOG_HIDE_TOKEN to "false". Documentation describes token logging and the masked-token behavior.

File permissions and image integrity

Layer / File(s) Summary
Owner-only file permissions
internal/store/sqlite/store.go, internal/store/sqlite/filemode_test.go, deploy/entrypoint.sh, deploy/manifests_test.go, .env.example, README.md
SQLite database, WAL, and SHM files are set to mode 0600. The deployment entrypoint sets umask 077.
Pinned images and verified downloads
Dockerfile, compose.yaml, render.yaml, render.staging.yaml, deploy/manifests_test.go, README.md
Third-party images are pinned by digest. Litestream downloads are checked against architecture-specific SHA-256 values before extraction.

Scheduled vulnerability scanning

Layer / File(s) Summary
Pinned scan command and dependencies
Makefile, go.mod, CLAUDE.md
The Makefile adds a pinned govulncheck target. The Go directive and listed dependencies are updated.
Scheduled workflow
.github/workflows/govulncheck.yml
A GitHub Actions workflow runs the scan on a schedule, relevant dependency-file pushes, and manual dispatch.

Estimated code review effort: 4 (Complex) | ~60 minutes

Change: Bug fix

Possibly related PRs

  • OneBusAway/sidecar#80: PR #82’s full-cutover contract check depends on the alerts.pb route added by this PR.

Suggested reviewers: nym01

Merge Risk: Moderate · up to 4f51a

This PR hardens imports, URL validation, response sizes and file permissions. Imported surveys can still gain locally authored questions that a later import may mistake for source questions. A directory passed as the database path would also have its mode changed. Both are narrow cases, but the question-provenance gap should be fixed or consciously accepted before merge.

- [ ] Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@aaronbrethorst

Copy link
Copy Markdown
Member Author

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

@aaronbrethorst

Copy link
Copy Markdown
Member Author

Maintainer sweep: this is ready on the merits, but I'm not merging it in this pass because it's authored by the account running the sweep, and I don't approve or merge my own PRs. It needs an approval and merge from another maintainer.

What I checked myself:

  • ownerOnly creates a missing database at 0600 using O_EXCL, narrows an existing database and its -wal/-shm files, and leaves :memory: and file: URIs alone. A missing directory still surfaces through SQLite as it did before.
  • CapResponseBodies wraps a copy of the client, so NoRedirectClient keeps its no-redirect behavior.
  • The new queries respect the CLAUDE.md sqlc rules: ASCII-only comments, no mixing sqlc.arg with ?, and no parameters inside IN or ON CONFLICT ... WHERE. imported_at is INTEGER epoch seconds.
  • compose.yaml, render.yaml and render.staging.yaml agree on the gorush digest and GORUSH_LOG_HIDE_TOKEN=false.
  • CI is green.

Non-blocking, worth knowing before rollout:

  • Open now chmods an existing database. A sidecar-admin run as a different, non-owning user will now fail to open the file where it used to work. The README and USER sidecar cover this, but it is a behavior change for anyone running the CLI on the host.
  • Rows that an older binary imported without created_at were stamped with the import time. On the next delta import they will report as conflicts. That's fine as long as OBACloud always exports created_at.

aaronbrethorst added a commit that referenced this pull request Nov 14, 2026
The alerts feed published whatever url was stored, so a javascript: URL
written straight into the database went out in alerts.pb and
alerts.pbtext. BuildFeed now emits url only when alerts.ValidateURL
accepts it (empty, or an absolute http/https URL with a host, at most
2048 bytes); otherwise the field is omitted and the alert still ships.

internal/alerts/url.go is a byte-identical copy of the validator from
PR #82 (harden-import-validation), which wires it into the admin API,
the CLI and import. Identical additions on both branches merge cleanly
in either order.
@aaronbrethorst

Copy link
Copy Markdown
Member Author

PR sweep disposition: blocked — not merged.

I authored this PR, so the sweep can't approve or merge it. The code review above covers the current head baab958 and found no blocking issues. CI is green and the branch merges cleanly with main. Note that CodeRabbit skipped this PR, so it hasn't had a second automated pass.

To unblock: another maintainer reviews and approves, then merges it.

@sonarqubecloud

Copy link
Copy Markdown

@aaronbrethorst
aaronbrethorst merged commit cca92a8 into main Nov 21, 2026
10 of 11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant