feat(databases): expose safe hints for masked encrypted_extra fields by gabotorresruiz · Pull Request #45996 · apache/superset

gabotorresruiz · 2026-11-04T03:31:01Z

SUMMARY

Database editors cannot tell which secret is stored in encrypted_extra once it is masked. Engine specs already declare which paths are secrets (encrypted_extra_sensitive_fields) and the modal masks them as XXXXXXXXXX, so after a credential rotation an editor has no safe way to confirm which value is live without pasting the secret again. A short tail (for a JWT, part of its signature) identifies a value without exposing it.

This PR adds an opt-in endpoint, GET /api/v1/database/<pk>/encrypted_extra_hints/, that returns, for each declared sensitive field, its label, path and last few characters. The value itself never leaves the server:

A helper next to the existing masking utilities computes the hints from the declared JSONPath expressions, trims whitespace, skips non-strings, expands wildcards per match, and withholds the suffix when the value is shorter than four times the configured length, so never more than 25% of a value is revealed, and reports nothing else about such a value, not even its length. PEM-armoured values (service-account and key-pair private keys) take their ending from the key body, without the -----END ... ----- footer, the line breaks or the base64 padding, so two different keys no longer share the fixed footer as their ending and the revealed characters are real base64 characters.
BaseEngineSpec.encrypted_extra_hints adds the engine's human labels. A command and the endpoint expose it; the response is serialised through a Marshmallow schema so only path, label and suffix can be returned. The number of characters comes from config only; request parameters are ignored.
BaseEngineSpec.get_public_information() gains encrypted_extra_sensitive_fields, a map from each declared JSONPath to its label, served in engine_information on the database GET, the list and available. On GET /api/v1/database/<pk>/connection, the write-gated payload the edit form loads, that map is replaced by the fields actually present as strings in the stored encrypted_extra (Database.stored_sensitive_fields()), keyed by concrete path and labelled by the declaration or, for wildcard matches, by the key name. Readers of the plain database object learn nothing about what is stored; they only see the engine's declared patterns. The paths every engine adds get readable default labels unless the engine declares its own: "OAuth2 client secret" for the OAuth2 client secret, and "Embedded guest password", "Embedded guest private key (PEM)" and "Embedded guest private key passphrase" for the embedded guest credential fields.

Compatibility:

New config ENCRYPTED_EXTRA_HINT_CHARS (int, default 0 = disabled), exposed to the frontend through FRONTEND_CONF_KEYS and documented in the configuration docs and UPDATING.md. Nothing changes for existing deployments unless it is set above 0.
New route mapped to the existing can_write on Database through MODEL_API_RW_METHOD_PERMISSION_MAP; no new permission, no migration. Hints are revealed to any role holding can_write on Database that passes the database filter, the same population that can already call /connection. Every authorised call is event-logged with the action, user and database id, without the request parameters.
Works for every engine that declares encrypted_extra_sensitive_fields (Druid, Presto, Trino, BigQuery, Snowflake and others). Engines that declare nothing inherit the base $.* declaration, so every top-level string value of their encrypted_extra gets a hint labelled by its key. Non-string values (objects, numbers) are masked but never hinted, so an engine whose secret is an object, like YDB's credentials, gets no hint for it.
engine_information carries one additional key on every engine; existing keys are unchanged. No secret values are included, only declared paths and labels.
Hints apply to every declared sensitive string, passwords included (Druid connect_args.password, Presto and Trino auth_params.password, Snowflake privatekey_pass), subject to the same cap: nothing for values under four times the configured length, never more than 25% otherwise. An operator enabling the key accepts that database editors, who can already read the masked payload and change the connection, see those endings.

Review focus: the 25% cap in the helper, the permission mapping, and the schema that bounds the response to three keys. The companion frontend PR, #45997, adds the UI.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Not applicable, backend only. The companion frontend PR has the screenshots.

TESTING INSTRUCTIONS

Automated:

tests/unit_tests/utils/json_tests.py and tests/unit_tests/db_engine_specs/test_{base,druid,presto,trino}.py: paths, cap, boundary at 4x, no length for a withheld value, blank and whitespace, unicode, mask literal, non-strings, malformed JSON, a 1 MB payload under a second; PEM keys with real and escaped newlines, padding never revealed and ignored by the threshold, rotation changes the ending, short body, dashes inside plain text; stored-field resolution (dotted, wildcard keeps strings only, nested, malformed); engine information default, set and dict declarations, OAuth2 label precedence.
tests/unit_tests/models/core_test.py: stored_sensitive_fields() on a catch-all engine and on Druid, missing and malformed encrypted_extra; engine_information keeps the declared patterns.
tests/unit_tests/databases/api_test.py and test_schemas.py: hints, exact keys, disabled, malformed, 404 for unknown and invalid ids, 403 without write, 401 without login, query params ignored and kept out of the audit row, string config value, available includes the map, /connection returns the stored fields while the plain GET and the list keep the declared patterns.
tests/integration_tests/databases/api_tests.py: admin 200, gamma 403, event log row without secret text, available expectations.

Manual:

Set ENCRYPTED_EXTRA_HINT_CHARS = 5.
Store {"connect_args": {"scheme": "https", "jwt": "<token>"}} in a Druid connection's Secure extra.
Call GET /api/v1/database/<pk>/encrypted_extra_hints/ as an editor: the response holds the label, path and last five characters, never the token or its length.
Call it as a read-only user: 403. Set the config to 0: [].

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

netlify · 2026-11-04T03:33:13Z

✅ Deploy Preview for superset-docs-preview ready!

Name	Link
🔨 Latest commit	`f946e81`
🔍 Latest deploy log	https://app.netlify.com/projects/superset-docs-preview/deploys/6af25d819dff18aa8496ba80
😎 Deploy Preview	https://deploy-preview-45996--superset-docs-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.
🤖 Make changes	Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

codecov · 2026-11-04T04:04:21Z

Codecov Report

❌ Patch coverage is 78.72340% with 20 lines in your changes missing coverage. Please review.
✅ Project coverage is 65.36%. Comparing base (a131fcb) to head (d855c16).

Files with missing lines	Patch %	Lines
superset/utils/json.py	75.00%	3 Missing and 4 partials ⚠️
superset/db_engine_specs/base.py	64.70%	4 Missing and 2 partials ⚠️
superset/databases/api.py	78.57%	2 Missing and 1 partial ⚠️
...uperset/commands/database/encrypted_extra_hints.py	89.47%	1 Missing and 1 partial ⚠️
superset/models/core.py	75.00%	2 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #45996   +/-   ##
=======================================
  Coverage   65.35%   65.36%           
=======================================
  Files        3083     3084    +1     
  Lines      211224   211317   +93     
  Branches    49803    49814   +11     
=======================================
+ Hits       138054   138127   +73     
- Misses      69992    70004   +12     
- Partials     3178     3186    +8

Flag	Coverage Δ
hive	`34.06% <43.61%> (+<0.01%)`	⬆️
mysql	`51.83% <78.72%> (+0.02%)`	⬆️
postgres	`51.86% <78.72%> (+0.02%)`	⬆️
presto	`35.70% <43.61%> (+<0.01%)`	⬆️
python	`52.15% <78.72%> (+0.02%)`	⬆️
sqlite	`51.53% <71.27%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codeant-ai-for-open-source · 2026-11-04T04:32:55Z

+    @event_logger.log_this_with_context(
+        action=lambda self, *args, **kwargs: (
+            f"{self.__class__.__name__}.encrypted_extra_hints"
+        ),
+        log_to_statsd=False,
+    )


Suggestion: The default logs query parameters and the referrer, so a caller can persist an arbitrary secret in audit events despite this endpoint being described as logging only the database id.

Assessment: 🟠 Major · 🔁 Occurrence: Rarely · 🏷️ Security

Prompt for AI Agent 🤖

This is a comment left during a code review. **Path:** superset/databases/api.py **Line:** 405:410 **Comment:** *Security: The default logs query parameters and the referrer, so a caller can persist an arbitrary secret in audit events despite this endpoint being described as logging only the database id. Validate the correctness of the flagged issue. If correct, How can I resolve this? If you propose a fix, implement it and please make it concise. Once fix is implemented, also check other comments on the same PR, and ask user if the user wants to fix the rest of the comments as well. if said yes, then fetch all the comments validate the correctness and implement a minimal fix

👍 | 👎

Good catch, this is the default log_this_with_context behaviour. Switched the endpoint to include_request_data=False, the same opt-out oauth2 uses in this file, so the audit row carries only the action, the user and pk; test_encrypted_extra_hints_ignores_query_params now asserts that.

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of eb9229d.

Set include_request_data=False on the event logger, so it does not include request parameters or referrer data in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of 4e5399c.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of bb57e3a.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of d57884e.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of 5d569fa.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of c210232.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of f946e81.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

✅ CodeAnt verified this suggestion was addressed in subsequent commits and marked this thread resolved as of d855c16.

Set include_request_data=False on the event logger, preventing request parameters and referrer data from being included in the audit event.

_{If that's not right, unresolve this thread and CodeAnt will leave it open.}

codeant-ai-for-open-source · 2026-11-04T04:33:32Z

CodeAnt PR Risk

🟢 Low Risk

The PR appears safe to merge; the audit logger excludes request data, addressing the concern about secrets appearing in audit events.

Assessed commit: f946e81d682f

msyavuz · 2026-11-06T15:21:35Z

+                "suffix": (
+                    revealable[-length:] if len(revealable) >= 4 * length else None
+                ),
+                "length": len(value),


length is returned even when the suffix is withheld, so a short password still leaks its exact length. Should this be null below the 4x threshold, or should password-labelled paths be excluded from hints?

@msyavuz good catch. The frontend stopped showing the count a while ago, so there was no reason to keep returning it. length is gone from the response and the schema; a value below the threshold now yields only its path and null suffix.

msyavuz · 2026-11-06T15:21:35Z

+            {
+                "path": path,
+                "suffix": (
+                    revealable[-length:] if len(revealable) >= 4 * length else None


PEM bodies usually end in =/==, so 1-2 of the revealed chars are padding. Strip trailing = before slicing.

@msyavuz right, the padding is noise. The PEM body is stripped of trailing = before the threshold and the slice are applied, so the revealed characters are real base64 characters. JWT segments are unpadded already, so nothing changes there.

msyavuz · 2026-11-06T15:21:35Z

+        log_to_statsd=False,
+        include_request_data=False,
+    )
+    def encrypted_extra_hints(self, pk: int, **kwargs: Any) -> Response:


**kwargs is never read.

@msyavuz removed.

msyavuz · 2026-11-06T15:21:35Z

+# who can edit a database may reveal as a hint, together with the value's length.
+# Never more than 25% of a value is revealed: shorter values only report their
+# length. 0 disables the hints.
+ENCRYPTED_EXTRA_HINT_CHARS = 0


New config has no docs entry.

@msyavuz added a section to configuring-superset.mdx (what the key does, the default, the 25% cap and the threshold, the PEM and JWT handling, who can call the endpoint) and an entry under Next in UPDATING.md.

Add a helper that reports, for every declared sensitive path of a database's encrypted_extra, the last few characters of the stored string, so a masked secret can be told apart from another one (for example after a token rotation). The number of characters comes from the new ENCRYPTED_EXTRA_HINT_CHARS setting (0 disables, which is the default) and is exposed to the frontend. At most 25% of a trimmed value is ever revealed: a value shorter than four times the configured length gets no suffix. The length of the value is never reported, because the ending is withheld below that threshold precisely so a short secret gives nothing away, and a length would still identify it. Non-string values are skipped, wildcard paths expand per match, and output is sorted by path. BaseEngineSpec.encrypted_extra_hints builds on encrypted_extra_sensitive_field_paths and takes labels from encrypted_extra_sensitive_fields when it is a dict. A value that is a single PEM block (a service account or key-pair private key) always ends with the same "-----END ...-----" footer, so for those the suffix and the 25% cap use the base64 body with line breaks, newline escapes and the trailing "=" padding removed, so the revealed characters are real base64 characters. The setting is documented in Configuring Superset and announced in UPDATING.md.

Add unhappy-path tests: unicode and trailing-newline values, a stored value equal to the mask literal, non-object or malformed JSON, and a 1 MB secret next to thousands of other keys.

…ra fields Add GET /api/v1/database/<pk>/encrypted_extra_hints/, which returns the label, path and short suffix of each sensitive field stored in a database's encrypted_extra, so an editor can tell which secret is stored without it being revealed. The suffix length comes only from the ENCRYPTED_EXTRA_HINT_CHARS config; 0 or less returns an empty list. The route maps to can_write on Database, so only users who can edit a connection can call it. Calls go to the event log with the database id and never the hint values, and the response is dumped through the schema so nothing beyond the three declared keys can leave the server. The length of a stored value is not returned: it could identify a short secret whose ending is withheld. The published OpenAPI spec gains the new path and its two schemas.

…formation Add the JSONPath and label of sensitive encrypted_extra fields to the engine information returned by the database endpoints. The connection modal needs to know, before any click, which Secure extra fields are stored secrets and what to call them, so it can render a field per stored secret up front and let the reveal control only change the value. The database GET, the list and the available endpoint carry the engine's declared patterns. The connection endpoint, which the edit modal loads and which requires write access to databases, replaces them with the fields stored for that database: every string value at a sensitive path, keyed by its concrete path. An engine on the catch-all "$.*" rule therefore shows one entry per stored string key instead of the pattern, and objects are left out. The label is the declared one when the path is declared exactly, else the key name. A missing or malformed encrypted_extra gives an empty map. Readers of a database see nothing about what is stored. BaseEngineSpec gains encrypted_extra_sensitive_field_labels(), which maps each path to its declared label (or the last path segment) with keys sorted. The secrets every engine adds get readable labels unless the engine declares its own: "OAuth2 client secret" for the OAuth2 client secret, and "Embedded guest password", "Embedded guest private key (PEM)" and "Embedded guest private key passphrase" for the embedded guest credential. encrypted_extra_hints reads its labels from the same method, and a new get_sensitive_field_paths helper shares its path resolution with the hints, so both list the same fields. Database.stored_sensitive_fields() builds the per-database map. The published OpenAPI spec documents the new key.

gabotorresruiz requested review from Vitor-Avila and betodealmeida November 4, 2026 03:31

pull-request-size Bot added the size/XXL label Nov 4, 2026

github-actions Bot added api Related to the REST API doc Namespace | Anything related to documentation labels Nov 4, 2026

gabotorresruiz mentioned this pull request Nov 4, 2026

feat(databases): reveal the ending of stored secret values on demand #45997

Open

2 of 9 tasks

gabotorresruiz force-pushed the feat/encrypted-extra-hints-api branch from cbbb23c to 5ccf5f1 Compare November 4, 2026 03:39

gabotorresruiz marked this pull request as ready for review November 4, 2026 04:23

gabotorresruiz requested review from geido, michael-s-molina, msyavuz, rusackas, sadpandajoe and villebro as code owners November 4, 2026 04:23

codeant-ai-for-open-source Bot reviewed Nov 4, 2026

View reviewed changes

gabotorresruiz force-pushed the feat/encrypted-extra-hints-api branch 2 times, most recently from eb9229d to 4e5399c Compare November 5, 2026 04:35

gabotorresruiz mentioned this pull request Nov 5, 2026

feat(databases): show when a stored JWT expires on reveal #46035

Open

2 of 9 tasks

gabotorresruiz force-pushed the feat/encrypted-extra-hints-api branch 2 times, most recently from bb57e3a to d57884e Compare November 6, 2026 01:32

msyavuz reviewed Nov 6, 2026

View reviewed changes

github-actions Bot added the requires:rebase Requires rebasing on top of current master label Nov 7, 2026

gabotorresruiz force-pushed the feat/encrypted-extra-hints-api branch 2 times, most recently from 5d569fa to c210232 Compare November 9, 2026 18:39

gabotorresruiz requested a review from msyavuz November 9, 2026 19:54

github-actions Bot removed the requires:rebase Requires rebasing on top of current master label Nov 9, 2026

test(databases): cover edge cases for encrypted_extra hints

2214a1f

Add unhappy-path tests: unicode and trailing-newline values, a stored value equal to the mask literal, non-object or malformed JSON, and a 1 MB secret next to thousands of other keys.

gabotorresruiz force-pushed the feat/encrypted-extra-hints-api branch from c210232 to f946e81 Compare November 9, 2026 23:55

gabotorresruiz added 2 commits November 9, 2026 21:31

gabotorresruiz force-pushed the feat/encrypted-extra-hints-api branch from f946e81 to d855c16 Compare November 10, 2026 00:35

Conversation

gabotorresruiz commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

netlify Bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for superset-docs-preview ready!

Uh oh!

codecov Bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

codeant-ai-for-open-source Bot Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 4, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 6, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

codeant-ai-for-open-source Bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodeAnt PR Risk

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gabotorresruiz commented Nov 4, 2026 •

edited

Loading

netlify Bot commented Nov 4, 2026 •

edited

Loading

codecov Bot commented Nov 4, 2026 •

edited

Loading

codeant-ai-for-open-source Bot commented Nov 4, 2026 •

edited

Loading