Skip to content

[BUGFIX] Prometheus: throw when a query is answered with an error body #899

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Gladorme merged 3 commits into from
Oct 30, 2026
Merged

[BUGFIX] Prometheus: throw when a query is answered with an error body #899

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d3424a3
[BUGFIX] Prometheus: throw when a query is answered with an error body
AnKiT-GaRG2 Oct 27, 2026
9ab35f8
[IGNORE] Prometheus: handle an error body without error text
AnKiT-GaRG2 Oct 28, 2026
6add51d
Merge branch 'main' into bugfix/prometheus-error-body
AnKiT-GaRG2 Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion prometheus/src/plugins/prometheus-time-series-query/get-time-series-data.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,12 @@ export const getTimeSeriesData: TimeSeriesQueryPlugin<PrometheusTimeSeriesQueryS
}
}

// TODO: What about error responses from Prom that have a response body?
// Non-2xx responses are already turned into an error by fetchJson. Some backends and proxies

@Gladorme Gladorme Oct 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which backend / proxy do that by curiosity?
To me it's an issue at the backend / proxy that must be fixed too.

@AnKiT-GaRG2 AnKiT-GaRG2 Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point. I haven't traced this to a specific backend or proxy. I wrote the guard defensively, because the old code silently returned an empty chart whenever the body had status: "error", whatever the status code. I've reworded the comment as you suggested to say these backends don't follow the Prometheus spec. If you'd rather not work around them in the plugin, I can drop the guard, but it's cheap and gives users a visible error instead of an empty panel.

@AnKiT-GaRG2 AnKiT-GaRG2 Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No specific one in mind, this is defensive. I agree it is a bug on the backend / proxy side and that it should be fixed there too. This only makes sure such a response shows an error in the panel instead of a silent empty chart. I reworded the comment to say the backend is non compliant.

// are non compliant with Prometheus spec and answer an invalid query with a 2xx status and an error body, which must not render as an empty chart.
if (response.status === 'error') {
throw new Error(response.error || `Prometheus query failed (${response.errorType || 'unknown error'})`);
}

const result = response.data;

// Custom display for response header warnings, configurable error responses display coming next
Expand Down
35 changes: 35 additions & 0 deletions prometheus/src/plugins/prometheus-time-series-query/plugin.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,41 @@ describe('PrometheusTimeSeriesQuery', () => {
expect(params.time).toBe(Math.floor(ctx.timeRange.end.getTime() / 1000));
});

it('should throw the error of a range query answered with an error body', async () => {
(promStubClient.rangeQuery as Mock).mockResolvedValueOnce({
status: 'error',
errorType: 'bad_data',
error: 'parse error: unexpected end of input',
} satisfies RangeQueryResponse);

await expect(PrometheusTimeSeriesQuery.getTimeSeriesData({ query: 'sum(' }, createStubContext())).rejects.toThrow(
'parse error: unexpected end of input',
);
});

it('should throw a fallback message when an error body has no error text', async () => {
(promStubClient.rangeQuery as Mock).mockResolvedValueOnce({
status: 'error',
errorType: 'bad_data',
} as unknown as RangeQueryResponse);

await expect(PrometheusTimeSeriesQuery.getTimeSeriesData({ query: 'sum(' }, createStubContext())).rejects.toThrow(
'Prometheus query failed (bad_data)',
);
});

it('should throw the error of an instant query answered with an error body', async () => {
(promStubClient.instantQuery as Mock).mockResolvedValueOnce({
status: 'error',
errorType: 'timeout',
error: 'query timed out',
} satisfies InstantQueryResponse);

await expect(
PrometheusTimeSeriesQuery.getTimeSeriesData({ query: 'up', instant: true }, createStubContext()),
).rejects.toThrow('query timed out');
});

it('should use instantQuery when spec.instant is true', async () => {
const ctx = createStubContext();
(promStubClient.instantQuery as Mock).mockClear();
Expand Down
Loading