Skip to content

fix: Egress Gateway should keep retrying to connect to Redis - #390

Open
Rylon wants to merge 1 commit into
LibreChat-AI:mainfrom
Rylon:fix/egress-ledger-reconnect
Open

Rylon wants to merge 1 commit into
LibreChat-AI:mainfrom
Rylon:fix/egress-ledger-reconnect

Conversation

@Rylon

@Rylon Rylon commented Nov 12, 2026 •

Copy link
Copy Markdown

If Redis is unavailable for more than about 12 seconds, the egress gateway never recovers from this.

The ledger client's retryStrategy returns null after 5 attempts 2 seconds apart, after which ioredis ends the connection for good. The /ready probe then fails on every request because pingEgressLedger gets Connection is closed, but the /live probe keeps returning 200. As a result, Kubernetes leaves the pod running in NotReady state indefinitely, and the system can never recover without manual intervention.

We hit this when the node running the bundled Redis was drained. Redis came back on another node within a minute or so, but the gateway stayed NotReady until we deleted the pod manually.

This PR changes the ledger client to use the existing shared helper used by the queue connections (redisReconnectDelay).

The options are moved into an exported egressLedgerRedisOptions() so the retry behaviour can be tested without a slow outage test.

I added a test in egress-ledger-reconnect.test.ts to verify we get suitable integers up to the cap of 2000ms, rather than returning null after the 6th retry.

I ran a local test which isn't included here as it takes ~25 seconds to run, but it tested the behaviour against a readl redis-server, killed the server, saw the pings fail, then restarted it and see pings return. It fails on main with Connection is closed, but works on this branch.

…ith a cap when retrying Redis connections, otherwise we don't recover from outages automatically.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant