Analysis by claude 🤖
Summary
api_key_auth (crates/pipeline-manager/src/auth.rs, currently lines ~277-311) collapses every DBError into the same 401 Unauthorized API key response. The match on the error distinguishes DBError::InvalidApiKey from other variants, but only to pick a log line — the client-facing response is identical. When the key lookup fails because postgres is unreachable, callers with perfectly valid keys get told their key is invalid.
Real-world cost
The cluster autoscaler evicted feldera-db-0 mid-run. Every request in merge-queue run 30568988517 (PR #6758) then failed with:
Error Code: AuthenticationFailed
Message: Unauthorized API key
Response Status: 401
The investigation initially chased key rotation/expiry because the response asserts the key is bad. The actual cause (db pod eviction, visible as api-server readiness 500s at the same timestamps) only surfaced through cluster events. Anyone hitting a db blip will misdiagnose the same way.
Expected behavior
DBError::InvalidApiKey -> 401, as today.- Any other lookup error (connection refused, timeout, pool exhausted) -> 503 with a message that says the server could not validate credentials, not that the credentials are wrong. Fail-closed stays intact; only the status/message stops lying.
Notes
The /healthz-style probes already reported 500 during the outage, so load balancers saw the truth; only the auth path rewrites the error.
Analysis by claude 🤖
Summary
api_key_auth(crates/pipeline-manager/src/auth.rs, currently lines ~277-311) collapses everyDBErrorinto the same 401Unauthorized API keyresponse. The match on the error distinguishesDBError::InvalidApiKeyfrom other variants, but only to pick a log line — the client-facing response is identical. When the key lookup fails because postgres is unreachable, callers with perfectly valid keys get told their key is invalid.Real-world cost
The cluster autoscaler evicted
feldera-db-0mid-run. Every request in merge-queue run 30568988517 (PR #6758) then failed with:The investigation initially chased key rotation/expiry because the response asserts the key is bad. The actual cause (db pod eviction, visible as api-server readiness 500s at the same timestamps) only surfaced through cluster events. Anyone hitting a db blip will misdiagnose the same way.
Expected behavior
DBError::InvalidApiKey-> 401, as today.Notes
The
/healthz-style probes already reported 500 during the outage, so load balancers saw the truth; only the auth path rewrites the error.