tokio-postgres: fix deadlock in query_typed/prepare on unknown OIDs (per-response parking variant — alternative to #1348) - #1349
Open
rubenfiszel wants to merge 1 commit into
Conversation
…a has unknown OIDs (per-response parking variant) This is an alternative to the unbounded-channel fix in rust-postgres#1348. Same root cause, different remediation — see rust-postgres#1348 for the deadlock analysis. Keeps the per-response `mpsc::channel(1)` and instead changes `Connection::poll_read` to keep draining the wire when a sender backs up, parking the unsent frame on a per-response `parked` queue inside `Response` itself. New wire frames are routed to the first response whose `ReadyForQuery` hasn't been observed yet, so a congested streaming query no longer blocks delivery to a concurrent typeinfo sub-query whose response is queued behind it on the same socket. Per-response (rather than a single global parked queue) is the critical detail: a global queue still deadlocks because once response[0]'s sender is full, the global queue accumulates frames for response[1] that we have no way to deliver without first delivering response[0]'s frames. Per-response queues let us poll each sender independently. A new `completion_seen: bool` on `Response` tracks whether the `ReadyForQuery` frame has been *observed* on the wire (even if it's parked, not yet delivered to the consumer). `target_response_idx()` uses this to route subsequent wire frames to the next response, while the parked completion frame is still flushed in-order to its own consumer when its sender unblocks. Trade-off versus rust-postgres#1348: - Pro: preserves the bounded `mpsc::channel(1)` semantics — Each response's parked queue is bounded only by the size of that particular request's result set; well-behaved consumers continue to apply backpressure to the connection task as before. - Con: ~100 lines of additional bookkeeping (per-response parked queues, completion tracking, fan-out drain loop) versus a single type swap in rust-postgres#1348. Memory profile is the same in the worst case (deadlock scenario): the buffering still has to cover one full result set during the typeinfo round-trip. The difference is *where* it's buffered, not *how much*. Minimal reproduction (passes with this patch, deadlocks on master): https://github.com/rubenfiszel/tokio-postgres-deadlock-repro
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This is an alternative to #1348 for the same deadlock. Same root
cause, different remediation. The two PRs are mutually exclusive — pick
whichever you prefer; closing the other.
The deadlock is described in detail in #1348 and reproduced minimally
here: https://github.com/rubenfiszel/tokio-postgres-deadlock-repro
TL;DR:
query::query_typedcallsget_type(client, oid).awaitsynchronously while still holding the streaming
Responses; on aresult schema with unknown OIDs (citext, custom enums / domains,
postgis geometry, …) the original query's
DataRows back up in theper-
Responsesmpsc::channel(1),Connection::poll_readstopsdraining the wire, and the typeinfo sub-query response (queued behind
those
DataRows on the same socket) never arrives.Approach (vs. #1348)
Responseschannelunboundedchannel(1)(unchanged)VecDeque<(BackendMessages, bool)>onResponseResponsesreceiver type)This PR keeps the
mpsc::channel(1)perResponsesand instead changesConnection::poll_readto keep draining the wire when a sender backs up— parking the unsent frame on a per-response queue inside
Responseitself. New wire frames are routed to
target_response_idx()— thefirst in-flight response whose
ReadyForQueryhasn't been observed yet— so a congested streaming query no longer blocks delivery to a
concurrent typeinfo sub-query whose response is queued behind it on the
same socket.
Why per-response (and not a single global parked queue)
A single global queue still deadlocks: once
response[0]'s sender isfull, the global queue accumulates frames for
response[1]that wehave no way to deliver without first delivering
response[0]'sframes. Per-response queues let us poll each sender independently in
drain_all_parked.The
completion_seenflagResponse::completion_seentracks whether theReadyForQueryframefor that response has been observed on the wire — independent of
whether it's been delivered to the consumer's channel yet.
target_response_idx()uses this to route subsequent wire frames tothe next response while the completion frame for the previous one is
still flushed in-order to its own consumer when its sender unblocks.
Reproduction
master:with this PR:
Test plan
cargo build -p tokio-postgrescleancargo test -p tokio-postgres --libpassesmaster)
query_typed_rawcall against a partitioned table with a
citextcolumn on Neon)that this PR's author originally hit — hangs indefinitely on
master, completes in ~420ms with this patch
Why two PRs
#1348 is the minimal fix (one type swap). This PR preserves the
bounded-channel API at the cost of ~100 lines of routing bookkeeping.
Functionally and memory-wise they're equivalent. I have a slight
preference for #1348 for the obvious reason (smaller surface), but
opened both so you can pick based on whether the bounded-vs-unbounded
distinction matters to you for some reason I might not be aware of.