Skip to content

tokio-postgres: fix deadlock in query_typed/prepare on unknown OIDs (per-response parking variant — alternative to #1348) - #1349

Open
rubenfiszel wants to merge 1 commit into
rust-postgres:masterfrom
rubenfiszel:fix-query-typed-raw-deadlock-surgical
Open

rubenfiszel wants to merge 1 commit into
rust-postgres:masterfrom
rubenfiszel:fix-query-typed-raw-deadlock-surgical

Conversation

@rubenfiszel

Copy link
Copy Markdown

Summary

This is an alternative to #1348 for the same deadlock. Same root
cause, different remediation. The two PRs are mutually exclusive — pick
whichever you prefer; closing the other.

The deadlock is described in detail in #1348 and reproduced minimally
here: https://github.com/rubenfiszel/tokio-postgres-deadlock-repro
TL;DR: query::query_typed calls get_type(client, oid).await
synchronously while still holding the streaming Responses; on a
result schema with unknown OIDs (citext, custom enums / domains,
postgis geometry, …) the original query's DataRows back up in the
per-Responses mpsc::channel(1), Connection::poll_read stops
draining the wire, and the typeinfo sub-query response (queued behind
those DataRows on the same socket) never arrives.

Approach (vs. #1348)

#1348 this PR
Per-Responses channel unbounded channel(1) (unchanged)
Per-response parked queue n/a VecDeque<(BackendMessages, bool)> on Response
Lines changed ~25 ~120
Memory profile in deadlock case identical (one result set's worth) identical
Public API impact (Responses receiver type) bounded → unbounded unchanged

This PR keeps the mpsc::channel(1) per Responses and instead changes
Connection::poll_read to keep draining the wire when a sender backs up
— parking the unsent frame on a per-response queue inside Response
itself. New wire frames are routed to target_response_idx() — the
first in-flight response whose ReadyForQuery hasn't been observed yet
— so a congested streaming query no longer blocks delivery to a
concurrent typeinfo sub-query whose response is queued behind it on the
same socket.

Why per-response (and not a single global parked queue)

A single global queue still deadlocks: once response[0]'s sender is
full, the global queue accumulates frames for response[1] that we
have no way to deliver without first delivering response[0]'s
frames. Per-response queues let us poll each sender independently in
drain_all_parked.

The completion_seen flag

Response::completion_seen tracks whether the ReadyForQuery frame
for that response has been observed on the wire — independent of
whether it's been delivered to the consumer's channel yet.
target_response_idx() uses this to route subsequent wire frames to
the next response while the completion frame for the previous one is
still flushed in-order to its own consumer when its sender unblocks.

Reproduction

git clone https://github.com/rubenfiszel/tokio-postgres-deadlock-repro
cd tokio-postgres-deadlock-repro
./setup.sh
# Test against master (deadlocks at limit ~100+):
cargo run --release
# Test against this PR (passes at all limits):
# edit Cargo.toml [patch.crates-io] to point at this branch
cargo run --release

master:

[limit 100]  ok (100 rows) in 663µs
[limit 200]  TIMEOUT after 10s
[limit 500]  TIMEOUT after 10s

with this PR:

[limit 100]  ok (100 rows) in 761µs
[limit 200]  ok (200 rows) in 1.5ms
[limit 500]  ok (500 rows) in 898µs

Test plan

  • cargo build -p tokio-postgres clean
  • cargo test -p tokio-postgres --lib passes
  • Standalone repro: passes at all LIMITs (deadlocks at 100+ on
    master)
  • End-to-end against a real workload (the same query_typed_raw
    call against a partitioned table with a citext column on Neon)
    that this PR's author originally hit — hangs indefinitely on
    master, completes in ~420ms with this patch

Why two PRs

#1348 is the minimal fix (one type swap). This PR preserves the
bounded-channel API at the cost of ~100 lines of routing bookkeeping.
Functionally and memory-wise they're equivalent. I have a slight
preference for #1348 for the obvious reason (smaller surface), but
opened both so you can pick based on whether the bounded-vs-unbounded
distinction matters to you for some reason I might not be aware of.

…a has unknown OIDs (per-response parking variant)

This is an alternative to the unbounded-channel fix in rust-postgres#1348. Same root
cause, different remediation — see rust-postgres#1348 for the deadlock analysis.

Keeps the per-response `mpsc::channel(1)` and instead changes
`Connection::poll_read` to keep draining the wire when a sender backs
up, parking the unsent frame on a per-response `parked` queue inside
`Response` itself. New wire frames are routed to the first response
whose `ReadyForQuery` hasn't been observed yet, so a congested
streaming query no longer blocks delivery to a concurrent typeinfo
sub-query whose response is queued behind it on the same socket.

Per-response (rather than a single global parked queue) is the critical
detail: a global queue still deadlocks because once response[0]'s
sender is full, the global queue accumulates frames for response[1]
that we have no way to deliver without first delivering response[0]'s
frames. Per-response queues let us poll each sender independently.

A new `completion_seen: bool` on `Response` tracks whether the
`ReadyForQuery` frame has been *observed* on the wire (even if it's
parked, not yet delivered to the consumer). `target_response_idx()`
uses this to route subsequent wire frames to the next response, while
the parked completion frame is still flushed in-order to its own
consumer when its sender unblocks.

Trade-off versus rust-postgres#1348:
- Pro: preserves the bounded `mpsc::channel(1)` semantics — Each
  response's parked queue is bounded only by the size of that
  particular request's result set; well-behaved consumers continue
  to apply backpressure to the connection task as before.
- Con: ~100 lines of additional bookkeeping (per-response parked
  queues, completion tracking, fan-out drain loop) versus a single
  type swap in rust-postgres#1348.

Memory profile is the same in the worst case (deadlock scenario): the
buffering still has to cover one full result set during the
typeinfo round-trip. The difference is *where* it's buffered, not
*how much*.

Minimal reproduction (passes with this patch, deadlocks on master):
https://github.com/rubenfiszel/tokio-postgres-deadlock-repro
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant