[dbsp] Retransmit unacknowledged multihost exchange messages - #6784

You're right that TCP doesn't lose messages, and it didn't. The message is lost above TCP, and I have it fully instrumented now.

What actually happens

Delivery in this protocol is only complete when the server writes the application-level ack at exchange.rs:919 (write_u64_le(next_sequence)). TCP's guarantee ends at the peer's socket buffer, and everything sitting in that buffer dies with the socket. Measured timeline for one wedged connection (client 49999 → server listener 49561):

┌───────────┬─────────────────────────────────────────────────────────────────────┐
│   time    │                                event                                │
├───────────┼─────────────────────────────────────────────────────────────────────┤
│ 35.557600 │ client connects; server accepts 45 µs later, serve task starts      │
├───────────┼─────────────────────────────────────────────────────────────────────┤
│ 35.951050 │ client finishes writing seq 1099 (46 messages total on this socket) │
├───────────┼─────────────────────────────────────────────────────────────────────┤
│ 35.952237 │ test injects "server failure" on the server side                    │
├───────────┼─────────────────────────────────────────────────────────────────────┤
│ 35.952269 │ serve future drops, accepted fd closes, seq 1099 never acked        │
├───────────┼─────────────────────────────────────────────────────────────────────┤
│ 45.953539 │ watchdog: tx parked, min_sequence=1100 channel_seq=1099 queued=1    │
└───────────┴─────────────────────────────────────────────────────────────────────┘

At the 10 s mark the client's socket is still fully alive: netstat shows it ESTABLISHED with Send-Q 0 (the peer's TCP had ACKed the bytes), and getpeername() still succeeds. The server side of that 4-tuple is gone from both netstat and lsof.

So the sender's view is "connection healthy, message sent", and the receiver that would have acked it no longer exists. The sender's only retransmit trigger is a connection error: run() catches it, reconnects, and resets min_sequence to 0 (exchange.rs:496-506). No error, no resend. The tx task parks on nonempty waiting for a new message that never comes, because every worker is blocked behind the unacked one.

The half-open leg

Why the client never observed the close is a rare race, not the normal path. I probed this box directly:

┌──────────────────────────────────┬───────────────────────────────────────────────────────────────┐
│        server closes with        │                          client sees                          │
├──────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ unread data in its receive queue │ RST, recv raises ECONNRESET, both PCBs vanish                 │
├──────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ data already consumed            │ FIN, recv returns EOF, client goes to CLOSE_WAIT              │
├──────────────────────────────────┼───────────────────────────────────────────────────────────────┤
│ never accept()ed (backlog)       │ connection stays healthy, server PCB still listed with Recv-Q │
└──────────────────────────────────┴───────────────────────────────────────────────────────────────┘

The log agrees: of ~6540 injected server failures in that run, nearly all produced a client-side Broken pipe or Connection reset by peer and recovered by reconnecting. Two did not, and those two are exactly the wedged sockets. Their peer PCB is simply gone with no FIN and no RST ever reaching the client, which is a genuinely half-open connection. Nothing probes it afterwards: keepalive is off, and neither side has anything left to send.

Why the fix is the right shape

A protocol layered on TCP cannot use "the connection reported an error" as its liveness signal, because that signal is not guaranteed to arrive. The retransmit timer (exchange.rs:337, 382, 2170) replaces it with an application-level ack timeout, which is the property the protocol actually needs. It covers the production versions of the same failure too: a peer whose process hangs, a lost RST, or a middlebox that silently forgets connection state. In all of those, TCP reports nothing and the current code waits forever.

One incidental find: AbortOnDrop in that file is dead code, defined and never used. Worth deleting, though I left it alone since it's unrelated to this fix.

✻ Cooked for 55m 32s

❯ is this macos-specific?

Good question, and it splits in two: the bug and the reproduction.

  Ran 2 shell commands

Two separate answers.

The bug is not macOS-specific. The sender's liveness signal is "TCP reported an error", and TCP does not report these, on any OS:

┌──────────────────────────────────────────────┬──────────────────────────────────────────────────────────────┐
│                  situation                   │                     what the sender sees                     │
├──────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ peer host crashes, panics, or loses power    │ nothing, ever. No FIN, no RST is generated                   │
├──────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ peer VM suspended or live-migrated           │ nothing while it is frozen                                   │
├──────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ RST lost in the network                      │ nothing. RST is a single datagram and is never retransmitted │
├──────────────────────────────────────────────┼──────────────────────────────────────────────────────────────┤
│ firewall or NAT evicts idle connection state │ nothing, and the wire is idle once the sender blocks         │
└──────────────────────────────────────────────┴──────────────────────────────────────────────────────────────┘

Any of these leaves the sender holding an unacked message with nothing new to send, parked forever, with the socket ESTABLISHED. The lost-RST row is the sharpest one: TCP retransmits data, but a reset is best-effort, so a single drop produces a permanently half-open connection on Linux exactly as on macOS. A protocol that acks at the application layer needs a timeout at the application layer; that conclusion does not depend on the platform.

The specific reproduction is macOS-only so far. The last leg, where the server-side PCB vanishes on loopback and the client's stays ESTABLISHED with no FIN and no RST, is what I measured on this box. I can't claim that microscopic race happens on Linux loopback, where the ordinary close path signals the same way my probes showed here.

Worth noting for calibration: these tests are not #[ignore]d, so Linux CI does run them, and the one recorded Linux failure (issue 6728) is a different symptom from the same fault-injection storm, a duplicate delivery off by one rather than a hang. So the area is racy on both platforms, but the wedge itself has only been seen here. Also, the trigger in the test, inject_fault("server failure"), is test-only; in production the equivalent is a real peer death, rarer but equally invisible to the sender.

If you want this settled empirically rather than by argument, Docker is available and I can run operators_multihost_dynamic in a Linux container against a scratch target dir. Cost is a full cold dbsp build in the container, roughly half an hour of saturated CPU, so I'd rather not start that on your machine while you're working unless you say go.