Skip to content

Fix two races in controller.rs - #6786

Merged
ryzhyk merged 2 commits into
mainfrom
fix-backpressure-late-reader-latch
Aug 1, 2026
Merged

Fix two races in controller.rs#6786
ryzhyk merged 2 commits into
mainfrom
fix-backpressure-late-reader-latch

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

See commit messages.

Describe Manual Test Plan

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

Mark if you think the answer is yes for any of these components:

Describe Incompatible Changes

ryzhyk added 2 commits August 1, 2026 00:14
The backpressure thread recorded an endpoint id in `running_endpoints` before
checking for that endpoint's reader, so an endpoint that was registered but not
yet opened lost its only `extend()`: `insert` returned false on every later
scan, leaving the endpoint paused forever. Skip a reader-less endpoint instead,
and the scan that follows `add_input_endpoint`'s unpark starts it.

`add_input_endpoint` registers an endpoint with `reader: None` and installs the
reader only after `open` returns, a window open since d24ba40. Only endpoints
added to a running pipeline can land in that window, because the connectors
named in the pipeline configuration are all created before the backpressure
thread starts. Every HTTP `/ingress` push creates such an endpoint, and the
endpoint blocks in `complete_request` until `extend` arrives, so a lost command
hangs the request until the client gives up. That is how
`server::test_http::test_concurrent_bootstrap` failed in CI, where the client
timed out after 120 seconds, and the same window can hang an ingress request in
production.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
…ance

`add_input_endpoint` compares the endpoint's fault tolerance with the pipeline's
only after it has registered and opened the endpoint, and the rejection path
returned the error without undoing either step. The endpoint stayed in the
status map, where it could never run because the error path also skipped the
unpark that starts it, yet it still counted toward the pipeline's statistics and
reserved its own name against a later attempt to add the connector. Disconnect
it instead, which both removes the entry and shuts the reader down.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
@ryzhyk
ryzhyk requested a review from gz August 1, 2026 07:21
@ryzhyk ryzhyk added the connectors Issues related to the adapters/connectors crate label Aug 1, 2026
@ryzhyk
ryzhyk added this pull request to the merge queue Aug 1, 2026
Merged via the queue into main with commit a7a7deb Aug 1, 2026
1 check passed
@ryzhyk
ryzhyk deleted the fix-backpressure-late-reader-latch branch August 1, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

connectors Issues related to the adapters/connectors crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants