[connectors] Fix completion counters on output removal - #6743
Merged
Conversation
…away `total_completed_records` and `total_completed_steps` are the minimum over the registered output endpoints. An endpoint that has not delivered a step holds both back, and `disconnect_output` drops an endpoint along with everything still queued for it, so a departing endpoint never reports the progress the others already have. `remove_output` only dropped the map entry: the only other refresh happens on a step or an output batch, and an idle pipeline produces neither, so the minimum stayed pinned at the departed endpoint's last step forever. `/completion_status` then reports `inprogress` for a pipeline that has delivered all of its output, and a checkpoint waiting on `total_completed_records` in `CheckpointThread::run` never unblocks. `server::test_with_kafka::test_server` hit this: it opens 200 `/egress` streams and drops each client instantly, and ~39 of them were still torn down in the window where the circuit ran the steps carrying the final output. Failure: https://github.com/feldera/feldera/actions/runs/30380682951/job/90350576953 Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
`wait_for_completion` unwrapped the timeout, so a stall said only that 20 seconds had passed. Dump the completion status and `/stats` first: the step the token waits for, next to each connector's `total_processed_steps`, names the endpoint that is holding `total_completed_steps` back. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
ryzhyk
force-pushed
the
fix-completion-counters-on-output-removal
branch
from
July 28, 2026 19:01
707820c to
824b05a
Compare
ryzhyk
enabled auto-merge
July 28, 2026 19:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A one-line fix for a real race found by CI.
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes