[python] Reclaim test pipelines a killed CI run leaves running - #6801
Merged
Conversation
Pipelines from the runtime suites keep running on the shared CI instance long after their run ends, consuming compute until the daily cleanup job stops them. Teardown only runs while pytest is alive. Cancelling the workflow run, evicting a runner or timing out a job kills it outright, and every pipeline in flight survives. A new 'if: always()' step sweeps up what such a run left behind, matching the name prefix unique_pipeline_name stamps on every test pipeline so it only ever touches its own run. It stops and clears storage but never deletes, so a failed run stays readable. Three defects made teardown fail on its own as well: - Every stop and clear waited without a timeout, so one wedged pipeline hung teardown until the runner killed the job, leaking all the rest. reclaim_pipeline() now bounds each wait. - stop() raising skipped the clear that followed it. Every step now runs regardless, since a pipeline that never reports Stopped can still release its storage. - Pipelines from new_pipeline_with_suffix() were in no teardown at all. SharedTestPipeline now tracks everything it builds. Cleanup failures warn rather than fail a test: a slow stop says nothing about what the test asserted, and the sweep reports anything genuinely left running. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
mihaibudiu
approved these changes
Aug 4, 2026
Review feedback: ci_run_prefix() wrapped one call for one call site, so the call site names the prefix itself. warn() now says why it prints a workflow command under Actions and plain text elsewhere: ::warning:: reaches the run's annotation summary, and nothing reads it outside a run. Drop tests/unit/test_reclaim_pipeline.py. It drove a mocked client and mostly restated the two functions it covered. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
tests/platform/helper.py built its Authorization header from FELDERA_API_KEY, read once when the module was imported. The CI auth step exports a GitHub ID token under that name and the token expires well inside a suite, so every raw request issued after that got a 401. test_adaptive_joins waits through two compilations before it reads circuit_json_profile, which is where it failed. The header now comes from feldera_bearer_token(), which resolves the same credential the SDK client uses and re-mints an ID token shortly before it expires. That function replaces _feldera_credential(): both spelled out the same precedence, and what is left to decide is whether a caller wants the resolver or its result. The client gets the resolver under Actions, where the token can lapse between requests. A 401 also made reset_pipeline() return before it stopped anything, so a suite that ran past the expiry left its pipelines on the shared instance. Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pipelines from the runtime suites keep running on the shared CI instance long after their run ends, consuming compute until the daily cleanup job stops them. Names like
adc8e_test_uda,17eca_negative_tests2_617224c4and532f6_jsonbench-defaultaccumulate across many commits.Why they survive
Teardown only runs while pytest is alive. Cancelling the workflow run, evicting a runner or timing out a job kills it outright, and every pipeline in flight survives.
ci.yml's cancel sentinels make this routine: any sibling job failing cancels the whole run, killing the runtime tests mid-test on both matrix legs.jsonbenchleaking despite a correctfinallyis the clearest evidence.Three defects made teardown fail on its own as well:
timeout_s=Noneinto the SDK'swhile Truepollshared_test_pipeline.py,platform/helper.py,aggtst_base.pyeach did stop-then-clear with no guardnew_pipeline_with_suffix(10 call sites intest_checkpoint_sync.py)Changes
python/tests/stop_ci_run_pipelines.py, wired intotest-integration-runtime.ymlas anif: always()step. Matches the prefixunique_pipeline_namestamps, so it only ever touches its own run. It stops and clears storage but never deletes, so a failed run stays readable; the daily job deletes afterwards.feldera.testutils.reclaim_pipeline()— one definition of reclaiming a pipeline safely: each wait bounded at 60 s, every step attempted even when the previous raised, failures returned rather than thrown. Used by all four cleanup paths.SharedTestPipelinetracks every pipeline it builds, sonew_pipeline_with_suffixresults are torn down too.reset_pipelineruns before a test as well as after, where a previous run's leftover would have failed an unrelated test.Not addressed:
test_tpch.pyandtest_now.pycallstop()outside atry, so a failure there leaks a large pipeline until the sweep runs at the end of the job. Fixing in place needs a ~150-line re-indent; the sweep bounds the damage.Checklist
Breaking Changes?
No.