[qa] test_bucket_preferred_over_read_bucket non-deterministic test failure #6750

FAILED tests/platform/test_checkpoint_sync.py::TestCheckpointSync::test_bucket_preferred_over_read_bucket - RuntimeError: Unable to START the pipeline:
Unable to transition the pipeline to one of the states: ['running', 'awaitingapproval'].
Reason: The pipeline is in a STOPPED state due to the following error:
Operation failed because the pipeline failed to initialize. Error details: DBSP error: storage error: /pipeline-storage/checkpoints.feldera.mut: rename failed: entity not found.

What failed

TestCheckpointSync::test_bucket_preferred_over_read_bucket (python amd64, merge-queue run for PR 1835) at step 2's first self.pipeline.start() on fresh storage:

DBSP error: storage error: /pipeline-storage/checkpoints.feldera.mut: rename failed: entity not found

PR 1835 is innocent (it only touches crates/coord). The regression is in the feldera submodule.

Root cause

feldera commit 92b29a43d "[dbsp] Avoid repeatedly listing files in storage with no checkpoints" (2026-07-27 18:48 UTC, still in origin/main) made Checkpointer::read_checkpoints write an empty catalog when checkpoints.feldera is absent — feldera/crates/dbsp/src/circuit/checkpointer.rs:468:

// Write an empty checkpoint file to save the cost of listing all the files next time.
backend.write_json(&file_name, &VecDeque::<CheckpointMetadata>::new())?;

read_checkpoints has three concurrent callers on a starting pipeline:

┌────────────────────────────────────────────┬────────────────┬─────────────────────────────────────┬─────────────────────┐
│                   caller                   │     thread     │              file:line              │      on error       │
├────────────────────────────────────────────┼────────────────┼─────────────────────────────────────┼─────────────────────┤
│ get_status (/status, /coordination/status) │ actix worker   │ adapters/src/server.rs:1647         │ logged, ignored     │
├────────────────────────────────────────────┼────────────────┼─────────────────────────────────────┼─────────────────────┤
│ get_checkpoints (/checkpoints)             │ actix worker   │ adapters/src/server.rs:2227         │ 500 to client       │
├────────────────────────────────────────────┼────────────────┼─────────────────────────────────────┼─────────────────────┤
│ Checkpointer::new                          │ circuit thread │ dbsp/src/circuit/checkpointer.rs:53 │ fails pipeline init │
└────────────────────────────────────────────┴────────────────┴─────────────────────────────────────┴─────────────────────┘

Every writer stages the catalog through one fixed name. create_named opens checkpoints.feldera.mut with create(true).truncate(true) (no O_EXCL, posixio_impl.rs:459-470), and complete() renames it away (posixio_impl.rs:290). Two writers in that window means the second fs::rename finds no source file: ENOENT, whose Display is "entity not found".

The pipeline's HTTP server serves /status while the init thread runs (server.rs:938-960), and the manager polls it every 1s while provisioning (pipeline_automata.rs:178). The main pipeline in this test sat in starting for ~150s with empty storage, so the missing-catalog branch was sampled repeatedly. The circuit thread lost the race, so the failure surfaced as InitializationError (server.rs:451), which is why the message reads "failed to initialize".

Reproduction

Two threads calling Checkpointer::read_checkpoints on a fresh PosixBackend — failed on trial 0 with the identical error:

trial 0: read_checkpoints failed: storage error: /var/folders/.../checkpoints.feldera.mut: rename failed: entity not found

Fix

The write is a cache warm-up, so it must never fail the caller. Making it best-effort turns the test green (200 trials) with the other 8 checkpointer tests still passing. Patch plus the regression test: scratchpad/fix_and_repro.patch, applied in the worktree scratchpad/repro (feldera origin/main).

Two things worth deciding before upstreaming:

- Best-effort is the minimal fix, but a read-only-looking status endpoint still mutates storage outside the storage dirlock (runtime.rs:429). Moving the write into Checkpointer::new, which owns the lock, and leaving read_checkpoints pure would be cleaner; /status then pays for a directory listing only until the circuit comes up.
- Latent hazard from the same commit, reasoned but not experimentally confirmed: a /status thread that entered the missing-catalog branch and then stalled can write [] over a real catalog that commit() wrote meanwhile, dropping a committed checkpoint from the catalog and exposing its UUID dir to the next gc_startup. Best-effort error handling does not close that hole; keeping the write off the status path does.