Postgres CDC: strict_checkpoint_sync config flag — default behavior decision #6121

Context

Follow-up from #5988. The Postgres CDC connector currently fires async_result on completion_watcher (step completion), which advances the replication slot as soon as Feldera finishes processing a step. This is fast but has a correctness race for stateful circuits.

The problem

1. Feldera processes step 55 — total_completed_steps advances to 55
2. We fire async_result → etl advances the Postgres replication slot past step 55's LSN
3. Crash before Feldera's next circuit checkpoint (last checkpoint was at step 50)
4. Restart: Feldera reloads circuit state from checkpoint 50; etl resumes from slot position (post step 55)
5. Events 51–55 are NOT redelivered — circuit state is missing those inputs; windows/joins are silently wrong

For stateless circuits (forward CDC events to a sink) this is fine — reprocessing same inputs yields same outputs. For stateful circuits (windowed aggregations, joins with state) this is a silent correctness bug.

Proposed fix

Add config flag strict_checkpoint_sync: bool:

• When true: wait on checkpoint_watcher instead of completion_watcher. Slot only advances after Feldera actually checkpoints.
• When false: current behavior (fast).

Tradeoff is real and user-visible: batches gated on checkpoint cadence instead of step completion. If Feldera checkpoints every ~30s, downstream latency goes from few-second to ~30s. Data sits in etl's buffers waiting for the checkpoint.

Open questions for discussion

1. What should the default be?

   • nmarasoiu argues strict should default, performance opt-in: "correctness-opt-in is a wrong default shape; performance-opt-in is the healthy one". The connector has no way to detect whether a user's pipeline is stateless + idempotent-sink (the only safe cell for fast mode).
   • Counter: most CDC pipelines are stateless forwarding; strict adds real latency for the common case.

2. Research needed: How does CheckpointCoordination enum map to step numbers? Earlier investigation didn't fully pin this down. We need to know which step was captured in each checkpoint to fire the right deferred async_results.

3. Typical checkpoint cadence in Feldera? This determines the latency hit.

Related

• Discussion started in #5988 comment from @dumitru-nicolae-marasoiu
• Parent tracking issue: Postgres CDC connector: follow-up fixes and improvements (tracking) #6126