Skip to content

Postgres CDC: schema evolution handling (ALTER TABLE during streaming) #6123

Description

@no-flaks-given

Context

Follow-up from #5988. Flagged by @dumitru-nicolae-marasoiu as deserving a dedicated design pass.

The problem

In `cdc_input.rs`, `Event::Relation` overwrites the connector's `relation_cache` without diffing against the deserializer schema configured at pipeline startup.

If an ALTER TABLE ADD/DROP COLUMN happens mid-stream:

  • New Relation event arrives with updated column list
  • Connector silently updates its cache
  • Subsequent Insert/Update/Delete events produce JSON with fields the deserializer doesn't expect (or misses fields the deserializer does expect)
  • Deserializer errors may surface as parse errors, but the schema mismatch itself is invisible

Options

  1. Conservative: detect any schema diff against startup schema and surface a fatal error. Forces pipeline restart with new schema.
  2. Informative: detect diff and emit a structured "schema changed" error that surfaces to operators but doesn't necessarily kill the pipeline.
  3. Best-effort reconcile: attempt to reconcile the deserializer schema with the new Relation, error only if incompatible (e.g., removed a required field).
  4. Ignore (current behavior): silently update cache, let downstream deserialization fail opaquely.

Open questions

  1. What's Feldera's general stance on source schema changes? Do other connectors handle this?
  2. Is there a way to update the deserializer schema at runtime, or is it fixed at pipeline start?
  3. For (1) conservative: is a clean restart viable, or does Feldera need graceful schema migration?

Related

  • nmarasoiu's note in #5988 comment: "Event::Relation overwrites the connector's relation_cache without diffing against the deserializer schema configured at startup. An ALTER TABLE ADD/DROP COLUMN mid-stream silently produces misaligned JSON. Worth a dedicated design pass in a follow-up: schema evolution deserves more than a patch here."

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions