Context
Follow-up from #5988. Flagged by @dumitru-nicolae-marasoiu as deserving a dedicated design pass.
The problem
In `cdc_input.rs`, `Event::Relation` overwrites the connector's `relation_cache` without diffing against the deserializer schema configured at pipeline startup.
If an ALTER TABLE ADD/DROP COLUMN happens mid-stream:
- New Relation event arrives with updated column list
- Connector silently updates its cache
- Subsequent Insert/Update/Delete events produce JSON with fields the deserializer doesn't expect (or misses fields the deserializer does expect)
- Deserializer errors may surface as parse errors, but the schema mismatch itself is invisible
Options
- Conservative: detect any schema diff against startup schema and surface a fatal error. Forces pipeline restart with new schema.
- Informative: detect diff and emit a structured "schema changed" error that surfaces to operators but doesn't necessarily kill the pipeline.
- Best-effort reconcile: attempt to reconcile the deserializer schema with the new Relation, error only if incompatible (e.g., removed a required field).
- Ignore (current behavior): silently update cache, let downstream deserialization fail opaquely.
Open questions
- What's Feldera's general stance on source schema changes? Do other connectors handle this?
- Is there a way to update the deserializer schema at runtime, or is it fixed at pipeline start?
- For (1) conservative: is a clean restart viable, or does Feldera need graceful schema migration?
Related
- nmarasoiu's note in #5988 comment: "Event::Relation overwrites the connector's relation_cache without diffing against the deserializer schema configured at startup. An ALTER TABLE ADD/DROP COLUMN mid-stream silently produces misaligned JSON. Worth a dedicated design pass in a follow-up: schema evolution deserves more than a patch here."
Context
Follow-up from #5988. Flagged by @dumitru-nicolae-marasoiu as deserving a dedicated design pass.
The problem
In `cdc_input.rs`, `Event::Relation` overwrites the connector's `relation_cache` without diffing against the deserializer schema configured at pipeline startup.
If an ALTER TABLE ADD/DROP COLUMN happens mid-stream:
Options
Open questions
Related