Context
Follow-up from #5988. Flagged by @dumitru-nicolae-marasoiu.
The Postgres CDC connector uses etl's PostgresStore to persist table replication phases across restarts (avoiding full re-snapshot). PostgresStore::new(pipeline_id, source_config) uses the source connection to create an etl schema with 4 tables directly in the customer's source Postgres database.
Implications
- Requires
CREATE SCHEMA / CREATE TABLE privileges on the source DB (not just REPLICATION + SELECT)
- Read-replicas can't serve as sources (CREATE is DDL, replicas are read-only)
- An
etl schema appears in the customer's DB with nothing in Feldera's config making that explicit
- Customer may be confused/surprised by unexpected schema appearing
Options
A. Document only — cheapest. Add prominent notes on PostgresCdcReaderConfig, known limitations page, privilege requirements.
B. Separate state DB config field (meaningful hardening)
- New optional config field like
state_uri: Option<String> defaulting to source
- Pass a separate
PgConnectionConfig to PostgresStore::new()
- Customer's source DB stays clean; state lives in a DB they chose
C. Custom minimal StateStore
- Implement etl's
StateStore trait with a single-table or file backing
- Decided against in PR discussion — maintenance burden high, trait evolves
D. Integrate with Feldera's checkpoint system
- Serialize etl's state into Feldera's checkpoint instead of a separate store
- Largest change — requires upstream work on both etl and Feldera sides
Open questions
- Is B worth doing, or is A sufficient for now?
- If B: what's the config field name/shape? Any concerns about supporting partial configs (source DB but with different credentials for DDL)?
- Is D ever going to happen? If so, B might be redundant effort.
Related
Context
Follow-up from #5988. Flagged by @dumitru-nicolae-marasoiu.
The Postgres CDC connector uses etl's
PostgresStoreto persist table replication phases across restarts (avoiding full re-snapshot).PostgresStore::new(pipeline_id, source_config)uses the source connection to create anetlschema with 4 tables directly in the customer's source Postgres database.Implications
CREATE SCHEMA/CREATE TABLEprivileges on the source DB (not justREPLICATION+SELECT)etlschema appears in the customer's DB with nothing in Feldera's config making that explicitOptions
A. Document only — cheapest. Add prominent notes on
PostgresCdcReaderConfig, known limitations page, privilege requirements.B. Separate state DB config field (meaningful hardening)
state_uri: Option<String>defaulting to sourcePgConnectionConfigtoPostgresStore::new()C. Custom minimal StateStore
StateStoretrait with a single-table or file backingD. Integrate with Feldera's checkpoint system
Open questions
Related