Skip to content

Postgres CDC: PostgresStore creates etl.* schema on source DB #6122

Description

@no-flaks-given

Context

Follow-up from #5988. Flagged by @dumitru-nicolae-marasoiu.

The Postgres CDC connector uses etl's PostgresStore to persist table replication phases across restarts (avoiding full re-snapshot). PostgresStore::new(pipeline_id, source_config) uses the source connection to create an etl schema with 4 tables directly in the customer's source Postgres database.

Implications

  • Requires CREATE SCHEMA / CREATE TABLE privileges on the source DB (not just REPLICATION + SELECT)
  • Read-replicas can't serve as sources (CREATE is DDL, replicas are read-only)
  • An etl schema appears in the customer's DB with nothing in Feldera's config making that explicit
  • Customer may be confused/surprised by unexpected schema appearing

Options

A. Document only — cheapest. Add prominent notes on PostgresCdcReaderConfig, known limitations page, privilege requirements.

B. Separate state DB config field (meaningful hardening)

  • New optional config field like state_uri: Option<String> defaulting to source
  • Pass a separate PgConnectionConfig to PostgresStore::new()
  • Customer's source DB stays clean; state lives in a DB they chose

C. Custom minimal StateStore

  • Implement etl's StateStore trait with a single-table or file backing
  • Decided against in PR discussion — maintenance burden high, trait evolves

D. Integrate with Feldera's checkpoint system

  • Serialize etl's state into Feldera's checkpoint instead of a separate store
  • Largest change — requires upstream work on both etl and Feldera sides

Open questions

  1. Is B worth doing, or is A sufficient for now?
  2. If B: what's the config field name/shape? Any concerns about supporting partial configs (source DB but with different credentials for DDL)?
  3. Is D ever going to happen? If so, B might be redundant effort.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions