Skip to content

Postgres CDC: expose additional configuration options #6124

Description

@no-flaks-given

Context

Follow-up from #5988. The initial config is intentionally minimal (`uri`, `publication`, `source_table`, `tls`). Several useful knobs are currently hidden or derived.

Candidates for exposure

Slot name customization

Currently derived from a deterministic hash of `(host:port/db, publication, source_table)`. Users can't:

  • Choose a specific slot name for operational visibility
  • Run two Feldera pipelines against the same publication + table (would collide)

Proposal: Optional `slot_name: Option` that overrides the derived hash.

Batch config

ETL's batch size and max fill duration are hardcoded to defaults. Power users may want to tune for their workload (e.g., high-throughput tables want larger batches).

Proposal: Optional `batch: Option` that overrides ETL defaults.

TCP keepalive

Hardcoded to `TcpKeepaliveConfig::default()`. Some networks need tuned keepalive.

Proposal: Optional `tcp_keepalive: Option`.

Memory backpressure

ETL's memory backpressure config is hardcoded. Users may want to tune it.

Proposal: Optional `memory_backpressure: Option`.

Open questions

  1. How much of ETL's config surface should we proxy through? All of it, or only the fields we think users need?
  2. For slot name: should we validate it's a legal Postgres identifier?
  3. Should these all go in the top level or be nested under an `advanced: {...}` config object to signal "most users don't need this"?

Metadata

Metadata

Assignees

No one assigned

    Labels

    connectorsIssues related to the adapters/connectors crate

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions