Skip to content

Postgres CDC connector: follow-up fixes and improvements (tracking) #6126

Description

@no-flaks-given

Context

Tracking issue for follow-up work on the Postgres CDC input connector (landed in #5988). Collects the smaller fixes and improvements from PR review that were deferred.

Items that need discussion have their own separate issues.

Checklist

Small fixes (clear, ~minutes each)

  • PgNumeric NaN/Infinity → Value::Null in cell_to_json and array_cell_to_json
  • Require schema-qualified table matching in FelderaDestination::is_target_table; default to public. when unqualified
  • Add tracing::warn! when write_table_rows falls back to col_0, col_1, ... due to missing relation info
  • Add unit test for nanosecond-precision timestamp round-trip (%.f suppresses trailing zeros)
  • Add // NOTE: comment near Resume::Seek construction explaining identity-vs-position semantics
  • Document that PostgresStore creates an etl.* schema on the source DB in PostgresCdcReaderConfig docstring

Real bug

  • Wire PostgresCdcReaderConfig::tls through to etl's TlsConfig. Currently parse_pg_uri hardcodes TlsConfig::disabled(), so the tls config field is silently ignored.

Config/validation

  • Validate uri, publication, source_table at config parse time (fail at deploy, not pipeline start)
  • Document duplicate-deployment limitation: two Feldera pipelines against the same (uri, publication, source_table) would compute the same pipeline_id and collide on the replication slot

Testing

  • Failover integration test using the full Feldera controller: insert → kill mid-batch → restart → verify no data loss/duplication
  • Integration tests for edge cases:
    • Schema changes mid-stream
    • Large transactions (thousands of rows in single tx, 2MB buffer splitting)
    • Multiple tables in publication (verify source_table filtering)
    • TRUNCATE events (currently logged and ignored — verify no crash)
    • Empty table snapshot (zero rows, then streaming starts)
    • REPLICA IDENTITY without FULL (updates/deletes without old row)
    • Rapid insert/update/delete cycles on the same row
    • Connection loss / Postgres restart mid-stream
    • Slot invalidation (slot dropped externally)
    • Long-running transactions
    • Special float values (NaN, Infinity) through the full pipeline

Documentation

  • User-facing connector documentation page under docs.feldera.com/docs/connectors/sources/ covering setup, privileges, config, limitations, known issues

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions