[connectors] Soft deletes - #6747

Soft deletes are a property of the connector rather than of the data
format, so the tests drive records through each real parser into a table
whose `is_delete` column comes from record metadata: JSON
`insert_delete`, `debezium`, and `raw`, CSV, raw, Parquet, and Avro
`debezium`. Formats that only express insertions are covered too, since
they must stay unaffected.

Connectors that parse columnar data themselves reach the table through
the Arrow stream, so that path is covered directly, including a batch
that mixes polarities, and end to end through the Delta Lake connector
in `cdc` mode. A controller test covers the property reaching the
connector from the pipeline configuration, and two more cover the
rejected configurations.

CSV records are positional and must therefore carry a value for every
column, which overrides the metadata; the test records that behavior.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

The docs include an example query that uses row_number to reconstruct
the state of the table.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

Wording, mostly: a table with soft deletes represents the stream of
updates it receives rather than accumulating state, so drop
"accumulates" from the docs, the changelog, the connector config, the
module documentation, and the Python test.

Also:

* Say that any column can order the changes of a key, with connector
  metadata as the recommended special case, instead of prescribing a
  metadata timestamp.
* Explain the `is_delete` tie-break as a tie-break, without claiming an
  arbitrary order.
* Report `is_delete = true` in the log line that records a soft-delete
  connector taking effect.
* Note why splitting a mixed-polarity Arrow batch is safe: a table with
  soft deletes has no primary key, so its updates commute.
* Correct the claim that every connector bypassing a parser writes to
  the Arrow stream; Postgres CDC writes to the record stream.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

…mary key

A deletion in a table with a primary key identifies a key rather than a
record, so a soft-delete connector has nothing to insert in its place.
The runtime already rejects such a configuration; the compiler now
rejects it too, and points at the offending property.

The same check rejects `soft_delete` on a view connector, since soft
deletes describe the records a connector ingests.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

The example materialized the whole change log as a `history` view, which
grows without bound. Remove it: the change log is what the table already
is, and the `live` view below it is what a reader wants.

Add, commented out pending further work, a section explaining that `live`
itself needs unbounded state, since a change carrying any timestamp can
displace the record that is currently latest for its key, and showing a
temporal filter as one way to bound it. The Python test for that variant
stays live: it checks that a change older than the window reaches the
table yet stays out of the view.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>

Four test modules each carried their own copy of the same three
functions: a random topic name, a topic creation that waits for the
broker, and a best-effort deletion. Collect them in tests/kafka.py,
along with a kafka_topics context manager that pairs creation with
deletion, and delete the copies.

test_connector_status also passed the AdminClient and NewTopic classes
around through a _kafka_clients indirection, which the module-level
imports already made unnecessary.

Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>