[docs] Document how to optimize multihost performance for unbalanced large data. - #6758
Conversation
| If the pipeline is stopped and restarted during a retry, the connector resumes from the last successfully | ||
| ingested table version. This guarantees that no data loss occurs due to object store read errors. | ||
|
|
||
| ## Optimizing multihost performance for unbalanced large data |
There was a problem hiding this comment.
much of this is not specific to Delta, maybe there should be a separate page about this.
There was a problem hiding this comment.
This is definitely true as the amount of advice increases; I wasn't sure whether it was yet for n = 2.
| If the pipeline is stopped and restarted during a retry, the connector resumes from the last successfully | ||
| ingested table version. This guarantees that no data loss occurs due to object store read errors. | ||
|
|
||
| ## Optimizing multihost performance for unbalanced large data |
There was a problem hiding this comment.
This advice doesn't work for tables with lateness, since different connectors can go out of sync.
There was a problem hiding this comment.
I fixed this by adding a warning paragraph:
> ⚠️ [Tables with LATENESS] cannot correctly be divided into multiple
> connectors this way, because the different connectors can read data
> "out of sync" from another.
[Tables with LATENESS]: /sql/streaming/#lateness-expressions
| be added to the partition whose events have been completely | ||
| processed. | ||
|
|
||
| ## Optimizing multihost performance for unbalanced large data |
There was a problem hiding this comment.
Kafka already doesn't guarantee lateness across multiple partitions, but we have the synchronize_partitions feature, which is not applicable if we split ingest across multiple connectors.
There was a problem hiding this comment.
I fixed this by adding a warning paragraph:
> ⚠️ The [synchronize_partitions] feature does not work across
> connectors, only within a single connector. Therefore, [tables with
> LATENESS] cannot correctly be divided into multiple connectors this
> way, because the different connectors can read data "out of sync"
> from another.
[tables with LATENESS]: /sql/streaming/#lateness-expressions
[synchronize_partitions]: #synchronize_partitions
mythical-fred
left a comment
There was a problem hiding this comment.
Useful docs addition — one typo to fix (appears in both files).
b5858de to
09eeeef
Compare
…data. Signed-off-by: Ben Pfaff <blp@feldera.com>
09eeeef to
10ddb18
Compare
mythical-fred
left a comment
There was a problem hiding this comment.
Re-approving after rebase + follow-up. The new warning callouts about LATENESS + synchronize_partitions not working across connectors are exactly the caveat this section needed. Nice.
No description provided.