Skip to content

[sql] Give the streams that leave a recursive circuit ids of their own - #6852

Merged
ryzhyk merged 1 commit into
mainfrom
fix-recursive-export-persistent-id
Aug 13, 2026
Merged

[sql] Give the streams that leave a recursive circuit ids of their own#6852
ryzhyk merged 1 commit into
mainfrom
fix-recursive-export-persistent-id

Conversation

@ryzhyk

@ryzhyk ryzhyk commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

See commit message.

The compiler assigned the same ids to streams inside the recursive fragment and
the same streams exported outside the fragment:

```rust
let (s6, s9) = circuit.recursive(|circuit, (unused_0, s3)| {
    ...
    let s6 = ...join...;  s6.set_persistent_id(Some("ed5fc63e…"));   // inside the scope
    Ok((s6, s9))
}).unwrap();
s6.set_persistent_id(Some("ed5fc63e…"));                             // exported — same id
```

However these are different streams with different integrals as can be seen
from the circuit:

```
    ┌───────────────────────────────────────────────────────────────┐
    │                                                               │
 i  │               ┌───┐ |-s6 inside the scope                     │
────┼──►δ0─────────►│   │ V    ┌────────┐       ┌───────────────┐   │   ┌───────────┐ s6 exported
    │               │ f ├─────►│distinct├──┬───►│integrate_trace├───┼──►│consolidate├───────►
    │       ┌──────►│   │      └────────┘  │    └───────────────┘   │   └───────────┘
    │       │       └───┘                  │                        │
    │       │                              │                        │
    │       │                              │                        │
    │       │       ┌────┐                 │                        │
    │       └───────┤z^-1│◄────────────────┘                        │
    │               └────┘                                          │
    │                                                               │
    └───────────────────────────────────────────────────────────────┘
```

Suffix the exported stream's id with '.export', in both the single-file and the
multi-crate writer, mirroring the '.delay' suffix that already distinguishes a
recursive input. The checkpoint then holds <id>.shard.accintegral for the inner
trace and <id>.export.shard.accintegral for the outer one.

Bump RECURSIVE_STATE_VERSION with it: the layout of what a recursive view stores
changes, so the ids in the dataflow graph have to change too, or the pipeline
manager would diff the views as unchanged while the runtime finds their state
missing, and refuse to start with UnexpectedBootstrap. The metadata test's
golden ids move for the same reason.

Signed-off-by: Leonid Ryzhyk <leonid@feldera.com>
(cherry picked from commit cce7456)
@ryzhyk
ryzhyk requested a review from mihaibudiu August 13, 2026 14:45
* separate state. Sharing it makes an inner trace and an outer one write
* the same file, and whichever restores second reads a batch that was
* written with the other's layout. */
void computeExportedHash(DBSPOperator operator) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this function is only used once maybe it can be inlined

@ryzhyk
ryzhyk added this pull request to the merge queue Aug 13, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Aug 13, 2026
@ryzhyk
ryzhyk added this pull request to the merge queue Aug 13, 2026
Merged via the queue into main with commit 5fda3b4 Aug 13, 2026
1 check passed
@ryzhyk
ryzhyk deleted the fix-recursive-export-persistent-id branch August 13, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants