Skip to content

[demos] Serve the batch-analytics TPC-H data from a live bucket - #6781

Merged
gz merged 1 commit into
mainfrom
demo-data-tpch-bucket
Aug 1, 2026
Merged

[demos] Serve the batch-analytics TPC-H data from a live bucket#6781
gz merged 1 commit into
mainfrom
demo-data-tpch-bucket

Conversation

@gz

@gz gz commented Jul 31, 2026

Copy link
Copy Markdown
Contributor

AWS disabled s3://batchtofeldera along with the account that owned it, so every
read answers 403 AllAccessDisabled. That breaks the accelerating-batch-analytics
packaged demo for anyone who opens it in the Web Console, and fails the
"Validate and run packaged demos" step in CI.

This points the demo, the TPC-H workload tests, and the batch use-case docs at
s3://feldera-demo-datasets/tpch/sf0.1 in us-west-1.

The bucket is provisioned as code with anonymous GetObject and ListBucket, so
readers still need no credentials and aws_skip_signature keeps working.

The replacement carries the same scale factor as the old bucket, so the record
counts in the tutorial are unchanged (lineitem 601k, customer 15.0k, ...).

scripts/tpch_demo_data.py regenerates the data from the official TPC-H dbgen
(v2.17.3). Scale factors 0.01, 0.1 and 1 are published under tpch/sf<N>/.

Describe Manual Test Plan

  • Generated all three scale factors with the official dbgen and confirmed the
    canonical TPC-H cardinalities (sf1: lineitem 6,001,215, orders 1,500,000;
    sf0.1: lineitem 600,572, customer 15,000).
  • Confirmed the Delta schema matches the demo's CREATE TABLE: integer,
    decimal(15,2), date, string, with only n_comment and r_comment
    nullable.
  • Read all 8 tables from S3 with every AWS credential stripped from the
    environment, mirroring how the connector reads them.
  • Verified anonymous LIST and GET over plain HTTPS return 200.

Checklist

  • Unit tests added/updated
  • Integration tests added/updated
  • Documentation updated
  • Changelog updated

Breaking Changes?

  • OpenAPI / REST HTTP API / feldera-types / manager
  • Feldera SQL (Syntax, Semantics)
  • feldera-sqllib
  • Python SDK
  • fda (CLI arguments)
  • Adapters (including configuration)
  • Storage Format / Checkpoints
  • Others (specify)

Describe Incompatible Changes

None. The demo's SQL schema and the tutorial's record counts are unchanged; only
the bucket the data is served from differs.

Comment thread scripts/tpch_demo_data.py
@@ -0,0 +1,306 @@
#!/usr/bin/env -S uv run --script

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how come this is new?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we didnt have a script to generate the dataset

AWS disabled s3://batchtofeldera along with the account that owned it, so every read answers AllAccessDisabled. That breaks the accelerating-batch-analytics demo for anyone who opens it, and fails the packaged-demo step in CI.

Point the demo, the TPC-H workload tests, and the batch use-case docs at s3://feldera-demo-datasets/tpch/sf0.1 in us-west-1, a bucket provisioned as code that allows anonymous reads. The replacement carries the same scale factor as the old bucket, so the record counts in the tutorial stay as they were.

Add scripts/tpch_demo_data.py to regenerate the tables from the official TPC-H dbgen.

Signed-off-by: Gerd Zellweger <mail@gerdzellweger.com>
@gz
gz force-pushed the demo-data-tpch-bucket branch from 0ad46e5 to 74ca430 Compare July 31, 2026 23:07
@gz
gz enabled auto-merge July 31, 2026 23:09
@gz
gz added this pull request to the merge queue Jul 31, 2026
Merged via the queue into main with commit c6acae9 Aug 1, 2026
1 check passed
@gz
gz deleted the demo-data-tpch-bucket branch August 1, 2026 00:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants