A multithreaded Trino client that fetches large result sets via the spooling protocol and assembles them into a single Pandas DataFrame. Designed for high-throughput queries where segments can be downloaded in parallel to saturate available I/O.

The script runs a SQL query against a Trino cluster with spooling enabled. As the cursor yields segments, the main thread enqueues them into a bounded raw_segment_queue. A configurable pool of worker threads dequeues segments, fetches and decompresses the data (over HTTP for spooled segments, or inline), deserialises it with orjson, and pushes the resulting DataFrame into a results_queue. Once all workers have finished, the main thread concatenates everything into a single final_df.

Cursor → raw_segment_queue (maxsize=20) → [worker threads] → results_queue → pd.concat → final_df

Spooled segments are fetched over HTTPS using httpx and decompressed with zstandard. Failed segment fetches are retried up to 3 times with exponential backoff via tenacity.

pandas
httpx
psutil
trino
orjson
tenacity
zstandard

Install with:

pip install pandas httpx psutil trino orjson tenacity zstandard

python trino_parallel_fetch.py [-f FETCH_THREADS]

Example — fetch with 8 threads:

python trino_parallel_fetch.py -f 8

Logs are written to logs/trino_client_f<N>.log where <N> is the number of fetch threads.

The Trino connection is hardcoded near the top of the script. Update the following before running:

conn = trino.dbapi.Connection(
    host='your-trino-host',
    user='your-user',
    port=443,
    catalog='your-catalog',
    schema='your-schema',
    auth=BasicAuthentication('your-user', 'your-password'),
    session_properties={
        'spooling_enabled': 'true'
    },
    encoding='json+zstd'
)

The target SQL query is also hardcoded and should be updated for your use case:

sql = "SELECT * FROM orders LIMIT 10000000"

Because multiple worker threads fetch and enqueue segments concurrently, results are added to results_queue in completion order, not cursor order. The final pd.concat therefore produces a DataFrame whose row order does not match the original query result order.

Each SpooledSegment exposes a rowOffset metadata attribute indicating where that segment's rows begin in the full result set:

To reconstruct the correct row order, workers would need to attach the rowOffset to each DataFrame (e.g. as metadata or a sort key) before enqueuing, and the final assembly step would need to sort by that value before concatenating.