Cloud database performance benchmarking as a service.

Data Bench is a comprehensive benchmark suite for managed cloud databases where you have no direct disk access. It measures I/O throughput, query optimizer efficiency, transaction isolation correctness, and network transport characteristics — all through standard SQL operations over the wire.

⚠️ DELETE YOUR TEST INSTANCE WHEN YOU ARE DONE ⚠️

Data Bench does not create or manage cloud infrastructure, and it does NOT delete it. The test database server / managed instance you provision to benchmark is yours to tear down. Cloud databases bill continuously for as long as they exist — a forgotten Hyperscale, Managed Instance, or high-tier server can quietly run up hundreds or thousands of dollars per month.

When your benchmarking is finished, delete the test server/instance (and its resource group / project resources) in your cloud console. Data Bench cleans up only its own test tables — never the server itself. You are solely responsible for any costs incurred.

Prerequisites: Python 3.11+ and git. That's it — no database is installed locally; Data Bench connects to a database you already have.

# 1. Clone the repo
git clone https://github.com/sqlshep/CloudBench.git
cd CloudBench

# 2. Create an isolated environment and install
python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -e .

# 3. Launch the web UI
sqlio-cloud web                  # add --port 9000 to change the port

Then open http://localhost:8080 in your browser and sign in:

From there, enter your database connection details, pick a preset, and run. See How do I use it? for the full walkthrough.

Troubleshooting: If the connection page shows "Data Bench IP address: unable to detect" or every connection times out, the machine running Data Bench has no outbound internet access — run it somewhere that can reach your database's host and port. Connection errors on the page include specific, copy-pasteable remediation steps.

Deploying to the cloud? See the Docker Deployment Guide for step-by-step instructions on running Data Bench as a container on Azure (Container Apps), Google Cloud (Cloud Run), and AWS (App Runner / ECS).

Do not run Data Bench against QA, staging, or production databases.

Data Bench generates synthetic load — concurrent reads, writes, bulk inserts, transaction storms, and full table scans. This will consume CPU, IOPS, memory, and transaction log throughput on the target database. Running it against a database that serves real traffic can degrade application performance, trigger throttling, or exhaust DTU/vCore budgets.

What to do instead:

1. Provision a dedicated test server. Create a new database server (or managed instance) on the same SKU/tier you plan to evaluate. This gives you an isolated environment and ensures the benchmark results reflect the true performance of that tier without interference from other workloads.
2. Create an empty database (or let Data Bench do it). Data Bench creates and manages its own test tables — you just need a blank database to connect to. Name it something like benchmarks or cloudbench_test. If the database doesn't exist, Data Bench will create it automatically when you test the connection, and enable snapshot isolation for SQL Server instances.
3. Make the database endpoint reachable over the internet. Data Bench connects to your database remotely, so the server must accept inbound connections on its SQL port (1433 for SQL Server, 5432 for PostgreSQL, 3306 for MySQL). Ensure that public network access is enabled and that any firewall rules allow traffic from external IPs. Consult your cloud provider's documentation for how to configure network access on your specific service. Remember to lock it back down or delete the test server when you're done.
4. Use a dedicated test credential. Create a SQL user specifically for benchmarking with only the required permissions (see "What database permissions do I need?" below). Do not reuse production service accounts.
5. Run Data Bench in the same region as the database. Every operation is a network round-trip, so the physical distance between Data Bench and your database is added to every measurement. Running from a laptop or a container in a different region can add tens to hundreds of milliseconds per operation, which dominates latency numbers and caps throughput — you end up measuring the network, not the database.
6. Tear down after testing. Once you've exported your results, delete the test server to avoid ongoing charges. Data Bench cleans up its own tables, but the server/instance itself is your responsibility.

For a serious benchmark, co-locate Data Bench with the database. Deploy the Data Bench container to the same cloud region as your target database (e.g., both in westus2, us-central1, or us-east-1). This keeps round-trip latency at the sub-millisecond level typical of intra-region traffic, so the results reflect the database's true performance rather than the distance between you and it. Cross-region or over-the-public-internet runs are fine for a quick smoke test, but the latency and IOPS figures will not be representative. See the Docker Deployment Guide for deploying into a specific region.

Signing in: The web UI is protected by a simple placeholder login. Use these credentials to access the benchmark lab:

Field	Value
Username	sqladmin
Password	cloudbench

This is not a secure credential — it exists only to gate the demo interface. Change it (_AUTH_USER / _AUTH_HASH in src/sqlio_cloud/web/app.py) before exposing Data Bench publicly.

1. Connect — Enter your database host, port, credentials, and dialect (SQL Server, PostgreSQL, or MySQL). The connection page displays the Data Bench host IP address for your firewall rules. Data Bench validates the connection, detects the server version, and auto-creates the database if it doesn't exist.
2. Configure — Choose a preset (Smoke, Standard, or Full Stress) or customize individual tests, concurrency levels, and block sizes.
3. Run — Data Bench executes the selected tests with live progress streaming. No agents or software are installed on your database server — everything runs through SQL over the network.
4. Analyze — Interactive results with per-metric tooltips, latency distribution charts, and thread-scaling curves. Export as CSV or JSON to compare multiple servers.

*Azure SQL Managed Instance: Managed Instance sits inside an Azure Virtual Network and has no public endpoint by default. To benchmark a MI:

1. Enable the public endpoint — Portal → MI → Security → Networking → Public endpoint → Enable. This can take 5–10 minutes to provision.
2. Add an NSG inbound rule — Allow TCP on port 3342 from the Data Bench IP address (displayed on the connection page) or from 0.0.0.0/0 for a throwaway test instance.
3. Use the public hostname — It contains .public. in the name (e.g., yourmi.public.abc123.database.windows.net). The default hostname without .public. is the private VNet endpoint and will not work from outside the VNet.
4. Port is 3342 (not 1433) — Data Bench sets this automatically when you select "Azure SQL Managed Instance" from the dropdown.
5. Database auto-creation — You do not need to create the database ahead of time. If the database name you enter does not exist, Data Bench will create it automatically and enable snapshot isolation.

Since you should be provisioning a dedicated test instance anyway, enabling the public endpoint is safe — just delete the instance when testing is complete.

Data Bench connects as a regular SQL user. The account needs:

For the simplest setup, use the server admin login — this is the credential set when you created the server or managed instance. If you prefer a least-privilege approach, create the database manually and grant the user db_owner on that database.

Yes. Credentials are held in memory only for the duration of the benchmark run and discarded when the session ends. Nothing is written to disk or logged. All communication with your database uses its native wire protocol (typically TLS-encrypted), and the web UI runs over HTTPS.

No. Data Bench only creates and operates on its own test tables, which use a bench_ or dsb_ prefix. These tables are dropped before each run and cleaned up after. Your existing tables are never read, modified, or dropped. Nothing is installed on the database server.

These tests measure the database's storage-layer throughput and latency through SQL-level operations, using a test table with configurable binary payloads (1 KB – 64 KB block size).

These tests validate data correctness and ACID guarantees under concurrent load.

Generates a normalized star-schema dataset (zones, countries, vendors, buyers, products, inventory, orders, order details) and executes 16 original analytical queries covering:

• Multi-way hash/merge joins across 4+ tables
• Correlated subqueries with EXISTS / NOT EXISTS
• Aggregation with GROUP BY, HAVING, CASE expressions
• Common Table Expressions (CTEs) and derived tables
• ORDER BY with LIMIT/TOP for ranking queries
• LEFT JOIN distributions and NULL handling

Each query runs cold (first execution, no plan cache) and warm (repeated execution) to measure plan compilation overhead and cache speedup ratio. Results include per-query duration, row output preview, and status.

Composite metrics:

• Geometric Mean — exp(Σ ln(duration_i) / N) — fairly weights queries across different magnitude ranges
• Power Score — (3600 × Scale_Factor) / Geometric_Mean — normalized throughput metric for cross-comparison across scale factors

Isolates the transport layer from the database engine to distinguish network-bound vs. compute-bound performance limits.

A quick sanity check to validate connectivity, basic IOPS, latency floor, and query compilation.

Best for: initial connection validation, quick regression checks, CI/CD smoke gates.

Balanced coverage of all test types. The recommended default for most benchmarks.

Best for: cloud tier comparison, capacity planning, SLA validation, pre-production baselines.

Comprehensive stress test with maximum concurrency. Designed for production-grade assessments.

Best for: production tier selection, regression testing, cloud provider comparison, identifying limits under sustained load.

Note: Execution time varies by instance tier. Burstable/Basic tiers with high network latency will take longer; Premium/Business Critical tiers will complete faster.

Every test produces detailed metrics. Hover over any metric in the results page for a detailed tooltip explanation. Here's a summary:

IOPS (I/O Operations Per Second) is calculated as total_operations / elapsed_seconds, where elapsed time is measured precisely using time.perf_counter() and frozen when the test completes. This prevents timing drift from subsequent tests inflating the elapsed time.

What counts as an "operation" depends on the test:

For sequential scan and bulk insert, IOPS represents rows per second because each row processed is the fundamental unit of work. This makes the IOPS and MB/s numbers consistent — if you see 10,000 IOPS at 8 KB block size, you should see approximately 78 MB/s throughput (10,000 × 8 KB / 1,024).

Every I/O test produces a Performance Over Time chart showing per-second IOPS and MB/s throughput throughout the test duration. The throughput tracker takes snapshots every second, recording:

• IOPS (per interval) — instantaneous operations/sec during that 1-second window
• Avg IOPS (cumulative) — running average from test start to that point
• MB/s — instantaneous data throughput during that window

These charts reveal performance variability that summary numbers hide — such as throttling events, noisy-neighbor interference, buffer pool warm-up effects, and cloud provider burst credit depletion.

Metric cards display color-coded ratings. The thresholds are context-sensitive — what's "GOOD" for a random point lookup is different from a sequential scan:

Random Read, Random Write, Mixed Workload:

Sequential Scan, Bulk Insert (rows/sec — naturally higher throughput):

Latency (p50 and p99):

These ratings are relative guidelines. Actual expectations depend on your cloud tier, network distance, and workload. A Basic-tier database at 200 IOPS is operating normally; a Premium-tier database at 200 IOPS warrants investigation.

The I/O tests run each workload at increasing thread counts (e.g., 1 → 4 → 8 → 16 → 32) and measure IOPS at each level. The scaling sweep produces:

From the scaling curve, Data Bench calculates Amdahl's serial fraction — the portion of the workload that cannot be parallelized, no matter how many concurrent connections are used.

The formula: Speedup = 1 / (f + (1 - f) / N), where f is the serial fraction and N is the thread count.

This metric is directly actionable for capacity planning: if writes show a serial fraction of 0.25, throughput will never exceed ~4x single-threaded performance regardless of connection count. The solution is to reduce write contention (batching, partitioning) or upgrade to a tier with a faster log subsystem.

The block size controls the binary payload stored in each row of the I/O test table. It affects whether the benchmark is latency-bound or throughput-bound:

The block size is also used as the page size for SHA-256 integrity verification.

Data Bench is built for teams benchmarking multiple servers. From the results page you have three export options:

• PDF Report — A professionally formatted, multi-page document ready to share with stakeholders. Includes a branded cover page with server details, an executive summary table, detailed per-test metrics with latency distributions, thread-scaling tables, embedded charts, and any errors encountered. Suitable for attaching to architecture review documents, vendor evaluations, or cloud migration assessments.
• CSV — Flat section, metric, value, unit format. Every metric from every test is flattened into rows with server metadata (host, dialect, version, timestamp, preset). Import into Excel, Google Sheets, or a Jupyter notebook and pivot to compare servers side-by-side.
• JSON — Complete structured result object with full nesting for programmatic analysis.

Filenames include the server host, preset, and timestamp for easy identification: benchmark_myserver.database.azure.com_standard_2026-03-30T14-25-00.pdf

Execution time depends on several factors:

• Instance tier — Burstable and Basic tiers have lower IOPS limits and higher per-operation latency than Premium or Business Critical tiers. A Smoke test that takes 5 minutes on a premium instance may take 15 minutes on a Basic tier.
• Network latency — Data Bench connects to your database over the network. The physical distance between Data Bench and your database adds round-trip time to every operation. Higher latency means each individual operation takes longer, which compounds across thousands of operations.
• Database load — If other workloads are running concurrently on the same database, they compete for resources and increase latency. This is why we recommend a dedicated test server (see "Before you start").
• Scale factor — Higher analytical scale factors generate more data and run longer queries.

If a test appears stuck, check the progress indicator — it updates with the current operation count (e.g., "150 / 500").

Yes. After choosing a preset, you can check or uncheck individual tests on the configure page. The preset is just a starting point — you can add or remove any test before starting the benchmark. You can also adjust the I/O block size independently.

All test tables are dropped when the benchmark completes. If the benchmark is interrupted (browser closed, timeout), the tables remain in the database but are automatically dropped at the start of the next run. The tables use distinctive prefixes (bench_io, bench_accounts, bench_net_bw, dsb_zone, dsb_country, etc.) so they're easy to identify and drop manually if needed.

Data Bench is released under the MIT License. You are free to use, modify, and distribute it, including for commercial purposes, provided the copyright notice and license text are retained.