feat(bigtable): bigtable-ycsb console script + accelerator system-test app-profile default - #17
Conversation
| try: | ||
| if op == "read": | ||
| table.read_row( | ||
| _key(chooser.next_index()), |
There was a problem hiding this comment.
This has O(N) complexity everytime we do a read. So if there are a million records, it's gonna be very slow. can we optimize it?
There was a problem hiding this comment.
Fixed. Replaced the O(N)-per-read inverse-CDF scan with the O(1) closed-form zipfian sampler from Gray et al. ("Quickly Generating Billion-Record Synthetic Databases") — the same math YCSB's ZipfianGenerator uses. The only O(n) cost left is the one-time zeta(n) precompute in Chooser.__init__; each read is now constant-time. Verified the skew is unchanged (index 0 hottest, decreasing).
| """Picks a record index under the workload's request distribution.""" | ||
|
|
||
| def __init__(self, record_count: int, distribution: str, rng: random.Random): | ||
| self._n = record_count |
There was a problem hiding this comment.
this doesnt increase with the newly inserted records. Which violates YCSB workload d.
There was a problem hiding this comment.
Fixed. Added a shared InsertCounter handed to every worker's Chooser; inserts now draw contiguous keys from it and the latest distribution samples over counter.current() (loaded rows + rows inserted during the run) instead of a frozen record_count. Verified: with 100 loaded rows, the max key returned by latest rises from 99 to 149 after 50 inserts.
| limit=cfg.scan_length, | ||
| row_filter=_LATEST, | ||
| ) | ||
| async for _row in await table.read_rows_stream( |
There was a problem hiding this comment.
I think this should just be _row in table.read_rows_stream( ?
There was a problem hiding this comment.
This one is on the async driver (_run_workload_async), where the async read_rows_stream is async def ... -> AsyncIterable[Row] — so you await the call to get the async iterator and then async for over it. That matches the documented pattern in the samples (e.g. samples/hello/async_main.py: async for row in await table.read_rows_stream(query)). The sync driver at the corresponding spot already uses the plain for _row in table.read_rows_stream(...) form you're describing. Left as-is — lmk if I'm misreading.
| i = self._rng.randrange(self._field_count) | ||
| return ( | ||
| f"field{i}".encode(), | ||
| bytes(self._rng.getrandbits(8) for _ in range(self._field_length)), |
There was a problem hiding this comment.
this is slow. Can we do
self._rng.randbytes(self._field_length)
instead?
There was a problem hiding this comment.
Done — switched both fields() and one_field() to self._rng.randbytes(self._field_length) (Python 3.9+; package is >=3.10).
0e84777 to
b7e5247
Compare
…_PROFILE Change-Id: Iaf31fbd700d37acadc05300183410725f0292aaa
Change-Id: Ibbc1cb619213769fc6eeda6704aa06ccd02f5212
ac414ce to
3ef4b2b
Compare
b7e5247 to
0890bb6
Compare
Stacked on #16 (base branch
accel-15-stale-tempdir-sweep).Two independent changes for the accelerator pre-release tooling:
1. Harness fix — default system tests to the configured app profile
test(bigtable): default accelerator system tests to BIGTABLE_TEST_APP_PROFILESeveral accelerator system tests build tables inline via
get_table(instance_id, table_id)with no app profile. Against a dev instance the accelerator needs single-cluster routing (e.g.jetstream100); on the default profile the daemon's session pool trips and the client silently falls back to native — which madetest_metrics_parityfail (accel ops were served natively and counted as native).Fix: an autouse session fixture in
tests/system/data/accelerator/conftest.pythat defaultsapp_profile_idfromBIGTABLE_TEST_APP_PROFILEonly when the caller passed none, so tests that pass their own (e.g. config-forwarding's explicittest-profile) are untouched. This is done centrally rather than threaded through ~10 call sites because several test bodies (not just fixtures) build tables inline.2. Ship the YCSB benchmark as a console script
feat(bigtable): ship YCSB benchmark as the bigtable-ycsb console scriptThe YCSB driver previously lived under
tests/system/data/and was not included in the wheel, so a wheel-only user could not run it (python -m tests...→ModuleNotFoundError). It is fully standalone (imports onlygoogle.cloud.bigtable.data), so this moves it into the package atgoogle/cloud/bigtable/data/_benchmarks/ycsb_perf.pyand adds abigtable-ycsbconsole entry point insetup.py.After install:
Verified: module imports,
--helpexits 0,mainresolves as the entry-point target, andfind_namespace_packages()picks up the new_benchmarkssubpackage.