tests: fix datafusion_memory_mb wiring and parameterize on workers/hosts - #6754

test_udp.py runs with FELDERA_TEST_NUM_WORKERS=8 by default and sets
datafusion_memory_mb to 512. The adhoc query engines pre-reserves
SORT_SPILL_RESERVATION_BYTES (64 MiB = 67,108,864 bytes) per partition,
which already exceeds the 512 MB pool.  This caused CI runs to fail.
This was introduced in commit 2ee3bf1 ("[manager] Set the memory
limits from the local OS/container when they are not specified"), which
added datafusion_memory_mb=512 to the test without accounting for the
8-worker reservation math.

This commit fixes the problem by parameterizing the memory for
datafusion on the number of workers.  It also fixes a bug in the Python
API, which didn't provide the datafusion_memory_mb RuntimeConfig field,
instead trying to set it via the resources field, which doesn't work.

Signed-off-by: Ben Pfaff <blp@feldera.com>

Commit 92b29a4 ("[dbsp] Avoid repeatedly listing files in storage with
no checkpoints.") changed Checkpointer::read_checkpoints() to write an
empty checkpoint file if it found that one did not exist.  A failing
CI run demonstrated that this in fact introduced a race against
checkpoint synchronization.

This commit fixes the problem by using a different approach: instead of
writing a checkpoint catalog file, we only do the full scan of the
directory once at startup instead of every time we try to read the catalog.
This still accomplishes the original goal of avoiding doing a full
directory scan every time.

Fixes: #6750

Signed-off-by: Ben Pfaff <blp@feldera.com>