Problem
RandomizedContraction creates a per-run checkpoint directory and deletes it only after the output has been materialized successfully. Its finally block currently unpersists DataFrames and deregisters _axpb, but does not remove the temporary checkpoint directory.
If an intermediate operation fails, checkpoint data can remain in the filesystem or a remote object store. Repeated executions may therefore leak temporary storage.
This is related to the historical cleanup concern in #162, but specifically covers failed executions in the current RandomizedContraction implementation.
Proposed solution
- Reuse a scoped cleanup routine for the per-run checkpoint directory.
- Run it on both successful completion and from finally.
- Do not mask the original failure if cleanup itself cannot complete; log a warning instead.
- Add a regression test that fails after intermediate parquet materialization and verifies that no randomized-contraction-* directory remains.
The change does not modify public APIs.
Problem
RandomizedContraction creates a per-run checkpoint directory and deletes it only after the output has been materialized successfully. Its finally block currently unpersists DataFrames and deregisters _axpb, but does not remove the temporary checkpoint directory.
If an intermediate operation fails, checkpoint data can remain in the filesystem or a remote object store. Repeated executions may therefore leak temporary storage.
This is related to the historical cleanup concern in #162, but specifically covers failed executions in the current RandomizedContraction implementation.
Proposed solution
The change does not modify public APIs.