Skip to content

RandomizedContraction can leave checkpoint directories behind after a failed run #894

Description

@SauronShepherd

Problem

RandomizedContraction creates a per-run checkpoint directory and deletes it only after the output has been materialized successfully. Its finally block currently unpersists DataFrames and deregisters _axpb, but does not remove the temporary checkpoint directory.

If an intermediate operation fails, checkpoint data can remain in the filesystem or a remote object store. Repeated executions may therefore leak temporary storage.

This is related to the historical cleanup concern in #162, but specifically covers failed executions in the current RandomizedContraction implementation.

Proposed solution

  • Reuse a scoped cleanup routine for the per-run checkpoint directory.
  • Run it on both successful completion and from finally.
  • Do not mask the original failure if cleanup itself cannot complete; log a warning instead.
  • Add a regression test that fails after intermediate parquet materialization and verifies that no randomized-contraction-* directory remains.

The change does not modify public APIs.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions