Skip to content

Pod quota chain runs on CSS-internal /.internal/ writes — full pod walks per chunk and before/after every write — causing WrappedExpiringReadWriteLocker 6s lock expiries on IDP adapter storage and O(chunks × pod) write overhead. #2208

Description

@bourgeoa

Environment

  • Pivot with CSS 7.2.0

Description

  • Problem
    With a per-pod quota enabled (config/storage/backend/pod-quota-file.json),
    every write — including writes to CSS-internal storage under /.internal/
    goes through the quota chain:
  1. QuotaValidator calls getAvailableSpace before and after every write
    and createQuotaGuard mid-stream.
  2. Default PodQuotaStrategy.createQuotaGuard calls getAvailableSpace (a
    full pod walk) per stream chunk: chunks × O(pod size).
  3. getAvailableSpace does pod discovery (searchPimStorage) + a full
    recursive size walk via FileSizeReporter when the pod isn't cached;
    FileSizeReporter also accumulates large dirent arrays in memory.
  • Observed impact (production)
    The IDP KeyValueStorage (AuthorizationCode store at
    /.internal/idp/adapter/) is backed by
    ResourceStore → ResourceStore_Backend → FileDataAccessor, so login writes
    hit the same chain:

    [WrappedExpiringReadWriteLocker] error: Lock expired after 6000ms on
    /.internal/idp/adapter/AuthorizationCode/
    [WrappedExpiringStorage] error: ... Failed to remove expired entries ...

On a sufficiently large pod, any internal write exceeds the 6s lock expiry —
logins and refresh flows fail intermittently.

  • Root cause
    ignoreFolders: ["^/\\.internal$"] only excludes .internal from the size
    reporter's walk — it does not stop QuotaValidator from validating
    internal writes. CSS-internal storage shares the user accessor chain with pod
    data.

  • Suggested fix
    Exempt /.internal/* at the quota strategy/validator level (return unlimited
    space without pod discovery/walk), not only in the reporter's walk excludes —
    as implemented in solid-contrib/pivot (FastQuotaStrategy.getAvailableSpace
    + delta-accessor skip).

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions