google-cloud-storage: background BucketMetadataCache request shares the upload transport and correlates with intermittent SSLEOFError

Determine this is the right repository

• I determined this is the correct repository in which to report this bug.

This concerns client-side behavior introduced by google-cloud-storage 3.11.0, not the semantics of the Cloud Storage API.

Summary of the issue

Context

We run a Python service on Google Cloud Run that performs multipart object uploads to Google Cloud Storage. Upload calls are synchronous and execute in worker threads. We have tried both a client scoped to each worker thread and a new storage.Client for each upload.

After upgrading from google-cloud-storage 3.10.1 to 3.13.0, intermittent uploads began exhausting the normal 120-second retry deadline with this underlying error:

google.api_core.exceptions.RetryError: Timeout of 120.0s exceeded, last exception:
HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded
with url: /upload/storage/v1/b/<bucket>/o?uploadType=multipart
(Caused by SSLError(SSLEOFError(8,
'[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1032)')))

The failures are infrequent, but they have occurred across multiple Cloud Run instances and independent upload jobs. They persisted when we stopped sharing clients between application worker threads and instead created and closed a client for every upload.

While investigating why client isolation did not eliminate concurrent use of a transport, we found that 3.11.0 introduced App-centric Observability bucket metadata enrichment. On the first blob operation for a bucket, create_trace_span_helper() calls BucketMetadataCache.get_or_queue_fetch(). A daemon thread then calls:

self._client.get_bucket(bucket_name, timeout=10.0)

BucketMetadataCache retains the same Client used by the foreground operation. Consequently, the metadata request and multipart upload can call the same google.auth.transport.requests.AuthorizedSession concurrently. This happens even when application code guarantees that only one worker thread uses a client.

When clients are short-lived, the cache begins empty for every upload. Cloud Storage request metrics then show approximately one GetBucketMetadata request for every WriteObject request. In other words, creating a fresh client per upload increases rather than removes this hidden concurrency.

This report does not claim that the shared session is conclusively the source of every TLS EOF. The evidence establishes the unexpected concurrent transport use and a strong version/timing correlation. We would like maintainer guidance on whether concurrent use of one AuthorizedSession by the cache and a foreground transfer is intended and supported.

Related issue #17650 describes the same cache's unexpected storage.buckets.get calls and IAM/audit-log impact. This report focuses on transport sharing and reliability during object transfers.

Expected Behavior

An object upload performed with an otherwise isolated storage.Client should not concurrently use that client's HTTP transport from an internal daemon thread.

If background bucket metadata collection is required, it should use an independently owned transport, be enabled only when its telemetry is used, or be configurable per client. A failure or retry storm in optional metadata enrichment should not overlap or interfere with the foreground transfer's transport and retry window.

Actual Behavior

The first blob operation on a cache miss starts _fetch_background in a daemon thread. That thread and the foreground upload use the same client and AuthorizedSession concurrently.

Operationally, multipart uploads intermittently fail after the full retry deadline with SSLEOFError: UNEXPECTED_EOF_WHILE_READING. The first observed failures appeared in the first deployment containing google-cloud-storage 3.13.0; earlier retained logs using 3.10.1 did not contain this failure signature. Changing application-level client ownership from thread-local clients to one client per upload did not remove the failures.

We are using a rollback to 3.10.1 as a controlled diagnostic because it predates BucketMetadataCache. We will update the upstream issue with the result after sufficient comparable traffic.

API client name and version

google-cloud-storage 3.13.0

The relevant behavior was introduced in 3.11.0 and remains in 3.13.1. Version 3.13.1 includes the process-wide DISABLE_GCS_PYTHON_CLIENT_OTEL_BUCKET_METADATA environment variable, but it does not isolate the background request's transport when metadata enrichment is enabled.

Reproduction steps: code

The intermittent TLS failure depends on a connection being closed at an unfortunate point and is not deterministic. The following complete diagnostic instead reproduces the prerequisite hidden concurrency: a metadata daemon and multipart upload entering the same AuthorizedSession.request() method at the same time.

It requires Application Default Credentials and an existing test bucket supplied through GCS_BUCKET. It uploads and deletes one temporary object.

file: main.py

import os
import threading
import uuid

# Ensure the behavior under investigation is enabled before importing storage.
os.environ.pop("DISABLE_GCS_PYTHON_CLIENT_OTEL_BUCKET_METADATA", None)

from google.cloud import storage


bucket_name = os.environ["GCS_BUCKET"]
client = storage.Client()
transport = client._http
original_request = transport.request

metadata_entered = threading.Event()
upload_entered = threading.Event()
metadata_finished = threading.Event()


def instrumented_request(method, url, *args, **kwargs):
    method = method.upper()
    is_metadata = method == "GET" and "/storage/v1/b/" in url
    is_multipart_upload = "uploadType=multipart" in url

    if is_metadata:
        print(
            "metadata request entered:",
            f"thread={threading.current_thread().name}",
            f"session_id={id(transport)}",
        )
        metadata_entered.set()
        upload_entered.wait(timeout=30.0)
    elif is_multipart_upload:
        metadata_entered.wait(timeout=30.0)
        print(
            "upload request entered:",
            f"thread={threading.current_thread().name}",
            f"session_id={id(transport)}",
            f"metadata_already_entered={metadata_entered.is_set()}",
        )
        upload_entered.set()

    try:
        return original_request(method, url, *args, **kwargs)
    finally:
        if is_metadata:
            metadata_finished.set()


transport.request = instrumented_request
blob = client.bucket(bucket_name).blob(f"metadata-cache-repro/{uuid.uuid4()}.txt")

try:
    blob.upload_from_string(b"test", content_type="text/plain")
    metadata_finished.wait(timeout=60.0)
finally:
    transport.request = original_request
    try:
        blob.delete()
    finally:
        client.close()

Reproduction steps: supporting files

None.

Reproduction steps: actual results

With google-cloud-storage 3.13.0, the diagnostic shows the metadata request entering from _fetch_background and the multipart upload entering from the application thread with the same session identifier:

metadata request entered: thread=Thread-1 (_fetch_background) session_id=<same id>
upload request entered: thread=MainThread session_id=<same id> metadata_already_entered=True

The two waits make the overlap deterministic: the metadata thread does not continue until the upload has entered the same transport.

In normal, uninstrumented operation, most uploads succeed. A controlled run of 64 concurrent uploads also completed successfully, confirming that a connection-level failure is needed to expose the intermittent problem. When the failure occurs in the deployed service, the final result after retries is the SSLEOFError traceback shown in the summary.

With 3.10.1, there is no metadata daemon and the diagnostic prints only the upload request with metadata_already_entered=False.

Reproduction steps: expected results

The upload should be the only request using its AuthorizedSession, or any automatic background metadata request should use a separate transport:

upload request entered: thread=MainThread session_id=<upload session> metadata_already_entered=False

OS & version + platform

Debian Linux container on Google Cloud Run.

We observed the deployed failures with OpenSSL reporting _ssl.c:1032. Local inspection and controlled testing used Linux with OpenSSL 3.5.6.

Python environment

Python 3.13.15 in the affected Cloud Run revision. Controlled local testing used Python 3.13.14.

Python dependencies

google-api-core==2.33.0
google-auth==2.56.2
google-cloud-core==2.6.0
google-cloud-storage==3.13.0
google-resumable-media==2.10.0
requests==2.34.2
urllib3==2.7.0
cryptography==50.0.0

Additional context

Version boundary and call chain

• 3.10.1 does not contain BucketMetadataCache.
• 3.11.0 added "Enhance Otel Span Attributes with BucketId and Location details for every Bucket/Blob operation."
• On a cache miss, get_or_queue_fetch() starts a daemon threading.Thread.
• _fetch_background() calls self._client.get_bucket(bucket_name, timeout=10.0).
• get_bucket() retains its default DEFAULT_RETRY; the 10-second request timeout does not by itself disable the normal retry policy.
• Both requests therefore use the same client's HTTP transport. Application-level thread-local or per-operation client ownership does not prevent this internal sharing.
• 3.13.1 still contains this implementation. Its environment-variable opt-out avoids the cache globally, but there is no per-client ownership choice and no transport isolation while the feature is enabled.

Relevant source:

• 3.11.0 changelog entry
• BucketMetadataCache.get_or_queue_fetch() in 3.13.1
• BucketMetadataCache._fetch_background() in 3.13.1
• Related issue #17650

Other causes investigated

During affected periods we did not find corresponding CPU exhaustion, memory pressure, Cloud Run request-concurrency spikes, outbound packet/byte throttling, instance lifecycle events, or a Cloud NAT path. Failures occurred on different instances rather than one persistently unhealthy process. Cloud Storage metrics showed successful writes surrounding a small number of UNAVAILABLE or CANCELLED requests.

urllib3 2.7.0 removes and closes a connection after an SSL exception before a retry, so the library's retries are not repeatedly selecting the same known-failed pooled socket. Recreating the whole storage.Client also did not improve recovery, which led us to inspect what else a new client starts during its first operation.

Questions and requested change

1. Is concurrent use of one AuthorizedSession by _fetch_background() and a foreground upload an intended and supported client contract?
2. Could BucketMetadataCache use a separately owned client/transport, or collect metadata synchronously before a transfer begins, so optional enrichment cannot overlap the transfer on one session?
3. Could the cache be disabled per Client through a public constructor option? The environment variable in 3.13.1 is useful but process-wide and unavailable to callers that need different behavior for different clients.
4. Could a regression test exercise uploads while the metadata request encounters delayed responses, connection resets, and TLS EOFs, verifying that the foreground transfer remains isolated?

We can test a proposed patch or provide additional sanitized timing and request-level diagnostics.