Skip to content

Pricing refresh: send Accept-Encoding and revalidate with ETag (1.7MB -> ~8KB per run) #1564

Description

@wishworldbetter

What do you want to change?

The LiteLLM pricing fetch in rust/crates/ccusage/src/http.rs. Two things: enable ureq's gzip feature (right now no Accept-Encoding is sent, so the table transfers uncompressed), and keep the body + ETag on disk so later runs revalidate with If-None-Match instead of re-downloading.

Why?

Every non-offline run downloads the full pricing table, 1.7 MB on the wire. I hit this while chasing constant traffic on my machine — a statusline plus a 1-minute usage collector were re-downloading the same JSON ~3000 times a day, about 5 GB. The server side already supports everything needed:

identity:               1.7 MB
gzip:                    82 KB
If-None-Match (match):  HTTP 304, empty body

How? (optional)

ureq gzip feature + a best-effort body/etag pair in the temp dir (same spirit as the statusline cache), If-None-Match on later runs, and falling back to the last validated body when revalidation fails. Working locally: 1.7 MB per run before -> 160 KB cold / 8 KB warm after (warm is just the TLS handshake plus 304 headers). Tested with unit tests, fmt and clippy clean. Happy to open a PR — the branch is ready.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions