Skip to content

Other download sites open their own connections and do not follow redirects #1152

Description

@aalhossary

#1151 gives FileDownloadUtils a redirect-following connection opener. Other code opens its own connections and has the same exposure: a redirect is either not followed, or its body is read as though it were the file.

Downloads that cache what they read

These are the CATH shape (#1138) — a redirect body written to disk and parsed later, far from the cause.

Location Current
chem/AllChemCompProvider.downloadFileFromRemote:97 remoteURL.openStream() into a hand-rolled copy loop, no status check
io/sifts/SiftsChainToUniprotMapping:153 new GZIPInputStream(DEFAULT_URL.openStream())
scop/Astral:150,168 new InputStreamReader(url.openStream())
symmetry/utils/BlastClustReader:157 u.openStream()
genome/parsers/cytoband/CytobandParser:70 new GZIPInputStream(u.openStream())
modfinder/phosphosite/Dataset:157 u.openStream()

Reads that would misinterpret a redirect

Location Consequence
PDBStatus:95,113,150 objectMapper.readValue(url.openStream(), …) — a redirect body is parsed as JSON. Same endpoint as #1146.
io/LocalPDBDirectory.getLastModifiedTime:622 reads Last-Modified off url.openConnection() with no status check, so on a redirect it reports the redirect's headers — a moved file looks like it has no timestamp, which feeds FetchBehavior.FETCH_IF_OUTDATED
align/util/URLConnectionTools.openURLConnection sets timeouts only; no redirect handling, no status check. Used by DownloadChemCompProvider and others.

One that already tried

core/sequence/loader/UniprotProxySequenceReader.openURLConnection:557 hand-rolls redirect following, and its own comment says:

// This method should be moved to a utility class in BioJava 5.0

It has the same 307/308 gap, no hop limit, and cycle detection that does not work — it compares the new location only against the current URL, so an A→B→A→B alternation loops forever. It does one thing #1151 does not: preserve Set-Cookie across hops, which Uniprot needs.

So migrating it needs openConnectionFollowingRedirects to grow optional cookie preservation first. That seems the natural first candidate, since it both removes duplicated logic and fixes two real bugs.

Suggested order

  1. UniprotProxySequenceReader — add cookie preservation to the shared opener, then delete the hand-rolled copy.
  2. URLConnectionTools — routing this covers several callers at once.
  3. The caching downloads above, which are the ones that can poison a cache.
  4. PDBStatus and getLastModifiedTime.

Not urgent individually; worth doing before the next service moves. TestChemCompRedirectNotCached and FileDownloadRedirectTest both show the offline HttpServer pattern for testing these without a network dependency (#1146).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions