-[](https://github.com/biojava/biojava/releases/tag/biojava-7.2.2) [](https://github.com/biojava/biojava/blob/master/LICENSE) [](https://gitter.im/biojava/biojava?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[](https://github.com/biojava/biojava/releases/tag/biojava-7.2.5) [](https://github.com/biojava/biojava/blob/master/LICENSE) [](https://gitter.im/biojava/biojava?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
BioJava is an open-source project dedicated to providing a Java framework for **processing biological data**. It provides analytical and statistical routines, parsers for common file formats, reference implementations of popular algorithms, and allows the manipulation of sequences and 3D structures. The goal of the biojava project is to facilitate rapid application development for bioinformatics.
@@ -29,7 +29,7 @@ If you are using Maven you can add the BioJava repository by adding the followin
md5sum, sha1sum and friends). */
+ private static final Pattern BARE_HEX_HASH = Pattern.compile("^([0-9a-fA-F]{32,128})(?:[\\s*].*)?$");
+
+ /** The BSD layout, e.g. MD5 (file.txt) = d41d8cd9.... */
+ private static final Pattern BSD_HASH = Pattern.compile("^\\w+\\s*\\(.*\\)\\s*=\\s*([0-9a-fA-F]{32,128})$");
+
public enum Hash{
MD5, SHA1, SHA256, UNKNOWN
}
+ /**
+ * What to do with the ETag response header when no explicit hash
+ * URL is available.
+ *
+ * Some archives — notably files.wwpdb.org and
+ * files.rcsb.org — return the MD5 digest of the content as
+ * the bare ETag value, which lets us record a real checksum
+ * without a second request. Others (the EBI servers, for instance) return a
+ * <modification-time>-<size> form instead; because that
+ * always contains a -, it can never be mistaken for a hex digest.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public enum ETagPolicy {
+ /** Never look at the ETag header. */
+ IGNORE,
+ /** Record the ETag as a checksum when it is a bare hex digest
+ * of a length matching one of the supported algorithms. */
+ USE_IF_HEX_DIGEST,
+ /** As {@link #USE_IF_HEX_DIGEST}, but log a warning when the header is
+ * missing or is not a usable digest. */
+ REQUIRE
+ }
+
/**
* Gets the file extension of a file, excluding '.'.
* If the file name has no extension the file name is returned.
@@ -95,44 +140,261 @@ public static void downloadFile(URL url, File destination) throws IOException {
int maxTries = 10;
int timeout = 60000; //60 sec
- File tempFile = Files.createTempFile(getFilePrefix(destination), "." + getFileExtension(destination)).toFile();
+ File tempFile = createTempFileFor(destination);
- // Took following recipe from stackoverflow:
- // http://stackoverflow.com/questions/921262/how-to-download-and-save-a-file-from-internet-using-java
- // It seems to be the most efficient way to transfer a file
- // See: http://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html
- ReadableByteChannel rbc = null;
- FileOutputStream fos = null;
- while (true) {
- try {
- URLConnection connection = prepareURLConnection(url.toString(), timeout);
- connection.connect();
- InputStream inputStream = connection.getInputStream();
-
- rbc = Channels.newChannel(inputStream);
- fos = new FileOutputStream(tempFile);
- fos.getChannel().transferFrom(rbc, 0, Long.MAX_VALUE);
- break;
- } catch (SocketTimeoutException e) {
- if (++count == maxTries) throw e;
- } finally {
- if (rbc != null) {
- rbc.close();
+ try {
+ while (true) {
+ try {
+ URLConnection connection = openConnectionFollowingRedirects(url, timeout);
+ checkHttpStatus(connection);
+ try (InputStream inputStream = connection.getInputStream()) {
+ // Files.copy loops until end of stream. FileChannel.transferFrom(), used
+ // here previously, is not guaranteed to drain a socket-backed channel in
+ // a single call and could silently truncate a download.
+ Files.copy(inputStream, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ }
+ break;
+ } catch (SocketTimeoutException e) {
+ if (++count == maxTries) throw e;
}
- if (fos != null) {
- fos.close();
+ }
+
+ logger.debug("Copying temp file [{}] to final location [{}]", tempFile, destination);
+ Files.copy(tempFile.toPath(), destination.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ } finally {
+ // on every path, including failure: the temp file used to leak whenever
+ // the download threw.
+ deleteQuietly(tempFile);
+ }
+ }
+
+ /**
+ * Downloads a file and writes its validation metadata in a single pass over a
+ * single connection.
+ *
+ * This is preferable to calling {@link #createValidationFiles(URL, File, URL, Hash)}
+ * followed by {@link #downloadFile(URL, File)}: those open separate connections,
+ * so the Content-Length recorded in the .size file
+ * comes from a different response than the bytes actually written. If the
+ * resource changes between the two requests, the cache entry is left
+ * permanently failing validation.
+ *
+ * The content is streamed to a temporary file and only moved into place once
+ * the declared length and (where available) the checksum have been confirmed,
+ * so a failed download never leaves a partial file at destination.
+ *
+ * @param url the remote file to download
+ * @param destination the local file to download into. Its parent directory must exist.
+ * @param hashURL URL of a file containing the expected hash. May be null.
+ * @param hash the hashing algorithm matching hashURL. Ignored when
+ * hashURL is null.
+ * @param eTagPolicy what to do with an ETag response header when
+ * hashURL is null. May be null,
+ * which is treated as {@link ETagPolicy#IGNORE}.
+ * @throws HttpStatusException if the server answered with a non-2xx status
+ * @throws IOException if the transfer failed, or the transferred content did not
+ * match the length or checksum the server declared
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static void downloadFileWithValidation(URL url, File destination, URL hashURL, Hash hash,
+ ETagPolicy eTagPolicy) throws IOException {
+ int timeout = 60000; //60 sec
+ ETagPolicy policy = eTagPolicy == null ? ETagPolicy.IGNORE : eTagPolicy;
+
+ File tempFile = createTempFileFor(destination);
+ try {
+ URLConnection connection = openConnectionFollowingRedirects(url, timeout);
+ checkHttpStatus(connection);
+
+ long declaredSize = connection.getContentLengthLong();
+ String eTag = connection.getHeaderField("ETag");
+
+ // Only digest when we have something to compare against; hashing every
+ // download would cost CPU for no benefit.
+ Hash eTagHash = policy == ETagPolicy.IGNORE ? Hash.UNKNOWN : hashFromETag(eTag);
+ if (policy == ETagPolicy.REQUIRE && eTagHash == Hash.UNKNOWN) {
+ logger.warn("ETag [{}] of {} is not a usable hex digest; no checksum will be recorded.", eTag, url);
+ }
+
+ MessageDigest digest = eTagHash == Hash.UNKNOWN ? null : newDigest(eTagHash);
+ long written;
+ try (InputStream raw = connection.getInputStream();
+ InputStream in = digest == null ? raw : new DigestInputStream(raw, digest)) {
+ written = Files.copy(in, tempFile.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ }
+
+ if (declaredSize >= 0 && written != declaredSize) {
+ throw new IOException(String.format(
+ "Incomplete download of %s: got %d bytes but the server declared %d.",
+ url, written, declaredSize));
+ }
+
+ String actualDigest = digest == null ? null : toHex(digest.digest());
+ if (actualDigest != null && !actualDigest.equalsIgnoreCase(normalizeETag(eTag))) {
+ throw new IOException(String.format(
+ "Corrupt download of %s: %s of the content is %s but the server's ETag says %s.",
+ url, eTagHash, actualDigest, normalizeETag(eTag)));
+ }
+
+ moveIntoPlace(tempFile, destination);
+
+ // Sidecars are written only once the content is known good, so a failed
+ // download can never leave validation metadata describing a file that is
+ // not there.
+ writeSizeFile(destination, written);
+ if (hashURL != null) {
+ if (hash == null || hash == Hash.UNKNOWN) {
+ throw new IllegalArgumentException("Hash URL given but algorithm is unknown");
}
+ downloadFile(hashURL, hashFileFor(destination, hash));
+ } else if (actualDigest != null) {
+ writeHashFile(destination, eTagHash, actualDigest);
+ }
+ } finally {
+ deleteQuietly(tempFile);
+ }
+ }
+
+ /**
+ * Opens a connection, following any redirect that {@link HttpURLConnection}
+ * declines to follow itself.
+ *
+ * The JDK follows 301, 302 and 303 within a protocol, but it never follows 307 or + * 308, and it never follows a redirect that changes http to https. Both gaps have + * broken downloads in practice: CATH began answering http with a 301 to https, and + * ECOD now answers with a 308 to a rewritten path. A browser follows either without + * comment, so a service making that change has no reason to expect it to break us. + *
+ * A redirect from https to http is deliberately not followed: a redirect
+ * must never silently downgrade the transport. Such a response is returned as it is,
+ * for {@link #checkHttpStatus(URLConnection)} to reject.
+ *
+ * @param url the URL to open
+ * @param timeout connect and read timeout, in milliseconds
+ * @return a connected {@link URLConnection} at the final location
+ * @throws HttpStatusException if the redirects loop or exceed the limit
+ * @throws IOException if the connection could not be opened
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static URLConnection openConnectionFollowingRedirects(URL url, int timeout) throws IOException {
+ Set
+ * Without this check a 404 error page is written into the cache as though it
+ * were the requested file — and because the
+ * Nothing is written when the connection reports a non-2xx status: previously an
+ * error page's
+ * Only a value consisting solely of hex characters is considered. The
+ *
+ * The file may have been downloaded verbatim from a server, so several common
+ * layouts are accepted: a bare hex digest, the
+ * This exists so that callers can tell apart the two very different reasons a
+ * download can fail:
+ *
+ * The JDK handles 301, 302 and 303 within a protocol, but never 307 or 308, and never
+ * a redirect that changes http to https. Both gaps have broken this project's builds:
+ * CATH began answering http with a 301 to https, and ECOD now answers with a 308 to a
+ * rewritten path. A browser follows either without comment.
+ *
+ * The rule tests need no network and no server. The end-to-end tests use a local
+ * {@link HttpServer} rather than a real service, so that they cannot fail because a
+ * third party is having a bad day.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+class FileDownloadRedirectTest {
+
+ private static final String PAYLOAD = "the file you were looking for\n";
+
+ @Nested
+ class RedirectRules {
+
+ private final URL from = url("http://example.org/ecod/distributions/ecod.latest.domains.txt");
+
+ @Test
+ void aRelativeLocationIsResolvedAgainstTheRequest() {
+ // exactly what ECOD sends: same host, same protocol, relative path
+ assertEquals(url("http://example.org/ecod-legacy/distributions/ecod.latest.domains.txt"),
+ FileDownloadUtils.redirectTargetFor(308,
+ "/ecod-legacy/distributions/ecod.latest.domains.txt", from));
+ }
+
+ @Test
+ void anAbsoluteLocationIsUsedAsGiven() {
+ assertEquals(url("https://example.org/elsewhere.txt"),
+ FileDownloadUtils.redirectTargetFor(301, "https://example.org/elsewhere.txt", from));
+ }
+
+ @Test
+ void everyRedirectStatusWeHandleIsRecognised() {
+ for (int code : new int[] { 301, 302, 303, 307, 308 }) {
+ assertEquals(url("http://example.org/x"),
+ FileDownloadUtils.redirectTargetFor(code, "/x", from),
+ "status " + code + " should be followed");
+ }
+ }
+
+ @Test
+ void aSuccessIsNotARedirect() {
+ assertNull(FileDownloadUtils.redirectTargetFor(200, null, from));
+ assertNull(FileDownloadUtils.redirectTargetFor(404, "/x", from));
+ }
+
+ /**
+ * A redirect must never quietly move us onto an unencrypted transport.
+ */
+ @Test
+ void httpsIsNeverDowngradedToHttp() {
+ URL secure = url("https://example.org/file.txt");
+ assertNull(FileDownloadUtils.redirectTargetFor(301, "http://example.org/file.txt", secure));
+ assertNull(FileDownloadUtils.redirectTargetFor(308, "http://elsewhere.org/file.txt", secure));
+ }
+
+ @Test
+ void httpToHttpsIsFollowed() {
+ // the CATH case
+ assertEquals(url("https://example.org/file.txt"),
+ FileDownloadUtils.redirectTargetFor(301, "https://example.org/file.txt",
+ url("http://example.org/file.txt")));
+ }
+
+ @Test
+ void anUnusableLocationIsNotFollowed() {
+ assertNull(FileDownloadUtils.redirectTargetFor(308, null, from));
+ assertNull(FileDownloadUtils.redirectTargetFor(308, " ", from));
+ assertNull(FileDownloadUtils.redirectTargetFor(308, "gopher://example.org/x", from));
+ }
+ }
+
+ @Nested
+ class EndToEnd {
+
+ private HttpServer server;
+ private String base;
+ private File dir;
+
+ @BeforeEach
+ void start() throws IOException {
+ server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+ base = "http://127.0.0.1:" + server.getAddress().getPort();
+ dir = Files.createTempDirectory("redirectTest").toFile();
+
+ serve("/final", 200, null);
+ // the ECOD shape: 308 with a relative Location
+ serve("/moved", 308, "/final");
+ serve("/temp", 307, "/final");
+ // a chain that returns to its start
+ serve("/loop-a", 308, "/loop-b");
+ serve("/loop-b", 308, "/loop-a");
+ // longer than the hop limit
+ for (int i = 0; i < 9; i++) {
+ serve("/hop" + i, 308, "/hop" + (i + 1));
+ }
+ serve("/hop9", 200, null);
+ serve("/nowhere", 308, null);
+ server.start();
+ }
+
+ private void serve(String path, int status, String location) {
+ server.createContext(path, exchange -> {
+ byte[] body = PAYLOAD.getBytes(StandardCharsets.UTF_8);
+ if (location != null) {
+ exchange.getResponseHeaders().add("Location", location);
+ }
+ exchange.sendResponseHeaders(status, status == 200 ? body.length : -1);
+ if (status == 200) {
+ try (OutputStream out = exchange.getResponseBody()) {
+ out.write(body);
+ }
+ }
+ exchange.close();
+ });
+ }
+
+ @AfterEach
+ void stop() throws IOException {
+ server.stop(0);
+ FileDownloadUtils.deleteDirectory(dir.getAbsolutePath());
+ }
+
+ @Test
+ void a308IsFollowed() throws IOException {
+ File got = new File(dir, "moved.txt");
+ FileDownloadUtils.downloadFile(new URL(base + "/moved"), got);
+ assertEquals(PAYLOAD, new String(Files.readAllBytes(got.toPath()), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void a307IsFollowed() throws IOException {
+ File got = new File(dir, "temp.txt");
+ FileDownloadUtils.downloadFile(new URL(base + "/temp"), got);
+ assertEquals(PAYLOAD, new String(Files.readAllBytes(got.toPath()), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ void theRedirectBodyIsNeverWhatWeStore() throws IOException {
+ File got = new File(dir, "validated.txt");
+ FileDownloadUtils.downloadFileWithValidation(new URL(base + "/moved"), got, null,
+ FileDownloadUtils.Hash.UNKNOWN, FileDownloadUtils.ETagPolicy.IGNORE);
+ assertEquals(PAYLOAD, new String(Files.readAllBytes(got.toPath()), StandardCharsets.UTF_8));
+ assertTrue(FileDownloadUtils.validateFile(got), "the recorded size must describe the real file");
+ }
+
+ @Test
+ void aLoopIsReportedRatherThanChasedForever() {
+ File got = new File(dir, "loop.txt");
+ HttpStatusException e = assertThrows(HttpStatusException.class,
+ () -> FileDownloadUtils.downloadFile(new URL(base + "/loop-a"), got));
+ assertTrue(e.getMessage().contains("loop"), e.getMessage());
+ }
+
+ @Test
+ void tooManyHopsGivesUp() {
+ File got = new File(dir, "hops.txt");
+ assertThrows(HttpStatusException.class,
+ () -> FileDownloadUtils.downloadFile(new URL(base + "/hop0"), got));
+ }
+
+ @Test
+ void aRedirectWithNoDestinationIsAnError() {
+ File got = new File(dir, "nowhere.txt");
+ assertThrows(HttpStatusException.class,
+ () -> FileDownloadUtils.downloadFile(new URL(base + "/nowhere"), got));
+ }
+ }
+
+ private static URL url(String spec) {
+ try {
+ return new URL(spec);
+ } catch (IOException e) {
+ throw new IllegalArgumentException(spec, e);
+ }
+ }
+}
diff --git a/biojava-core/src/test/java/org/biojava/nbio/core/util/FileDownloadUtilsTest.java b/biojava-core/src/test/java/org/biojava/nbio/core/util/FileDownloadUtilsTest.java
index 201ad88e48..bec6def90f 100644
--- a/biojava-core/src/test/java/org/biojava/nbio/core/util/FileDownloadUtilsTest.java
+++ b/biojava-core/src/test/java/org/biojava/nbio/core/util/FileDownloadUtilsTest.java
@@ -4,6 +4,7 @@
import static org.biojava.nbio.core.util.FileDownloadUtils.getFilePrefix;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -11,6 +12,7 @@
import java.io.IOException;
import java.io.PrintStream;
import java.net.URL;
+import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import org.junit.jupiter.api.Nested;
@@ -187,17 +189,268 @@ void testValidationFiles() throws IOException{
assertTrue(sizeFile.exists(), "couldn't create size file");
assertTrue(FileDownloadUtils.validateFile(destFile), "file not detected to be invalid although there is correct size validation file");
+ // files.wwpdb.org returns the content MD5 as the ETag, so the default
+ // ETag policy records a real checksum without a separate hash URL.
+ assertTrue(hashFile.exists(), "no hash file was derived from the ETag");
+ assertTrue(FileDownloadUtils.validateFile(destFile), "correctly downloaded file failed hash validation");
+
PrintStream temp2 = new PrintStream(hashFile);
- temp2.print("ABCD"); // some wrong hash value
+ temp2.print("ABCD"); // not a digest of any supported length
temp2.close();
- //This is not yet implemented. I am using this test for documentation purpose.
- assertThrows(UnsupportedOperationException.class,
- () -> FileDownloadUtils.validateFile(destFile),
+ // An unreadable sidecar must not condemn an otherwise good download.
+ assertTrue(FileDownloadUtils.validateFile(destFile),
+ "a malformed hash file should be ignored, not treated as a mismatch");
+
+ PrintStream temp3 = new PrintStream(hashFile);
+ temp3.print("00000000000000000000000000000000"); // well-formed but wrong MD5
+ temp3.close();
+ assertFalse(FileDownloadUtils.validateFile(destFile),
"file not detected to be invalid although hash value is wrong.");
-
+ System.out.println("Just ignore the previous warning. It is expected.");
+
destFile.delete();
sizeFile.delete();
hashFile.delete();
}
}
+
+ @Nested
+ class HttpStatus {
+
+ /**
+ * Which status an absent file comes back with is the server's business, and it
+ * changes: files.wwpdb.org moved behind Amazon S3 in September 2026, and S3
+ * answers a missing key with 403 rather than 404 when the caller cannot list
+ * the bucket. Pinning the code made this test fail on an upstream hosting
+ * change that broke nothing.
+ *
+ * What must hold is the contract: an error status throws, and nothing is left
+ * on disk for it. {@link HttpStatusException#isNotFound()} is pinned separately
+ * below, without a network.
+ */
+ @Test
+ void anAbsentFileThrowsAndLeavesNothingBehind() throws IOException {
+ URL missing = new URL("https://files.wwpdb.org/pub/pdb/data/structures/divided/mmCIF/zz/zzzz.cif.gz");
+ File dest = new File(System.getProperty("java.io.tmpdir"), "bj-missing.cif.gz");
+ File sizeFile = new File(dest.getParentFile(), dest.getName() + ".size");
+ dest.delete();
+ sizeFile.delete();
+
+ HttpStatusException e = assertThrows(HttpStatusException.class,
+ () -> FileDownloadUtils.downloadFile(missing, dest));
+ assertTrue(e.getStatusCode() >= 400,
+ "an absent file must report an error status, got " + e.getStatusCode());
+ assertFalse(dest.exists(), "an error body must never be written to the destination");
+
+ // ... and no validation metadata may be recorded for it either, or the
+ // cached error page would later pass validation.
+ FileDownloadUtils.createValidationFiles(missing, dest, null, FileDownloadUtils.Hash.UNKNOWN);
+ assertFalse(sizeFile.exists(), "no size file should be written for an error response");
+ }
+
+ @Test
+ void isNotFoundCoversTheAbsentStatusesOnly() {
+ assertTrue(new HttpStatusException(404, "http://example.org/x", "Not Found").isNotFound());
+ assertTrue(new HttpStatusException(410, "http://example.org/x", "Gone").isNotFound());
+ // 403 is what an S3-backed archive returns for a missing key, but it is not a
+ // statement that the file does not exist, so it must not claim to be one
+ assertFalse(new HttpStatusException(403, "http://example.org/x", "Forbidden").isNotFound());
+ assertFalse(new HttpStatusException(500, "http://example.org/x", "Server Error").isNotFound());
+ }
+ }
+
+ @Nested
+ class Hashing {
+
+ private File writeTemp(String name, byte[] content) throws IOException {
+ File f = new File(System.getProperty("java.io.tmpdir"), name);
+ Files.write(f.toPath(), content);
+ f.deleteOnExit();
+ return f;
+ }
+
+ @Test
+ void digestsOfEmptyFileMatchKnownValues() throws IOException {
+ File empty = writeTemp("bj-empty.bin", new byte[0]);
+ assertEquals("d41d8cd98f00b204e9800998ecf8427e",
+ FileDownloadUtils.computeHash(empty, FileDownloadUtils.Hash.MD5));
+ assertEquals("da39a3ee5e6b4b0d3255bfef95601890afd80709",
+ FileDownloadUtils.computeHash(empty, FileDownloadUtils.Hash.SHA1));
+ assertEquals("e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
+ FileDownloadUtils.computeHash(empty, FileDownloadUtils.Hash.SHA256));
+ }
+
+ @Test
+ void digestOfKnownContent() throws IOException {
+ File abc = writeTemp("bj-abc.bin", "abc".getBytes(StandardCharsets.UTF_8));
+ assertEquals("900150983cd24fb0d6963f7d28e17f72",
+ FileDownloadUtils.computeHash(abc, FileDownloadUtils.Hash.MD5));
+ assertTrue(FileDownloadUtils.verifyHash(abc, FileDownloadUtils.Hash.MD5,
+ "900150983CD24FB0D6963F7D28E17F72"), "comparison should be case-insensitive");
+ assertFalse(FileDownloadUtils.verifyHash(abc, FileDownloadUtils.Hash.MD5,
+ "00000000000000000000000000000000"));
+ }
+
+ @Test
+ void algorithmNames() {
+ assertEquals("MD5", FileDownloadUtils.getAlgorithmName(FileDownloadUtils.Hash.MD5));
+ assertEquals("SHA-1", FileDownloadUtils.getAlgorithmName(FileDownloadUtils.Hash.SHA1));
+ assertEquals("SHA-256", FileDownloadUtils.getAlgorithmName(FileDownloadUtils.Hash.SHA256));
+ assertThrows(IllegalArgumentException.class,
+ () -> FileDownloadUtils.getAlgorithmName(FileDownloadUtils.Hash.UNKNOWN));
+ }
+ }
+
+ @Nested
+ class HashFileParsing {
+
+ private static final String MD5 = "900150983cd24fb0d6963f7d28e17f72";
+
+ private String parse(String content) throws IOException {
+ File f = new File(System.getProperty("java.io.tmpdir"), "bj-hashfile.txt");
+ Files.write(f.toPath(), content.getBytes(StandardCharsets.UTF_8));
+ f.deleteOnExit();
+ return FileDownloadUtils.parseHashFile(f);
+ }
+
+ @Test
+ void bareHex() throws IOException {
+ assertEquals(MD5, parse(MD5));
+ assertEquals(MD5, parse(MD5 + "\n"));
+ }
+
+ @Test
+ void uppercaseHexIsKeptVerbatim() throws IOException {
+ assertEquals(MD5.toUpperCase(), parse(MD5.toUpperCase()));
+ }
+
+ @Test
+ void coreutilsLayouts() throws IOException {
+ assertEquals(MD5, parse(MD5 + " somefile.cif.gz\n"));
+ assertEquals(MD5, parse(MD5 + " *somefile.cif.gz\n"));
+ }
+
+ @Test
+ void bsdLayout() throws IOException {
+ assertEquals(MD5, parse("MD5 (somefile.cif.gz) = " + MD5 + "\n"));
+ }
+
+ @Test
+ void blankLeadingLinesAreSkipped() throws IOException {
+ assertEquals(MD5, parse("\n \n" + MD5 + "\n"));
+ }
+
+ @Test
+ void garbageYieldsNull() throws IOException {
+ assertNull(parse("not a hash at all\n"));
+ assertNull(parse("ABCD\n"));
+ assertNull(parse(""));
+ }
+ }
+
+ @Nested
+ class ETagParsing {
+
+ @Test
+ void wwpdbStyleMd5IsRecognised() {
+ assertEquals(FileDownloadUtils.Hash.MD5,
+ FileDownloadUtils.hashFromETag("\"f99fb9d964e1e1c22f2ea559ac5745cf\""));
+ assertEquals("f99fb9d964e1e1c22f2ea559ac5745cf",
+ FileDownloadUtils.normalizeETag("\"f99fb9d964e1e1c22f2ea559ac5745cf\""));
+ }
+
+ @Test
+ void sha1AndSha256LengthsAreRecognised() {
+ assertEquals(FileDownloadUtils.Hash.SHA1,
+ FileDownloadUtils.hashFromETag("da39a3ee5e6b4b0d3255bfef95601890afd80709"));
+ assertEquals(FileDownloadUtils.Hash.SHA256,
+ FileDownloadUtils.hashFromETag(
+ "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"));
+ }
+
+ @Test
+ void ebiStyleTimeSizeETagIsNotMistakenForADigest() {
+ // nginx and Apache emit file:, ftp:, ...) are left alone.
+ * .size sidecar is
+ * then taken from that same error response, {@link #validateFile(File)} would
+ * subsequently declare it valid.
+ *
+ * @param connection an already connected {@link URLConnection}
+ * @throws HttpStatusException if the status is outside the 2xx range
+ * @throws IOException if the status could not be read
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static void checkHttpStatus(URLConnection connection) throws IOException {
+ if (!(connection instanceof HttpURLConnection)) {
+ return;
+ }
+ HttpURLConnection http = (HttpURLConnection) connection;
+ int code = http.getResponseCode();
+ if (code >= 200 && code < 300) {
+ return;
+ }
+ if (code == 301 || code == 302 || code == 307 || code == 308) {
+ // openConnectionFollowingRedirects handles the redirects the JDK will not,
+ // so one reaching here was declined deliberately: an https to http
+ // downgrade, a missing or unusable Location, or too many hops.
+ logger.warn("{} returned redirect {} to [{}], which was not followed.",
+ connection.getURL(), code, http.getHeaderField("Location"));
+ }
+ throw new HttpStatusException(code, connection.getURL().toString(), http.getResponseMessage());
}
-
+
/**
* Creates validation files beside a file to be downloaded.
* Whenever possible, for a file.ext file, it creates
@@ -146,9 +408,27 @@ public static void downloadFile(URL url, File destination) throws IOException {
* @param hash The Hashing algorithm. Ignored if hashURL is null.
*/
public static void createValidationFiles(URL url, File localDestination, URL hashURL, Hash hash){
+ createValidationFiles(url, localDestination, hashURL, hash, ETagPolicy.USE_IF_HEX_DIGEST);
+ }
+
+ /**
+ * Creates validation files beside a file to be downloaded, with explicit control
+ * over how the ETag response header is treated.
+ *
+ * @param url the remote file URL to download
+ * @param localDestination the local file to download into
+ * @param hashURL the URL of the hash file to download. Can be null.
+ * @param hash The Hashing algorithm. Ignored if hashURL is null.
+ * @param eTagPolicy how to treat the ETag header when hashURL
+ * is null. May be null, treated as {@link ETagPolicy#IGNORE}.
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static void createValidationFiles(URL url, File localDestination, URL hashURL, Hash hash,
+ ETagPolicy eTagPolicy){
try {
- URLConnection resourceConnection = url.openConnection();
- createValidationFiles(resourceConnection, localDestination, hashURL, FileDownloadUtils.Hash.UNKNOWN);
+ URLConnection resourceConnection = openConnectionFollowingRedirects(url, 60000);
+ createValidationFiles(resourceConnection, localDestination, hashURL, hash, eTagPolicy);
} catch (IOException e) {
logger.warn("could not open connection to resource file due to exception: {}", e.getMessage());
}
@@ -169,31 +449,246 @@ public static void createValidationFiles(URL url, File localDestination, URL has
* @since 7.0.0
*/
public static void createValidationFiles(URLConnection resourceUrlConnection, File localDestination, URL hashURL, Hash hash){
+ createValidationFiles(resourceUrlConnection, localDestination, hashURL, hash, ETagPolicy.USE_IF_HEX_DIGEST);
+ }
+
+ /**
+ * Creates validation files beside a file to be downloaded, with explicit control
+ * over how the ETag response header is treated.
+ * Content-Length would be recorded as the expected
+ * size, so the cached error page then passed validation.
+ *
+ * @param resourceUrlConnection the remote file URLConnection to download
+ * @param localDestination the local file to download into
+ * @param hashURL the URL of the hash file to download. Can be null.
+ * @param hash The Hashing algorithm. Ignored if hashURL is null.
+ * @param eTagPolicy how to treat the ETag header when hashURL
+ * is null. May be null, treated as {@link ETagPolicy#IGNORE}.
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static void createValidationFiles(URLConnection resourceUrlConnection, File localDestination, URL hashURL,
+ Hash hash, ETagPolicy eTagPolicy){
+ try {
+ checkHttpStatus(resourceUrlConnection);
+ } catch (IOException e) {
+ logger.warn("Not writing validation metadata for {}: {}", resourceUrlConnection.getURL(), e.getMessage());
+ return;
+ }
+
long size = resourceUrlConnection.getContentLengthLong();
if(size == -1) {
logger.debug("Could not find expected file size for resource {}. Size validation metadata file won't be available for this download.", resourceUrlConnection.getURL());
} else {
logger.debug("Content-Length: {}", size);
- File sizeFile = new File(localDestination.getParentFile(), localDestination.getName() + SIZE_EXT);
- try (PrintStream sizePrintStream = new PrintStream(sizeFile)) {
- sizePrintStream.print(size);
- } catch (FileNotFoundException e) {
- logger.warn("Could not write size validation metadata file due to exception: {}", e.getMessage());
- }
+ writeSizeFile(localDestination, size);
}
-
- if(hashURL == null)
+
+ if(hashURL == null) {
+ ETagPolicy policy = eTagPolicy == null ? ETagPolicy.IGNORE : eTagPolicy;
+ if (policy != ETagPolicy.IGNORE) {
+ String eTag = resourceUrlConnection.getHeaderField("ETag");
+ Hash eTagHash = hashFromETag(eTag);
+ if (eTagHash == Hash.UNKNOWN) {
+ if (policy == ETagPolicy.REQUIRE) {
+ logger.warn("ETag [{}] of {} is not a usable hex digest; no checksum recorded.",
+ eTag, resourceUrlConnection.getURL());
+ }
+ } else {
+ writeHashFile(localDestination, eTagHash, normalizeETag(eTag));
+ }
+ }
return;
+ }
- if(hash == Hash.UNKNOWN)
+ if(hash == null || hash == Hash.UNKNOWN)
throw new IllegalArgumentException("Hash URL given but algorithm is unknown");
try {
- File hashFile = new File(localDestination.getParentFile(), String.format("%s%s_%s", localDestination.getName(), HASH_EXT, hash));
- downloadFile(hashURL, hashFile);
+ downloadFile(hashURL, hashFileFor(localDestination, hash));
+ } catch (IOException e) {
+ logger.warn("Could not write validation hash file due to exception: {}", e.getMessage());
+ }
+ }
+
+ /**
+ * Determines which hashing algorithm an ETag header value
+ * represents, based on the length of the hex digest it contains.
+ * <time>-<size> form used by nginx and Apache always
+ * contains a - and therefore never matches.
+ *
+ * @param eTagHeaderValue the raw header value, possibly quoted or weak-prefixed.
+ * May be null.
+ * @return the matching algorithm, or {@link Hash#UNKNOWN} if the value is not a
+ * hex digest of a recognised length
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static Hash hashFromETag(String eTagHeaderValue) {
+ String value = normalizeETag(eTagHeaderValue);
+ if (value == null || !value.matches("[0-9a-fA-F]+")) {
+ return Hash.UNKNOWN;
+ }
+ switch (value.length()) {
+ case 32: return Hash.MD5;
+ case 40: return Hash.SHA1;
+ case 64: return Hash.SHA256;
+ default: return Hash.UNKNOWN;
+ }
+ }
+
+ /**
+ * Strips the weak-validator prefix and surrounding quotes from an
+ * ETag header value.
+ *
+ * @param eTagHeaderValue the raw header value. May be null.
+ * @return the bare value, or null if the input was null
+ * or blank
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static String normalizeETag(String eTagHeaderValue) {
+ if (eTagHeaderValue == null) {
+ return null;
+ }
+ String value = eTagHeaderValue.trim();
+ if (value.startsWith("W/")) {
+ value = value.substring(2).trim();
+ }
+ if (value.length() >= 2 && value.startsWith("\"") && value.endsWith("\"")) {
+ value = value.substring(1, value.length() - 1);
+ }
+ return value.isEmpty() ? null : value;
+ }
+
+ /**
+ * Writes a <name>.hash_<ALGORITHM> sidecar file
+ * containing the given digest as bare lowercase hex.
+ *
+ * @param localDestination the file the digest describes
+ * @param hash the hashing algorithm
+ * @param hexDigest the digest, in hex
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static void writeHashFile(File localDestination, Hash hash, String hexDigest) {
+ if (hash == null || hash == Hash.UNKNOWN || hexDigest == null) {
+ return;
+ }
+ File hashFile = hashFileFor(localDestination, hash);
+ try (PrintStream out = new PrintStream(hashFile, StandardCharsets.UTF_8.name())) {
+ out.println(hexDigest.toLowerCase());
} catch (IOException e) {
logger.warn("Could not write validation hash file due to exception: {}", e.getMessage());
}
}
+
+ /**
+ * Computes the digest of a file.
+ *
+ * @param file the file to digest
+ * @param hash the algorithm to use
+ * @return the digest as bare lowercase hex
+ * @throws IOException if the file could not be read
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static String computeHash(File file, Hash hash) throws IOException {
+ try (InputStream in = new BufferedInputStream(Files.newInputStream(file.toPath()), DIGEST_BUFFER_SIZE)) {
+ return computeHash(in, hash);
+ }
+ }
+
+ /**
+ * Computes the digest of a stream. The stream is read to its end but not closed.
+ *
+ * @param in the stream to digest
+ * @param hash the algorithm to use
+ * @return the digest as bare lowercase hex
+ * @throws IOException if the stream could not be read
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static String computeHash(InputStream in, Hash hash) throws IOException {
+ MessageDigest digest = newDigest(hash);
+ byte[] buffer = new byte[DIGEST_BUFFER_SIZE];
+ int read;
+ while ((read = in.read(buffer)) != -1) {
+ digest.update(buffer, 0, read);
+ }
+ return toHex(digest.digest());
+ }
+
+ /**
+ * Checks a file against an expected digest.
+ *
+ * @param file the file to check
+ * @param hash the algorithm to use
+ * @param expectedHex the expected digest, in hex; compared case-insensitively
+ * @return true if the digests match
+ * @throws IOException if the file could not be read
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static boolean verifyHash(File file, Hash hash, String expectedHex) throws IOException {
+ return expectedHex != null && expectedHex.trim().equalsIgnoreCase(computeHash(file, hash));
+ }
+
+ /**
+ * The JDK name of a hashing algorithm, which differs from the enum constant for
+ * the SHA variants.
+ *
+ * @param hash the algorithm
+ * @return the name to pass to {@link MessageDigest#getInstance(String)}
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static String getAlgorithmName(Hash hash) {
+ switch (hash) {
+ case MD5: return "MD5";
+ case SHA1: return "SHA-1";
+ case SHA256: return "SHA-256";
+ default: throw new IllegalArgumentException("Hashing algorithm not known: " + hash);
+ }
+ }
+
+ /**
+ * Reads the expected digest out of a .hash_XXXX sidecar file.
+ * md5sum style
+ * <hex> <filename>, and the BSD style
+ * MD5 (<filename>) = <hex>.
+ *
+ * @param hashFile the sidecar file
+ * @return the digest in hex, or null if nothing recognisable was found
+ */
+ static String parseHashFile(File hashFile) {
+ try (Scanner scanner = new Scanner(hashFile, StandardCharsets.UTF_8.name())) {
+ while (scanner.hasNextLine()) {
+ String line = scanner.nextLine().trim();
+ if (line.isEmpty()) {
+ continue;
+ }
+ Matcher bare = BARE_HEX_HASH.matcher(line);
+ if (bare.matches()) {
+ return bare.group(1);
+ }
+ Matcher bsd = BSD_HASH.matcher(line);
+ if (bsd.matches()) {
+ return bsd.group(1);
+ }
+ return null; // first meaningful line was not a digest
+ }
+ } catch (IOException e) {
+ logger.warn("Could not read hash file [{}]: {}", hashFile, e.getMessage());
+ }
+ return null;
+ }
+
/**
* Validate a local file based on pre-existing metadata files for size and hash.
@@ -210,45 +705,154 @@ public static void createValidationFiles(URLConnection resourceUrlConnection, Fi
* @since 7.0.0
*/
public static boolean validateFile(File localFile) {
- File sizeFile = new File(localFile.getParentFile(), localFile.getName() + SIZE_EXT);
+ // getParentFile() is null for a bare relative name such as new File("x.cif"),
+ // which used to make this method throw a NullPointerException.
+ File parent = localFile.getAbsoluteFile().getParentFile();
+ if (parent == null) {
+ logger.debug("Cannot determine the parent directory of [{}]; nothing to validate against.", localFile);
+ return true;
+ }
+
+ File sizeFile = new File(parent, localFile.getName() + SIZE_EXT);
if(sizeFile.exists()) {
try (Scanner scanner = new Scanner(sizeFile)) {
- long expectedSize = scanner.nextLong();
- long actualSize = localFile.length();
- if (expectedSize != actualSize) {
- logger.warn("File [{}] size ({}) does not match expected size ({}).", localFile, actualSize, expectedSize);
- return false;
+ if (!scanner.hasNextLong()) {
+ // An empty or truncated .size file used to raise an unchecked
+ // NoSuchElementException that escaped the catch below.
+ logger.warn("Size metadata file [{}] is empty or malformed; skipping size validation.", sizeFile);
+ } else {
+ long expectedSize = scanner.nextLong();
+ long actualSize = localFile.length();
+ if (expectedSize != actualSize) {
+ logger.warn("File [{}] size ({}) does not match expected size ({}).", localFile, actualSize, expectedSize);
+ return false;
+ }
}
} catch (FileNotFoundException e) {
logger.warn("could not validate size of file [{}] because no size metadata file exists.", localFile);
}
}
- File[] hashFiles = localFile.getParentFile().listFiles(new FilenameFilter() {
- final String hashPattern = String.format("%s%s_(%s|%s|%s)", localFile.getName(), HASH_EXT, Hash.MD5, Hash.SHA1, Hash.SHA256);
+ File[] hashFiles = parent.listFiles(new FilenameFilter() {
+ final String hashPattern = String.format("%s%s_(%s|%s|%s)", Pattern.quote(localFile.getName()), HASH_EXT, Hash.MD5, Hash.SHA1, Hash.SHA256);
@Override
public boolean accept(File dir, String name) {
return name.matches(hashPattern);
}
});
- if(hashFiles.length > 0) {
- File hashFile = hashFiles[0];
+ // listFiles() returns null if the parent is not a directory or cannot be read.
+ if (hashFiles == null || hashFiles.length == 0) {
+ return true;
+ }
+
+ // Verify against every sidecar present, not only the first one found.
+ for (File hashFile : hashFiles) {
String name = hashFile.getName();
String algo = name.substring(name.lastIndexOf('_') + 1);
- switch (Hash.valueOf(algo)) {
- case MD5:
- case SHA1:
- case SHA256:
- throw new UnsupportedOperationException("Not yet implemented");
- case UNKNOWN:
- default: // No need. Already checked above
+ Hash hash;
+ try {
+ hash = Hash.valueOf(algo);
+ } catch (IllegalArgumentException e) {
+ throw new IllegalArgumentException("Hashing algorithm not known: " + algo, e);
+ }
+ if (hash == Hash.UNKNOWN) {
throw new IllegalArgumentException("Hashing algorithm not known: " + algo);
}
+
+ String expected = parseHashFile(hashFile);
+ if (expected == null) {
+ // A sidecar we cannot read should not condemn an otherwise good download.
+ logger.warn("Could not read a digest from [{}]; skipping {} validation of [{}].", hashFile, hash, localFile);
+ continue;
+ }
+ try {
+ if (!verifyHash(localFile, hash, expected)) {
+ logger.warn("File [{}] {} does not match the expected digest {}.", localFile, hash, expected);
+ return false;
+ }
+ } catch (IOException e) {
+ logger.warn("Could not compute the {} of [{}]: {}", hash, localFile, e.getMessage());
+ return false;
+ }
}
-
+
return true;
}
+ /**
+ * The <name>.hash_<ALGORITHM> sidecar file for a
+ * downloaded file.
+ */
+ private static File hashFileFor(File localDestination, Hash hash) {
+ return new File(localDestination.getAbsoluteFile().getParentFile(),
+ String.format("%s%s_%s", localDestination.getName(), HASH_EXT, hash));
+ }
+
+ /**
+ * Writes the <name>.size sidecar file.
+ */
+ private static void writeSizeFile(File localDestination, long size) {
+ File sizeFile = new File(localDestination.getAbsoluteFile().getParentFile(),
+ localDestination.getName() + SIZE_EXT);
+ try (PrintStream sizePrintStream = new PrintStream(sizeFile, StandardCharsets.UTF_8.name())) {
+ sizePrintStream.print(size);
+ } catch (IOException e) {
+ logger.warn("Could not write size validation metadata file due to exception: {}", e.getMessage());
+ }
+ }
+
+ private static MessageDigest newDigest(Hash hash) {
+ try {
+ return MessageDigest.getInstance(getAlgorithmName(hash));
+ } catch (NoSuchAlgorithmException e) {
+ // MD5, SHA-1 and SHA-256 are required of every Java platform.
+ throw new IllegalStateException("Required hashing algorithm is unavailable: " + hash, e);
+ }
+ }
+
+ private static String toHex(byte[] bytes) {
+ StringBuilder sb = new StringBuilder(bytes.length * 2);
+ for (byte b : bytes) {
+ sb.append(Character.forDigit((b >> 4) & 0xF, 16));
+ sb.append(Character.forDigit(b & 0xF, 16));
+ }
+ return sb.toString();
+ }
+
+ /**
+ * Creates a temp file whose name is derived from the destination.
+ * {@link Files#createTempFile} rejects prefixes shorter than 3 characters, so
+ * short names are padded.
+ */
+ private static File createTempFileFor(File destination) throws IOException {
+ String prefix = getFilePrefix(destination);
+ while (prefix.length() < 3) {
+ prefix = prefix + "_";
+ }
+ return Files.createTempFile(prefix, "." + getFileExtension(destination)).toFile();
+ }
+
+ private static void moveIntoPlace(File tempFile, File destination) throws IOException {
+ try {
+ Files.move(tempFile.toPath(), destination.toPath(),
+ StandardCopyOption.REPLACE_EXISTING, StandardCopyOption.ATOMIC_MOVE);
+ } catch (AtomicMoveNotSupportedException e) {
+ // The temp directory is often on a different filesystem than the cache.
+ Files.copy(tempFile.toPath(), destination.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ }
+ }
+
+ private static void deleteQuietly(File file) {
+ if (file == null) {
+ return;
+ }
+ try {
+ Files.deleteIfExists(file.toPath());
+ } catch (IOException e) {
+ logger.debug("Could not delete temporary file [{}]: {}", file, e.getMessage());
+ }
+ }
+
/**
* Converts path to Unix convention and adds a terminating slash if it was
* omitted.
diff --git a/biojava-core/src/main/java/org/biojava/nbio/core/util/HttpStatusException.java b/biojava-core/src/main/java/org/biojava/nbio/core/util/HttpStatusException.java
new file mode 100644
index 0000000000..c74c6ffb05
--- /dev/null
+++ b/biojava-core/src/main/java/org/biojava/nbio/core/util/HttpStatusException.java
@@ -0,0 +1,84 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.core.util;
+
+import java.io.IOException;
+
+/**
+ * Signals that an HTTP request completed but returned a status code outside the
+ * 2xx range.
+ *
+ *
+ * Without a distinct exception type the only way to tell these apart is by
+ * parsing the message of a plain {@link IOException}, which is brittle.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class HttpStatusException extends IOException {
+
+ private static final long serialVersionUID = 1L;
+
+ private final int statusCode;
+ private final String url;
+
+ /**
+ * @param statusCode the HTTP status code returned by the server
+ * @param url the URL that was requested
+ * @param responseMessage the HTTP reason phrase, may be null
+ */
+ public HttpStatusException(int statusCode, String url, String responseMessage) {
+ super(String.format("HTTP %d%s for %s", statusCode,
+ responseMessage == null || responseMessage.isEmpty() ? "" : " " + responseMessage, url));
+ this.statusCode = statusCode;
+ this.url = url;
+ }
+
+ /**
+ * @return the HTTP status code returned by the server
+ */
+ public int getStatusCode() {
+ return statusCode;
+ }
+
+ /**
+ * @return the URL that was requested
+ */
+ public String getUrl() {
+ return url;
+ }
+
+ /**
+ * Whether the status indicates that the resource is simply not there, as
+ * opposed to a transport or server problem.
+ *
+ * @return true for HTTP 404 (Not Found) and 410 (Gone)
+ */
+ public boolean isNotFound() {
+ return statusCode == 404 || statusCode == 410;
+ }
+}
diff --git a/biojava-core/src/main/java/org/biojava/nbio/core/util/PrettyXMLWriter.java b/biojava-core/src/main/java/org/biojava/nbio/core/util/PrettyXMLWriter.java
index 437085866f..6e4a7db77c 100644
--- a/biojava-core/src/main/java/org/biojava/nbio/core/util/PrettyXMLWriter.java
+++ b/biojava-core/src/main/java/org/biojava/nbio/core/util/PrettyXMLWriter.java
@@ -72,7 +72,7 @@ public void declareNamespace(String nsURI, String prefixHint)
private void handleDeclaredNamespaces()
throws IOException
{
- if (namespacesDeclared.size() == 0) {
+ if (namespacesDeclared.isEmpty()) {
for (Iterator
+ * The version is read from the file's header without parsing the domains, so this + * additionally parses the first few thousand lines of the same file. That is enough to + * notice a column change — which is what ECOD did at v294.1, unnoticed for months — + * without building the three million domains the whole file now holds. + */ @Test public void testVersion() throws IOException { EcodDatabase ecod3 = EcodFactory.getEcodDatabase("latest"); String version = ecod3.getVersion(); assertNotNull(version); assertNotEquals("latest", version); + System.out.println("latest version of ECOD is "+version); + + File domainsFile = new File(((EcodInstallation) ecod3).getCacheLocation(), + "ecod.latest.domains.txt"); + assertTrue("No local copy of the domains file at "+domainsFile, domainsFile.exists()); + + EcodParser parser = new EcodParser(firstLines(domainsFile, 5000)); + assertEquals(version, parser.getVersion()); + assertFalse("No domains parsed from ECOD "+version + + "; the distribution format has probably changed", + parser.getDomains().isEmpty()); + } + + /** + * @return a reader over the first {@code maxLines} lines of the file + */ + private static Reader firstLines(File f, int maxLines) throws IOException { + StringBuilder head = new StringBuilder(); + try (BufferedReader in = new BufferedReader(new FileReader(f))) { + String line; + int n = 0; + while (n < maxLines && (line = in.readLine()) != null) { + head.append(line).append('\n'); + n++; + } + } + return new StringReader(head.toString()); } /** diff --git a/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/TestSeqResParsing.java b/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/TestSeqResParsing.java index de6c072719..6ea23aef8a 100644 --- a/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/TestSeqResParsing.java +++ b/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/TestSeqResParsing.java @@ -53,7 +53,7 @@ public void test11GS() throws IOException, StructureException{ s = StructureIO.getStructure(pdbID); assertNotNull(s); - assertTrue(s.getChains().size() > 0); + assertFalse(s.getChains().isEmpty()); Chain c = s.getChainByIndex(0); assertTrue(c.getSeqResGroups().size() > 2); diff --git a/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/density/DensityMapIntegrationTest.java b/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/density/DensityMapIntegrationTest.java new file mode 100644 index 0000000000..97bf76a294 --- /dev/null +++ b/biojava-integrationtest/src/test/java/org/biojava/nbio/structure/test/io/density/DensityMapIntegrationTest.java @@ -0,0 +1,215 @@ +/** + * BioJava development code + * + * This code may be freely distributed and modified under the terms of the GNU + * Lesser General Public Licence. This should be distributed with the code. If + * you do not have a copy, see: + * + * http://www.gnu.org/copyleft/lesser.html + * + * Copyright for this code is held jointly by the individual authors. These + * should be listed in @author doc comments. + * + * For more information on the BioJava project and its aims, or to join the + * biojava-l mailing list, visit the home page at: + * + * http://www.biojava.org/ + */ +package org.biojava.nbio.structure.test.io.density; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotNull; +import static org.junit.jupiter.api.Assertions.assertTrue; +import static org.junit.jupiter.api.Assertions.fail; + +import java.io.File; +import java.io.IOException; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.util.Arrays; +import java.util.List; + +import org.biojava.nbio.core.util.FileDownloadUtils; +import org.biojava.nbio.structure.PdbId; +import org.biojava.nbio.structure.io.density.Ccp4Header; +import org.biojava.nbio.structure.io.density.DensityFileFormat; +import org.biojava.nbio.structure.io.density.DensityMapCache; +import org.biojava.nbio.structure.io.density.DensityMapKind; +import org.biojava.nbio.structure.io.density.DensityMapRequest; +import org.biojava.nbio.structure.io.density.DensityMapResult; +import org.biojava.nbio.structure.io.density.DensityMapSource; +import org.biojava.nbio.structure.io.density.NoDensityMapException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; + +/** + * Density fetching against the real services. + *
+ * Deliberately frugal: the entries chosen keep a full run to a couple of
+ * megabytes plus a few small metadata calls. In particular the cryo-EM path is
+ * exercised with the size limit set so low that the 116 MB map is declined
+ * before any of its body is transferred, which tests the whole resolution and
+ * guard sequence without the download.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DensityMapIntegrationTest {
+
+ private File cacheRoot;
+ private DensityMapCache cache;
+
+ @BeforeEach
+ public void setUp() throws IOException {
+ cacheRoot = Files.createTempDirectory("bj-density-it").toFile();
+ cache = new DensityMapCache(cacheRoot.getAbsolutePath());
+ }
+
+ @AfterEach
+ public void tearDown() throws IOException {
+ FileDownloadUtils.deleteDirectory(cacheRoot.toPath());
+ }
+
+ /** The default path for an X-ray entry: the smallest source answers first. */
+ @Test
+ public void fetchesAnXrayMapFromTheFirstSourceTried() throws IOException {
+ DensityMapResult result = cache.getDensityMap(new PdbId("1cbs"), DensityMapKind.TWO_FO_FC);
+
+ assertEquals(DensityMapSource.RCSB_VOLUME_SERVER, result.getSource());
+ assertEquals(DensityMapKind.TWO_FO_FC, result.getKind());
+ assertTrue(result.isRenderable());
+ assertFalse(result.isFromCache());
+ assertTrue(result.getFileSizeBytes() > 1024);
+ assertTrue(DensityMapResult.metaFileFor(result.getFile()).isFile(),
+ "a .meta sidecar makes the result reconstructible offline");
+
+ // second call must come from the cache without another download
+ DensityMapResult again = cache.getDensityMap(new PdbId("1cbs"), DensityMapKind.TWO_FO_FC);
+ assertTrue(again.isFromCache());
+ assertEquals(result.getFile(), again.getFile());
+ }
+
+ /**
+ * Both map kinds come out of one download, and the difference map is presented
+ * under the companion name that makes Jmol read the other data block.
+ */
+ @Test
+ public void bothKindsShareASingleDownload() throws IOException {
+ DensityMapResult twoFoFc = cache.getDensityMap(new PdbId("1cbs"), DensityMapKind.TWO_FO_FC);
+ DensityMapResult foFc = cache.getDensityMap(new PdbId("1cbs"), DensityMapKind.FO_FC);
+
+ assertEquals(DensityMapKind.FO_FC, foFc.getKind());
+ assertFalse(twoFoFc.getFile().equals(foFc.getFile()), "the difference map needs its own file name");
+ assertTrue(foFc.getFile().getName().contains("&diff=1"),
+ "the marker has to be in the name for Jmol to select the FO-FC block");
+ assertEquals(twoFoFc.getFileSizeBytes(), foFc.getFileSizeBytes(),
+ "both names must address the same bytes");
+ }
+
+ /** PDBe serves real CCP4 files, which the header check should recognise. */
+ @Test
+ public void pdbeServesAGenuineCcp4Map() throws IOException {
+ cache.setSourceChain(DensityMapKind.TWO_FO_FC, Arrays.asList(DensityMapSource.PDBE_CCP4));
+ DensityMapResult result = cache.getDensityMap(new PdbId("1cbs"), DensityMapKind.TWO_FO_FC);
+
+ assertEquals(DensityMapSource.PDBE_CCP4, result.getSource());
+ assertEquals(DensityFileFormat.CCP4, result.getFormat());
+ assertTrue(Ccp4Header.isCcp4(result.getFile()), "the CCP4 stamp should be present at byte 208");
+ assertTrue(FileDownloadUtils.validateFile(result.getFile()));
+ }
+
+ /**
+ * The whole cryo-EM route: resolve the EMDB entry, pick up the author contour
+ * level, and decline the full map on size without transferring it.
+ */
+ @Test
+ public void resolvesCryoEmEntriesAndHonoursTheSizeLimit() throws IOException {
+ List
+ * The divided archive paths on files.wwpdb.org and files.rcsb.org return the content
+ * MD5 as the ETag. The flat /validation/download/ endpoint this provider now uses
+ * returns neither an ETag nor a Content-Length, and neither does the beta archive on
+ * those two hosts, so no digest can be recorded there. The size sidecar is written
+ * from the bytes actually read, so it exists either way.
+ *
+ * The digest is therefore asserted when the server offered one and skipped when it
+ * did not, rather than being required: requiring it would fail against the endpoint
+ * we use, and hard-coding the divided path would only work until the archive
+ * transition in July 2027.
+ */
+ @Test
+ public void mapCoefficientsArriveIntactAndVerifiable() throws IOException {
+ cache.setSourceEnabled(DensityMapSource.WWPDB_MAP_COEFFICIENTS, true);
+ cache.setSourceChain(DensityMapKind.TWO_FO_FC, Arrays.asList(DensityMapSource.WWPDB_MAP_COEFFICIENTS));
+
+ DensityMapResult result = cache.getDensityMap(DensityMapRequest.builder(new PdbId("1cbs"))
+ .kind(DensityMapKind.TWO_FO_FC)
+ .allowNonRenderableFormats(true)
+ .build());
+
+ assertEquals(DensityMapSource.WWPDB_MAP_COEFFICIENTS, result.getSource());
+ assertFalse(result.isRenderable(),
+ "structure factors are not a map and must not claim to be renderable");
+
+ // written from the observed byte count, so it is present whether or not the
+ // server declared a length
+ assertTrue(FileDownloadUtils.validateFile(result.getFile()),
+ "a freshly downloaded file must validate against its own sidecars");
+
+ File hashFile = new File(result.getFile().getParentFile(), result.getFile().getName() + ".hash_MD5");
+ if (hashFile.isFile()) {
+ String recorded = new String(Files.readAllBytes(hashFile.toPath()), StandardCharsets.UTF_8).trim();
+ assertTrue(FileDownloadUtils.verifyHash(result.getFile(), FileDownloadUtils.Hash.MD5, recorded),
+ "the recorded MD5 must match the file it describes");
+ } else {
+ System.out.println("No MD5 recorded for " + result.getSourceUrl()
+ + " - the server offered no usable ETag. Size validation still applies.");
+ }
+
+ // corrupt it and confirm validation actually catches it
+ Files.write(result.getFile().toPath(), new byte[] {0, 1, 2, 3});
+ assertFalse(FileDownloadUtils.validateFile(result.getFile()),
+ "a truncated file must not validate");
+ }
+}
diff --git a/biojava-modfinder/pom.xml b/biojava-modfinder/pom.xml
index 876203f207..343aa74f6c 100644
--- a/biojava-modfinder/pom.xml
+++ b/biojava-modfinder/pom.xml
@@ -4,7 +4,7 @@
+ * Both maps are displayed: the 2mFo-DFc map in blue at 1 sigma, which should hug
+ * the ligand closely, and the mFo-DFc difference map as a red and green pair at
+ * 3 sigma, which for a well refined structure should show very little.
+ *
+ * Once the window is up, the map can be manipulated from the Rasmol command box
+ * at the bottom, for instance
+ *
+ * The download happens on a {@link SwingWorker} rather than the event dispatch
+ * thread. Even the smallest source runs to a few hundred kilobytes and a
+ * full-resolution map can be far larger, so fetching inline would freeze the
+ * interface for as long as it took.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class MyShowDensityListener implements ActionListener {
+
+ private static final Logger logger = LoggerFactory.getLogger(MyShowDensityListener.class);
+
+ private static final String OPTION_2FOFC = "2Fo-Fc (electron density)";
+ private static final String OPTION_FOFC = "Fo-Fc (difference)";
+ private static final String OPTION_BOTH = "Both";
+ private static final String OPTION_AUTO = "Whatever is available";
+
+ private final AbstractAlignmentJmol parent;
+
+ /**
+ * @param parent the viewer to draw into
+ */
+ public MyShowDensityListener(AbstractAlignmentJmol parent) {
+ this.parent = parent;
+ }
+
+ @Override
+ public void actionPerformed(ActionEvent e) {
+ Structure structure = parent == null ? null : parent.getStructure();
+ if (structure == null) {
+ JOptionPane.showMessageDialog(parent == null ? null : parent.getFrame(),
+ "There is no structure on screen to fetch a density map for.",
+ "Show Electron Density", JOptionPane.INFORMATION_MESSAGE);
+ return;
+ }
+
+ PdbId pdbId = structure.getPdbId();
+ if (pdbId == null) {
+ String typed = JOptionPane.showInputDialog(parent.getFrame(),
+ "This structure has no PDB ID. Enter one to look up its density map:",
+ "Show Electron Density", JOptionPane.QUESTION_MESSAGE);
+ if (typed == null || typed.trim().isEmpty()) {
+ return;
+ }
+ try {
+ pdbId = new PdbId(typed.trim());
+ } catch (IllegalArgumentException ex) {
+ JOptionPane.showMessageDialog(parent.getFrame(), typed + " is not a valid PDB ID.",
+ "Show Electron Density", JOptionPane.ERROR_MESSAGE);
+ return;
+ }
+ }
+
+ Object[] options = {OPTION_2FOFC, OPTION_FOFC, OPTION_BOTH, OPTION_AUTO};
+ Object choice = JOptionPane.showInputDialog(parent.getFrame(),
+ "Which map would you like to see for " + pdbId.getId() + "?",
+ "Show Electron Density", JOptionPane.QUESTION_MESSAGE, null, options, OPTION_2FOFC);
+ if (choice == null) {
+ return;
+ }
+
+ List
+ * Contouring follows the convention for each kind of map: the 2mFo-DFc map at
+ * 1 sigma in blue, the mFo-DFc difference map as a signed red/green pair at 3
+ * sigma, and a cryo-EM map at the level its depositors recommend when the entry
+ * states one, falling back to 3 sigma when it does not.
+ *
+ * Clipping matters for more than tidiness: contouring a whole cryo-EM grid at
+ * mesh resolution can take long enough to make the interface appear frozen.
+ *
+ * @param map the map to display
+ * @param atomSelection a Jmol atom expression such as
+ * The option order is not a matter of taste:
+ * Going through {@link File#toURI()} percent-encodes spaces and removes
+ * backslashes, which Jmol would otherwise read as escape characters in a script
+ * string. On Windows the result is a single-slash
+ * The three entries chosen exercise the three outcomes the fallback chain has to
+ * handle:
+ *
+ * Parallel primitive arrays are used rather than an array of index-distance objects because there are ~30
+ * neighbors per atom: for a large structure that would mean millions of small short-lived objects.
+ */
+ static class Neighbors {
+
+ /** The neighbor atom indices, ordered by increasing distance to the central atom */
+ final int[] indices;
+ /** The distances to the central atom, in increasing order and parallel to {@link #indices} */
+ final double[] dists;
+
+ private Neighbors(int[] indices, double[] dists) {
+ this.indices = indices;
+ this.dists = dists;
+ }
+
+ /**
+ * Creates a Neighbors from the first count elements of the given buffers, copying them to exact-size arrays
+ * and sorting them by increasing distance.
+ * @param indicesBuffer the neighbor indices, only the first count elements are used
+ * @param distsBuffer the neighbor distances, only the first count elements are used
+ * @param count the number of neighbors
+ * @return the sorted neighbors
+ */
+ static Neighbors createSorted(int[] indicesBuffer, double[] distsBuffer, int count) {
+ int[] indices = Arrays.copyOf(indicesBuffer, count);
+ double[] dists = Arrays.copyOf(distsBuffer, count);
+ // Sorting by closest to farthest away neighbors achieves faster runtimes when checking for occluded
+ // sphere sample points in calcSingleAsa. This follows the ideas exposed in
+ // Eisenhaber et al, J Comp Chemistry 1994 (https://onlinelibrary.wiley.com/doi/epdf/10.1002/jcc.540160303)
+ // This is essential for performance: it brings down the number of occlusion checks to
+ // an average of n_sphere_points/10 per atom, producing ~ x4 performance gain overall.
+ // An insertion sort is used because the arrays are small (~30 elements on average) and because it avoids
+ // both the boxing of a comparator-based sort and the object allocation an index-distance array would need.
+ for (int i = 1; i < count; i++) {
+ double dist = dists[i];
+ int index = indices[i];
+ int j = i - 1;
+ while (j >= 0 && dists[j] > dist) {
+ dists[j + 1] = dists[j];
+ indices[j + 1] = indices[j];
+ j--;
+ }
+ dists[j + 1] = dist;
+ indices[j + 1] = index;
+ }
+ return new Neighbors(indices, dists);
}
}
@@ -116,9 +159,16 @@ static class IndexAndDistance {
private final double[] radii;
private final double probe;
private final int nThreads;
- private Vector3d[] spherePoints;
+ /**
+ * The sphere points to sample, as a flat array of interleaved x,y,z coordinates (thus of size 3 x nSpherePoints).
+ * A flat array of primitives (rather than an array of Vector3d objects) is used for performance: it keeps the
+ * points contiguous in memory and avoids a pointer dereference per point in the innermost loop of
+ * {@link #calcSingleAsa(int)}.
+ */
+ private double[] spherePoints;
+ private int nSpherePoints;
private double cons;
- private IndexAndDistance[][] neighborIndices;
+ private Neighbors[] neighbors;
private boolean useSpatialHashingForNeighbors;
@@ -239,7 +289,8 @@ private void initSpherePoints(int nSpherePoints) {
logger.debug("Will use {} sphere points", nSpherePoints);
// initialising the sphere points to sample
- spherePoints = generateSpherePoints(nSpherePoints);
+ this.nSpherePoints = nSpherePoints;
+ this.spherePoints = generateSpherePoints(nSpherePoints);
cons = 4.0 * Math.PI / nSpherePoints;
}
@@ -285,10 +336,10 @@ public double[] calculateAsas() {
long start = System.currentTimeMillis();
if (useSpatialHashingForNeighbors) {
logger.debug("Will use spatial hashing to find neighbors");
- neighborIndices = findNeighborIndicesSpatialHashing();
+ neighbors = findNeighborIndicesSpatialHashing();
} else {
logger.debug("Will not use spatial hashing to find neighbors");
- neighborIndices = findNeighborIndices();
+ neighbors = findNeighborIndices();
}
long end = System.currentTimeMillis();
logger.debug("Took {} s to find neighbors", (end-start)/1000.0);
@@ -334,109 +385,126 @@ void setUseSpatialHashingForNeighbors(boolean useSpatialHashingForNeighbors) {
* Returns list of 3d coordinates of points on a unit sphere using the
* Golden Section Spiral algorithm.
* @param nSpherePoints the number of points to be used in generating the spherical dot-density
- * @return the array of points as Vector3d objects
+ * @return a flat array of interleaved x,y,z coordinates, of size 3 x nSpherePoints
*/
- private Vector3d[] generateSpherePoints(int nSpherePoints) {
- Vector3d[] points = new Vector3d[nSpherePoints];
+ private double[] generateSpherePoints(int nSpherePoints) {
+ double[] points = new double[3 * nSpherePoints];
double inc = Math.PI * (3.0 - Math.sqrt(5.0));
double offset = 2.0 / nSpherePoints;
for (int k=0;k
+ * Contacts are keyed by the ordered pair of {@link AtomIdentifier}s of the 2 atoms, i.e. the
+ * pair (a,b) and the pair (b,a) are 2 different keys. Thus look-ups ({@link #hasContact(Atom, Atom)},
+ * {@link #getContact(Atom, Atom)}) must give the 2 atoms in the same order in which the contact was
+ * calculated. The order produced by the calculation ({@link Grid}) is:
+ *
+ * Note that the order is not the order of PDB serials or of any other property of the atoms
+ * themselves: it is only the order in which the atoms were given to the calculation.
*
* @author duarte_j
*
@@ -37,6 +53,12 @@ public class AtomContactSet implements Serializable, Iterable
+ * The 2 atoms have to be passed in the same order in which the contacts of this set were
+ * calculated, otherwise this returns false even if the 2 atoms are within the distance cutoff.
+ * See the class documentation for the ordering convention. If the order is not known, both orders
+ * have to be queried.
+ * @param atom1 the first atom of the pair
+ * @param atom2 the second atom of the pair
+ * @return true if the 2 atoms are in contact in the given order, false otherwise
+ * @see #getContact(Atom, Atom)
+ */
public boolean hasContact(Atom atom1, Atom atom2) {
return hasContact(
new AtomIdentifier(atom1.getPDBserial(),atom1.getGroup().getChainId()),
new AtomIdentifier(atom2.getPDBserial(),atom2.getGroup().getChainId()) );
}
+ /**
+ * Tells whether a contact exists between the 2 given atom identifiers, in the given order,
+ * see {@link #hasContact(Atom, Atom)}.
+ * @param atomId1 the identifier of the first atom of the pair
+ * @param atomId2 the identifier of the second atom of the pair
+ * @return true if the 2 atoms are in contact in the given order, false otherwise
+ */
public boolean hasContact(AtomIdentifier atomId1, AtomIdentifier atomId2) {
return contacts.containsKey(new Pair
+ * As in {@link #hasContact(Atom, Atom)} the order of the 2 atoms matters: they have to be passed
+ * in the same order in which the contacts of this set were calculated, otherwise null is returned
+ * even if the 2 atoms are within the distance cutoff. See the class documentation for the
+ * ordering convention.
+ * @param atom1 the first atom of the pair
+ * @param atom2 the second atom of the pair
+ * @return the contact between the 2 atoms in the given order, or null if there is none
*/
public AtomContact getContact(Atom atom1, Atom atom2) {
return contacts.get(new Pair
+ * Since 7.3.0 this reads only the file's header rather than parsing the whole
+ * file, so it no longer has the side effect of loading every domain.
* @return the ECOD version
* @throws IOException If an error occurs while downloading or parsing the file
*/
@Override
public String getVersion() throws IOException {
- ensureDomainsFileInstalled();
+ domainsFileLock.readLock().lock();
+ logger.trace("LOCK readlock");
+ try {
+ if( parsedVersion != null ) {
+ return parsedVersion;
+ }
+ } finally {
+ logger.trace("UNLOCK readlock");
+ domainsFileLock.readLock().unlock();
+ }
+
+ // The version is declared in the first few lines of the file, so read those rather
+ // than the millions of domain records behind them. The current release is 657 MB and
+ // holds nearly three million records; parsing it in full to answer this question
+ // costs over a gigabyte of heap and several seconds.
+ ensureDomainsFileDownloaded();
+
+ domainsFileLock.writeLock().lock();
+ logger.trace("LOCK writelock");
+ try {
+ if( parsedVersion == null ) {
+ parsedVersion = parseVersionOnly();
+ }
+ } finally {
+ logger.trace("UNLOCK writelock");
+ domainsFileLock.writeLock().unlock();
+ }
if( parsedVersion == null) {
return requestedVersion;
@@ -285,6 +326,30 @@ public String getVersion() throws IOException {
return parsedVersion;
}
+ /**
+ * Reads the version from the header of the local domains file without parsing the
+ * domains themselves.
+ * @return the version, or null if the header does not declare one
+ * @throws IOException if the file cannot be read
+ * @since 7.3.0
+ */
+ private String parseVersionOnly() throws IOException {
+ try( BufferedReader in = new BufferedReader(new FileReader(getDomainFile())) ) {
+ String line;
+ while( (line = in.readLine()) != null ) {
+ Matcher match = EcodParser.VERSION_RE.matcher(line);
+ if( match.matches() ) {
+ return match.group(1);
+ }
+ if( !line.startsWith("#") ) {
+ // past the header block; from v294.1 the column names are not commented
+ return null;
+ }
+ }
+ }
+ return null;
+ }
+
/**
* Get the top-level ECOD server URL. Defaults to "http://prodata.swmed.edu"
* @return the url to the ecod server
@@ -325,6 +390,24 @@ public void setCacheLocation(String cacheLocation) {
domainsFileLock.writeLock().unlock();
}
+ /**
+ * Ensures the domains file is present and current locally, without parsing it.
+ * @throws IOException in cases of file I/O, including failure to download a healthy file
+ * @since 7.3.0
+ */
+ private void ensureDomainsFileDownloaded() throws IOException {
+ domainsFileLock.writeLock().lock();
+ logger.trace("LOCK writelock");
+ try {
+ if( !domainsAvailable() ) {
+ downloadDomains();
+ }
+ } finally {
+ logger.trace("UNLOCK writelock");
+ domainsFileLock.writeLock().unlock();
+ }
+ }
+
/**
* Blocks until ECOD domains file has been downloaded and parsed.
*
@@ -549,6 +632,24 @@ Current version (1.4) contains the following columns:
v1.2 - added f-group identifiers to fasta file, domain description file. ECODf identifiers now used when available for F-group name.
Domain assemblies now represented by assembly uid in domain assembly status.
v1.4 - added seqid_range and headers (develop101)
+v1.6 - renamed column 4 from f_id to t_id and inserted unp_acc (UniProt accession) as
+ column 9, giving 16 columns (seen in develop291)
+
+From v294.1 the distribution was redesigned. The header comment changed from
+"#ECOD version develop291" to "# Version: v294.1", the column header row is no longer
+commented out, and the columns became:
+
+ uid ecod_domain_id manual_rep f_id pdb chain pdb_range seqid_range architecture_name
+ x_name h_name t_name f_name assembly_id domain_id_short range_count arch_manual
+ x_manual h_manual t_manual f_manual valid_structure ligand_binding
+
+v295 appends ligand_comp_ids and ligand_pdbnum, for 25 columns. Also note that
+manual_rep now holds True/False rather than MANUAL_REP/AUTO_NONREP, that assembly_id
+and domain_id_short are empty on every row, that f_name is empty rather than
+F_UNCLASSIFIED for unclassified domains, and that uid restarts from 0.
+
+Because the columns have been renamed, reordered and added to repeatedly, files that
+declare a column header are read by column name rather than by position.
*/
/** String for unclassified F-groups */
@@ -561,10 +662,28 @@ Current version (1.4) contains the following columns:
public static final String IS_REPRESENTATIVE = "MANUAL_REP";
/** Indicates not a manual representative */
public static final String NOT_REPRESENTATIVE = "AUTO_NONREP";
+ /**
+ * Matches the comment declaring the version, which has taken two forms:
+ * {@code #ECOD version develop291} up to develop292, and {@code # Version: v295}
+ * from v294.1 onwards.
+ * @since 7.3.0
+ */
+ static final Pattern VERSION_RE = Pattern.compile(
+ "^\\s*#\\s*(?:ECOD\\s+)?version\\s*:?\\s*(\\S+).*", Pattern.CASE_INSENSITIVE);
private List
+ * Every distribution since develop101 carries such a header. It is commented
+ * (
+ * The characters are taken relative to the end of the identifier rather
+ * than the start, so that both spellings of the same entry land in the same
+ * bucket:
+ * This is reported through the logger and on both standard output and standard
+ * error. A user who asked for a map and silently got a coarser one from another
+ * source deserves to be told why, and a library log configuration that discards
+ * warnings should not be able to hide it.
+ *
+ * @param url the resource that was skipped
+ * @param size its size in bytes
+ * @param maxBytes the limit in bytes
+ * @param request the request being served
+ */
+ protected void reportTooLarge(URL url, long size, long maxBytes, DensityMapRequest request) {
+ String entry = request.getPdbId() != null ? request.getPdbId().getId()
+ : (request.getEmdbId() != null ? request.getEmdbId() : "?");
+ String message = String.format(
+ "BioJava density: skipping %s for %s (%,d bytes exceeds the %,d byte limit). "
+ + "Trying a smaller representation from another source; "
+ + "raise DensityMapCache.setMaxDownloadBytes() to allow it.",
+ url, entry, size, maxBytes);
+ logger.warn(message);
+ System.out.println(message);
+ System.err.println(message);
+ }
+
+ /**
+ * The
+ * This is a cheap but effective guard against a cached file that is not
+ * actually a map. A misbehaving or overloaded server can answer with an HTML
+ * error page and an HTTP 200, in which case nothing about the status code or the
+ * content length reveals the problem — but the missing
+ * PDB-keyed maps follow the divided layout the rest of BioJava already uses, so
+ * that a cache with many entries does not end up with one enormous directory:
+ *
+ * EMDB maps are keyed by EMDB identifier instead, mirroring the EMDB archive:
+ *
+ * The two-character directory needs no attention when the archive moves to
+ * extended identifiers in July 2027. It is a device for spreading files over
+ * directories, not a copy of the archive's own layout, and
+ * {@link LocalPDBDirectory#getMiddleHash(String)} counts from the right hand end
+ * of the identifier, so
+ * Deliberately not the archive's per-entry layout. A cache is not a mirror
+ * and cannot become one: the archive publishes structure factors and map
+ * coefficients but never grids, so density has to be fetched whatever else is
+ * mirrored, and writing it into a directory whose contents are an exact copy of
+ * upstream puts it at the mercy of the next
+ * The separate kind token exists for sources that deliver more than one kind of
+ * map in a single file. A density server response, for instance, carries both
+ * the 2Fo-Fc and the Fo-Fc blocks, so it is cached once under
+ * {@link #BOTH_KINDS_TOKEN} rather than downloaded and stored twice.
+ *
+ * @param cacheRoot the BioJava cache directory
+ * @param pdbId the entry
+ * @param kindToken the token naming what the file holds
+ * @param source the service it came from
+ * @param format the file format
+ * @param qualifier an extra discriminator such as a detail level, or
+ * A density server response holds both a
+ * This was verified against Jmol 14.31.10 and is unchanged in current Jmol: the
+ * relevant line in
+ * Every entry in the archive today has a four-character form, and the density
+ * services accept only that spelling. Extended-only identifiers will appear
+ * eventually; rather than refusing them, the extended spelling is passed
+ * through so that the services can start accepting it without a change here.
+ *
+ * @param pdbId the identifier
+ * @return the short spelling if available, otherwise the full one
+ */
+ public static String shortIdOrFull(PdbId pdbId) {
+ return pdbId.getId(true);
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityFileFormat.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityFileFormat.java
new file mode 100644
index 0000000000..11cbb0e27e
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityFileFormat.java
@@ -0,0 +1,89 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+/**
+ * The file format a density map was delivered in.
+ *
+ * The important distinction here is {@link #isJmolLoadable()}. Most of these
+ * formats are sampled grids that a viewer can contour directly; map
+ * coefficients are not, they are structure factors that require a Fourier
+ * transform first. Handing the latter to a viewer produces nothing at all, so
+ * the difference has to be visible to callers.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public enum DensityFileFormat {
+
+ /** A CCP4/MRC sampled map, as served pre-computed by PDBe. */
+ CCP4(".ccp4", true),
+
+ /** A gzipped CCP4/MRC map, the form EMDB distributes its primary maps in. */
+ CCP4_GZ(".map.gz", true),
+
+ /** A BinaryCIF volume slice from a Mol* density server. */
+ BCIF_VOLUME(".bcif", true),
+
+ /** A text CIF volume slice from a Mol* density server. */
+ CIF_VOLUME(".cif", true),
+
+ /**
+ * Structure-factor amplitudes and phases in mmCIF, as published with the wwPDB
+ * validation reports. Not a density grid. A Fourier transform (for
+ * example
+ * Cached files live under the BioJava cache directory (
+ * Source order. Sources are tried smallest-adequate-first, because they
+ * differ enormously in size for the same entry and the smallest is usually
+ * perfectly adequate to look at. For 1cbs a density-server slice is roughly a
+ * tenth the size of the equivalent pair of CCP4 files; for the cryo-EM entry
+ * behind EMD-0262 it is a few hundred kilobytes against 106 MB. Anything
+ * that cannot be displayed at all is tried last, and
+ * {@link DensityMapSource#WWPDB_MAP_COEFFICIENTS} is disabled altogether by
+ * default for that reason. Override with {@link #setSourceChain(DensityMapKind,
+ * java.util.List)} or {@link #setSourceEnabled(DensityMapSource, boolean)}.
+ *
+ * Failures. A source that has nothing for an entry is skipped and the next
+ * is tried; if every source is exhausted, {@link NoDensityMapException} is thrown
+ * carrying the reason from each one, so a caller can explain what happened.
+ * Genuine transport failures abort the chain instead, so that a network outage is
+ * never reported as "this entry has no density".
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DensityMapCache {
+
+ private static final Logger logger = LoggerFactory.getLogger(DensityMapCache.class);
+
+ /**
+ * Default ceiling on a single download, 256 MiB. Exceeding it is not an
+ * error: the chain moves on to a source that offers a smaller representation.
+ * Set to 0 to remove the limit.
+ */
+ public static final long DEFAULT_MAX_DOWNLOAD_BYTES = 256L * 1024 * 1024;
+
+ /** Order in which sources are tried for X-ray and neutron entries. */
+ public static final List
+ * {@link DensityMapSource#WWPDB_MAP_COEFFICIENTS} is disabled by default
+ * because it delivers structure factors rather than a map; enable it explicitly
+ * if you want the archival form and are prepared to run a Fourier transform.
+ *
+ * @param source the source
+ * @param enabled whether to try it
+ */
+ public void setSourceEnabled(DensityMapSource source, boolean enabled) {
+ if (enabled) {
+ disabled.remove(source);
+ } else {
+ disabled.add(source);
+ }
+ }
+
+ /**
+ * Replaces the provider used for a source. Mainly a testing seam, but also the
+ * way to plug in a site-local mirror.
+ *
+ * @param provider the provider to use
+ */
+ public void registerProvider(DensityMapProvider provider) {
+ register(provider);
+ applySettingsToProviders();
+ }
+
+ /** @return the resolver used to map PDB entries to EMDB entries */
+ public EmdbEntryResolver getEmdbResolver() {
+ return emdbResolver;
+ }
+
+ /** @param resolver the resolver used to map PDB entries to EMDB entries */
+ public void setEmdbResolver(EmdbEntryResolver resolver) {
+ this.emdbResolver = resolver;
+ applySettingsToProviders();
+ }
+
+ /**
+ * Fetches a density map, using the cache where possible.
+ *
+ * @param pdbId the entry
+ * @param kind the kind of map, or {@link DensityMapKind#AUTO} to take whatever
+ * the entry has
+ * @return the map
+ * @throws NoDensityMapException if no enabled source has a map for this entry
+ * @throws IOException on transport failure
+ */
+ public DensityMapResult getDensityMap(PdbId pdbId, DensityMapKind kind) throws IOException {
+ return getDensityMap(DensityMapRequest.builder(pdbId).kind(kind).build());
+ }
+
+ /**
+ * Fetches a density map for a PDB or EMDB identifier.
+ *
+ * @param id a PDB identifier, or an EMDB identifier such as
+ * Knowing the structure lets the experimental method be read directly rather
+ * than guessed, so a cryo-EM entry goes straight to the EM sources instead of
+ * trying the X-ray ones first. No extra network request is involved.
+ *
+ * @param structure the structure
+ * @param kind the kind of map wanted
+ * @return the map
+ * @throws NoDensityMapException if no enabled source has a map for this entry
+ * @throws IOException on transport failure
+ */
+ public DensityMapResult getDensityMap(Structure structure, DensityMapKind kind) throws IOException {
+ PdbId pdbId = structure.getPdbId();
+ if (pdbId == null) {
+ throw new IOException("The structure has no PDB ID, so no density map can be looked up for it.");
+ }
+ DensityMapRequest request = DensityMapRequest.builder(pdbId).kind(kind).build();
+ return getDensityMap(request, kindOrderFor(structure, kind));
+ }
+
+ /**
+ * Fetches a density map.
+ *
+ * @param request what is wanted
+ * @return the map
+ * @throws NoDensityMapException if no enabled source has a map for this entry
+ * @throws IOException on transport failure
+ */
+ public DensityMapResult getDensityMap(DensityMapRequest request) throws IOException {
+ return getDensityMap(request, request.getKind().resolve());
+ }
+
+ /**
+ * As {@link #getDensityMap(PdbId, DensityMapKind)} but returning an empty
+ * {@link Optional} rather than throwing when nothing is available. Genuine
+ * transport failures are still logged and swallowed, so use this only where
+ * "no map" and "could not reach the server" need not be told apart.
+ *
+ * @param pdbId the entry
+ * @param kind the kind of map wanted
+ * @return the map, if one could be obtained
+ */
+ public Optional
+ * A density server returns voxels and nothing else, but an EM map is
+ * conventionally displayed at the level its depositors chose rather than at a
+ * multiple of sigma, so a viewer needs that number whichever source the map came
+ * from. It costs one small metadata request, cached thereafter.
+ */
+ private DensityMapResult withContourLevel(DensityMapResult result) {
+ if (result.getKind() != DensityMapKind.EM || result.getRecommendedContourLevel() != null
+ || result.getEmdbId() == null || emdbResolver == null) {
+ return result;
+ }
+ EmdbEntryInfo info = emdbResolver.getEntryInfo(result.getEmdbId());
+ if (info == null || (info.getRecommendedContourLevel() == null && info.getSigma() == null)) {
+ return result;
+ }
+ DensityMapResult enriched = new DensityMapResult(result.getFile(), result.getSource(), result.getFormat(),
+ result.getKind(), result.getPdbId(), result.getEmdbId(), result.getSourceUrl(), result.isFromCache(),
+ info.getRecommendedContourLevel(), info.getSigma());
+ enriched.writeMeta();
+ return enriched;
+ }
+
+ private String resolveEmdbId(DensityMapRequest request) {
+ if (request.getPdbId() == null || emdbResolver == null) {
+ return null;
+ }
+ List
+ * Implementations are combined into an ordered chain by {@link DensityMapCache},
+ * which tries each in turn until one produces a map. The contract around
+ * exceptions is what makes that chain safe:
+ *
+ * Instances are immutable; build them with {@link #builder(PdbId)} or
+ * {@link #builder(String)}.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DensityMapRequest {
+
+ private final PdbId pdbId;
+ private final String emdbId;
+ private final DensityMapKind kind;
+ private final FetchBehavior fetchBehavior;
+ private final File cacheDir;
+ private final boolean allowNonRenderableFormats;
+ private final long maxDownloadBytes;
+ private final List
+ * Which source answered matters to the caller and is therefore part of the
+ * result: the file might be a full CCP4 grid, a downsampled volume slice, or
+ * — if non-renderable formats were allowed — a set of structure
+ * factors that must be Fourier-transformed before anything can be drawn. See
+ * {@link #isRenderable()}.
+ *
+ * Every field is persisted to a
+ * These differ enormously in size for the same entry, which is why more than one
+ * is supported. For PDB entry 1cbs a density-server slice at the coarsest detail
+ * level is about 210 kB and contains both the 2Fo-Fc and Fo-Fc maps, where
+ * the two pre-computed CCP4 files come to about 2.1 MB together. For cryo-EM
+ * the gap is far wider: the primary map of EMD-0262 is about 106 MB, against
+ * roughly 480 kB for the equivalent density-server slice.
+ *
+ * Note that RCSB's own
+ * Cryo-EM primary maps in particular can be very large — hundreds of
+ * megabytes is common and gigabyte maps exist — while a downsampled slice
+ * of the same map from a density server is usually a fraction of a percent of
+ * that size and quite adequate for display. Rather than failing, the fallback
+ * chain treats this as "try the next source", which is exactly why the density
+ * servers are tried ahead of the full-resolution archives.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DensityMapTooLargeException extends IOException {
+
+ private static final long serialVersionUID = 1L;
+
+ private final long sizeBytes;
+ private final long limitBytes;
+
+ /**
+ * @param url the resource that was too large
+ * @param sizeBytes its size, or a negative value if only a lower bound is known
+ * @param limitBytes the configured limit
+ */
+ public DensityMapTooLargeException(String url, long sizeBytes, long limitBytes) {
+ super(String.format("%s is %s, which exceeds the %s download limit.",
+ url, describe(sizeBytes), describe(limitBytes)));
+ this.sizeBytes = sizeBytes;
+ this.limitBytes = limitBytes;
+ }
+
+ private static String describe(long bytes) {
+ if (bytes < 0) {
+ return "of unknown size";
+ }
+ if (bytes < 1024) {
+ return bytes + " B";
+ }
+ double value = bytes;
+ String[] units = {"kB", "MB", "GB", "TB"};
+ int unit = -1;
+ while (value >= 1024 && unit < units.length - 1) {
+ value /= 1024;
+ unit++;
+ }
+ return String.format("%.1f %s", value, units[unit]);
+ }
+
+ /**
+ * @return the size of the resource in bytes, or a negative value if unknown
+ */
+ public long getSizeBytes() {
+ return sizeBytes;
+ }
+
+ /**
+ * @return the configured limit in bytes
+ */
+ public long getLimitBytes() {
+ return limitBytes;
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/EmdbEntryInfo.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/EmdbEntryInfo.java
new file mode 100644
index 0000000000..7c8b0dd302
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/EmdbEntryInfo.java
@@ -0,0 +1,85 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+/**
+ * The few facts about an EMDB entry's primary map that matter when fetching and
+ * displaying it.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class EmdbEntryInfo {
+
+ private final String emdbId;
+ private final Double recommendedContourLevel;
+ private final Double sigma;
+ private final Long mapSizeBytes;
+
+ /**
+ * @param emdbId the entry identifier, canonical form
+ * @param recommendedContourLevel the author-recommended contour level in absolute
+ * map units, or
+ * The primary lookup is EMDB's own search API, which returns the identifier and
+ * the author-recommended contour level together in a single small CSV response.
+ * That is also the route Jmol uses, so BioJava's answer matches what a user sees
+ * elsewhere. If it fails, RCSB's entry API is consulted for the identifier alone.
+ *
+ * The experimental method is deliberately not inferred from a structure's
+ * resolution: BioJava is known to mis-parse resolution for some cryo-EM entries
+ * (biojava/biojava#1000), so the presence of an EMDB identifier is the reliable
+ * signal.
+ *
+ * Answers are cached on disk. Under
+ * {@link FetchBehavior#LOCAL_ONLY} the cached answer is used whatever its age and
+ * no connection is opened.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class EmdbEntryResolver {
+
+ private static final Logger logger = LoggerFactory.getLogger(EmdbEntryResolver.class);
+
+ /**
+ * Default template for the EMDB search that maps a PDB entry to the EM
+ * reconstructions it was fitted into, returning the contour level as well.
+ */
+ public static final String DEFAULT_SEARCH_URL_TEMPLATE =
+ "https://www.ebi.ac.uk/emdb/api/search/fitted_pdbs:{pdbid_lc}?fl=emdb_id,map_contour_level_value&wt=csv";
+
+ /** Default template for the EMDB entry map metadata. */
+ public static final String DEFAULT_MAP_INFO_URL_TEMPLATE =
+ "https://www.ebi.ac.uk/emdb/api/entry/map/{emdb_id}";
+
+ /** Default template for the RCSB entry record, used as a fallback. */
+ public static final String DEFAULT_RCSB_ENTRY_URL_TEMPLATE =
+ "https://data.rcsb.org/rest/v1/core/entry/{pdbid_lc}";
+
+ private static final int TIMEOUT_MILLIS = 30000;
+ private static final ObjectMapper MAPPER = new ObjectMapper();
+
+ private static String searchUrlTemplate = DEFAULT_SEARCH_URL_TEMPLATE;
+ private static String mapInfoUrlTemplate = DEFAULT_MAP_INFO_URL_TEMPLATE;
+ private static String rcsbEntryUrlTemplate = DEFAULT_RCSB_ENTRY_URL_TEMPLATE;
+
+ private File cacheRoot;
+ private FetchBehavior fetchBehavior = FetchBehavior.FETCH_FILES;
+ private int mappingMaxAgeDays = 30;
+
+ /**
+ * @param cacheRoot the BioJava cache directory
+ */
+ public EmdbEntryResolver(File cacheRoot) {
+ this.cacheRoot = cacheRoot;
+ }
+
+ /** @param cacheRoot the BioJava cache directory */
+ public void setCacheRoot(File cacheRoot) {
+ this.cacheRoot = cacheRoot;
+ }
+
+ /** @param fetchBehavior how aggressively to refresh cached answers */
+ public void setFetchBehavior(FetchBehavior fetchBehavior) {
+ this.fetchBehavior = fetchBehavior == null ? FetchBehavior.FETCH_FILES : fetchBehavior;
+ }
+
+ /**
+ * @param days how long a cached mapping stays fresh. Mappings change only when
+ * an entry is re-released, so this can be generous.
+ */
+ public void setMappingMaxAgeDays(int days) {
+ this.mappingMaxAgeDays = days;
+ }
+
+ /** @param template the EMDB search URL template */
+ public static void setSearchUrlTemplate(String template) {
+ searchUrlTemplate = template == null ? DEFAULT_SEARCH_URL_TEMPLATE : template;
+ }
+
+ /** @param template the EMDB map metadata URL template */
+ public static void setMapInfoUrlTemplate(String template) {
+ mapInfoUrlTemplate = template == null ? DEFAULT_MAP_INFO_URL_TEMPLATE : template;
+ }
+
+ /** @param template the RCSB entry URL template used as a fallback */
+ public static void setRcsbEntryUrlTemplate(String template) {
+ rcsbEntryUrlTemplate = template == null ? DEFAULT_RCSB_ENTRY_URL_TEMPLATE : template;
+ }
+
+ /** Restores all default URL templates. */
+ public static void resetToDefaults() {
+ searchUrlTemplate = DEFAULT_SEARCH_URL_TEMPLATE;
+ mapInfoUrlTemplate = DEFAULT_MAP_INFO_URL_TEMPLATE;
+ rcsbEntryUrlTemplate = DEFAULT_RCSB_ENTRY_URL_TEMPLATE;
+ }
+
+ /**
+ * The EMDB entries a PDB entry was fitted into.
+ *
+ * @param pdbId the PDB entry
+ * @return the EMDB identifiers, most relevant first; empty if the entry is not
+ * an EM structure or has no associated map
+ */
+ public List
+ * The response is cached verbatim, so asking twice costs one request.
+ *
+ * @param emdbId the EMDB entry, in any accepted form
+ * @return the metadata, or
+ * These are gzipped CCP4/MRC files, and they are big: the primary map of
+ * EMD-0262 is about 106 MB, and gigabyte maps exist. For most display
+ * purposes a slice from a density server is a far better trade — a few
+ * hundred kilobytes for the same entry — which is why
+ * {@link DensityMapCache} tries {@link VolumeServerProvider} first and only falls
+ * back here. Use this source when the full sampling genuinely matters.
+ *
+ * The size limit is checked against the size EMDB itself reports before any of
+ * the body is transferred, so exceeding it costs one small metadata request
+ * rather than a partial download.
+ *
+ * Jmol recognises gzip from the file's magic bytes, so the cached
+ *
+ * This is a routine outcome rather than an error: not every entry has density.
+ * Structures deposited without structure factors have none at all (4HHB, for
+ * instance), and a cryo-EM entry has no X-ray maps by construction. The
+ * per-source reasons are kept so that a caller — particularly a user
+ * interface — can explain why rather than just reporting a failure.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class NoDensityMapException extends IOException {
+
+ private static final long serialVersionUID = 1L;
+
+ private final PdbId pdbId;
+ private final DensityMapKind kind;
+ private final Map
+ * These are full-resolution sampled grids, one file per map kind, and are the
+ * source Jmol itself uses for its built-in map shortcuts. They are larger than a
+ * density-server slice — about 1 MB per map for a small entry such as
+ * 1cbs — but need no interpretation beyond contouring.
+ *
+ * The service accepts only the lower-case four-character spelling of an
+ * identifier;
+ * The recognised placeholders are:
+ *
+ *
+ * This is deliberately separate from the template mechanism in
+ * {@code DownloadChemCompProvider}, which resolves a single chemical-component
+ * identifier with optional substring indices. The two solve different problems:
+ * here several distinct values are substituted by name.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class UrlTemplates {
+
+ private static final Pattern PLACEHOLDER = Pattern.compile("\\{([a-zA-Z_]+)\\}");
+
+ private UrlTemplates() {
+ }
+
+ /**
+ * Expands a template against a set of named values.
+ *
+ * @param template the template string
+ * @param values the values, keyed by placeholder name without the braces
+ * @return the expanded string
+ */
+ public static String expand(String template, Map
+ * This is usually the most economical source by a wide margin. For PDB entry
+ * 1cbs the coarsest slice is about 210 kB against roughly 2.1 MB for
+ * the two pre-computed CCP4 files, and for cryo-EM the difference is far larger
+ * still: about 480 kB against the 106 MB primary map of EMD-0262. A
+ * single response carries both the 2Fo-Fc and Fo-Fc data blocks for X-ray
+ * entries.
+ *
+ * Caveats worth knowing. Responses are chunked, with no
+ *
+ * The companion is a hard link where the filesystem allows one, so the second
+ * name costs no additional space; a copy is only made if linking is refused.
+ */
+ private DensityMapResult presentAsDifferenceMap(DensityMapResult result) throws IOException {
+ File marker = DensityCacheLayout.differenceMarkerFile(result.getFile());
+ if (!marker.isFile() || marker.length() != result.getFile().length()) {
+ Files.deleteIfExists(marker.toPath());
+ try {
+ Files.createLink(marker.toPath(), result.getFile().toPath());
+ } catch (IOException | UnsupportedOperationException e) {
+ Files.copy(result.getFile().toPath(), marker.toPath(), StandardCopyOption.REPLACE_EXISTING);
+ }
+ }
+ return new DensityMapResult(marker, result.getSource(), result.getFormat(), result.getKind(),
+ result.getPdbId(), result.getEmdbId(), result.getSourceUrl(), result.isFromCache(),
+ result.getRecommendedContourLevel(), result.getSigma());
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/WwpdbMapCoefficientsProvider.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/WwpdbMapCoefficientsProvider.java
new file mode 100644
index 0000000000..bd1b066a64
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/WwpdbMapCoefficientsProvider.java
@@ -0,0 +1,219 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+
+import org.biojava.nbio.structure.PdbId;
+
+/**
+ * Fetches the map coefficients published alongside the wwPDB validation reports.
+ *
+ * These are not density maps. They are structure-factor amplitudes and
+ * phases in mmCIF, exactly as used to produce the pictures in a validation
+ * report, and a Fourier transform is required before anything can be drawn from
+ * them —
+ * They are supported because this is the route RCSB documents since
+ *
+ * One useful property: these servers return the content MD5 as the HTTP
+ *
+ * The URLs are built against the documented download endpoint, which resolves an
+ * entry by file name, rather than against the divided archive path. This is the
+ * only provider here that ever had a choice — the density servers and the
+ * EMDB archive are addressed by identifier already — and it matters because
+ * the PDB moves to extended identifiers and a per-entry directory layout in July
+ * 2027. A name survives that move; a constructed directory path does not. Mirrors
+ * that publish directories instead of an endpoint are still reachable, through
+ * {@link #DIVIDED_TWO_FO_FC_TEMPLATE} and {@link #ENTRIES_TWO_FO_FC_TEMPLATE}.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class WwpdbMapCoefficientsProvider extends AbstractDensityMapProvider {
+
+ /** Default base URL, the wwPDB validation report download endpoint. */
+ public static final String DEFAULT_SERVER_URL = "https://files.wwpdb.org/validation/download/";
+
+ /** An RCSB mirror serving byte-identical files. */
+ public static final String RCSB_MIRROR_URL = "https://files.rcsb.org/validation/download/";
+
+ /**
+ * The wwPDB beta archive, which already holds the re-organised content that
+ * replaces the current archive on 21 July 2027 and serves the same endpoint.
+ *
+ * Deliberately not the default, despite being the newer archive. The wwPDB
+ * describes this host as transitional: on the cutover date the beta archive
+ * replaces the main one, after which the beta URL is supported by redirection
+ * for three years. So it is the hostname that needs changing, twice, whereas
+ * {@link #DEFAULT_SERVER_URL} becomes the new archive and needs changing never.
+ *
+ * Its value is in testing. Because this host is the post-2027 content today, a
+ * request against it checks the endpoint against the archive as it will be,
+ * rather than against the archive as it is.
+ */
+ public static final String BETA_SERVER_URL = "https://files-beta.wwpdb.org/validation/download/";
+
+ /**
+ * An EBI mirror serving byte-identical files.
+ *
+ * Unlike the two above, EBI publishes no name-resolving endpoint — only
+ * full directory paths — so selecting it means setting the divided
+ * templates as well:
+ *
+ * The endpoint resolves an entry by name, so no directory path is built here.
+ * That is deliberate. In July 2027 the archive moves to extended identifiers
+ * and a per-entry directory layout, and a path assembled from a hash and an
+ * identifier would have to be rewritten for it; a name does not. Both spellings
+ * of an identifier resolve, so whichever {@code PdbId} yields is accepted.
+ */
+ public static final String DEFAULT_TWO_FO_FC_TEMPLATE =
+ "{pdbid_lc}_validation_2fo-fc_map_coef.cif.gz";
+
+ /** Default path template for the mFo-DFc coefficients; see {@link #DEFAULT_TWO_FO_FC_TEMPLATE}. */
+ public static final String DEFAULT_FO_FC_TEMPLATE =
+ "{pdbid_lc}_validation_fo-fc_map_coef.cif.gz";
+
+ /**
+ * Path template for the 2mFo-DFc coefficients in the divided archive, for
+ * mirrors that publish directories rather than an endpoint.
+ */
+ public static final String DIVIDED_TWO_FO_FC_TEMPLATE =
+ "{mid}/{pdbid_lc}/{pdbid_lc}_validation_2fo-fc_map_coef.cif.gz";
+
+ /** Path template for the mFo-DFc coefficients in the divided archive. */
+ public static final String DIVIDED_FO_FC_TEMPLATE =
+ "{mid}/{pdbid_lc}/{pdbid_lc}_validation_fo-fc_map_coef.cif.gz";
+
+ /**
+ * Path template for the 2mFo-DFc coefficients in the per-entry archive that
+ * replaces the divided one in July 2027, relative to a base URL ending in
+ *
+ * Provided so that a mirror of the new layout can be used the day it exists,
+ * without waiting for a release.
+ */
+ public static final String ENTRIES_TWO_FO_FC_TEMPLATE =
+ "entries/{mid}/{extid}/validation_reports/{extid}_validation_2fo-fc_map_coef.cif.gz";
+
+ /** Path template for the mFo-DFc coefficients in the per-entry archive. */
+ public static final String ENTRIES_FO_FC_TEMPLATE =
+ "entries/{mid}/{extid}/validation_reports/{extid}_validation_fo-fc_map_coef.cif.gz";
+
+ private static String serverBaseUrl = DEFAULT_SERVER_URL;
+ private static String twoFoFcTemplate = DEFAULT_TWO_FO_FC_TEMPLATE;
+ private static String foFcTemplate = DEFAULT_FO_FC_TEMPLATE;
+
+ /**
+ * @param cacheRoot the BioJava cache directory
+ */
+ public WwpdbMapCoefficientsProvider(File cacheRoot) {
+ super(cacheRoot);
+ }
+
+ /** @return the base URL of the validation report archive */
+ public static String getServerBaseUrl() {
+ return serverBaseUrl;
+ }
+
+ /** @param url the base URL; a trailing slash is added if missing */
+ public static void setServerBaseUrl(String url) {
+ serverBaseUrl = url == null ? DEFAULT_SERVER_URL : (url.endsWith("/") ? url : url + "/");
+ }
+
+ /**
+ * Overrides the path template for a map kind.
+ *
+ * @param kind {@link DensityMapKind#TWO_FO_FC} or {@link DensityMapKind#FO_FC}
+ * @param template a template understood by {@link UrlTemplates}
+ */
+ public static void setPathUrlTemplate(DensityMapKind kind, String template) {
+ if (kind == DensityMapKind.TWO_FO_FC) {
+ twoFoFcTemplate = template == null ? DEFAULT_TWO_FO_FC_TEMPLATE : template;
+ } else if (kind == DensityMapKind.FO_FC) {
+ foFcTemplate = template == null ? DEFAULT_FO_FC_TEMPLATE : template;
+ } else {
+ throw new IllegalArgumentException("Map coefficients exist only for 2Fo-Fc and Fo-Fc, not " + kind);
+ }
+ }
+
+ /** Restores the default server and templates. */
+ public static void resetToDefaults() {
+ serverBaseUrl = DEFAULT_SERVER_URL;
+ twoFoFcTemplate = DEFAULT_TWO_FO_FC_TEMPLATE;
+ foFcTemplate = DEFAULT_FO_FC_TEMPLATE;
+ }
+
+ @Override
+ public DensityMapSource getSource() {
+ return DensityMapSource.WWPDB_MAP_COEFFICIENTS;
+ }
+
+ @Override
+ public DensityFileFormat getFormat() {
+ return DensityFileFormat.MAP_COEFFICIENTS_CIF_GZ;
+ }
+
+ @Override
+ public boolean supports(DensityMapKind kind) {
+ return kind == DensityMapKind.TWO_FO_FC || kind == DensityMapKind.FO_FC;
+ }
+
+ /**
+ * Builds the URL for a set of coefficients without fetching them.
+ *
+ * @param pdbId the entry
+ * @param kind the kind of map
+ * @return the URL as a string
+ */
+ public String buildUrl(PdbId pdbId, DensityMapKind kind) {
+ String template = kind == DensityMapKind.FO_FC ? foFcTemplate : twoFoFcTemplate;
+ return serverBaseUrl + UrlTemplates.expand(template, UrlTemplates.values(urlId(pdbId), null, -1));
+ }
+
+ @Override
+ public DensityMapResult fetch(DensityMapRequest request) throws IOException {
+ if (request.getPdbId() == null || !supports(request.getKind())) {
+ return null;
+ }
+ URL url = new URL(buildUrl(request.getPdbId(), request.getKind()));
+ File target = DensityCacheLayout.pdbMapFile(effectiveCacheRoot(request), request.getPdbId(),
+ request.getKind(), getSource(), getFormat(), null);
+ return obtain(request, url, target, request.getKind(), null, null, null);
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/mmtf/MmtfStructureReader.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/mmtf/MmtfStructureReader.java
index c2830c1685..865c9e0da4 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/mmtf/MmtfStructureReader.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/mmtf/MmtfStructureReader.java
@@ -372,7 +372,7 @@ public void setInterGroupBond(int indOne, int indTwo, int bondOrder) {
private Group getCorrectAltLocGroup(Character altLoc) {
// see if we know this altLoc already;
List
+ * This is the failure that took the CATH downloader out when
+ *
+ * The test serves the responses from a local {@link HttpServer} rather than a real
+ * service. Pointing it at a third-party server that happens to redirect today would
+ * make the test fail on the day they stop, which is precisely the coupling that made
+ * the build unreliable in the first place.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class TestChemCompRedirectNotCached {
+
+ private HttpServer server;
+ private String originalServerUrl;
+
+ @Before
+ public void setUp() {
+ originalServerUrl = DownloadChemCompProvider.serverBaseUrl;
+ }
+
+ @After
+ public void tearDown() {
+ if (server != null) {
+ server.stop(0);
+ }
+ // Static state: leaving either of these set would corrupt unrelated tests.
+ DownloadChemCompProvider.serverBaseUrl = originalServerUrl;
+ FlatFileCache.clear();
+ }
+
+ /**
+ * Starts a local server that answers every request with the given status and body.
+ *
+ * @return the base URL to point the provider at
+ */
+ private String startServer(int status, String location, String body) throws IOException {
+ server = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0);
+ server.createContext("/", exchange -> {
+ byte[] bytes = body.getBytes(StandardCharsets.UTF_8);
+ if (location != null) {
+ exchange.getResponseHeaders().add("Location", location);
+ }
+ exchange.sendResponseHeaders(status, bytes.length);
+ try (OutputStream out = exchange.getResponseBody()) {
+ out.write(bytes);
+ }
+ });
+ server.start();
+ return "http://127.0.0.1:" + server.getAddress().getPort() + "/";
+ }
+
+ private File cacheFileFor(String id) {
+ File file = new File(DownloadChemCompProvider.getLocalFileName(id));
+ file.delete();
+ FlatFileCache.clear();
+ return file;
+ }
+
+ /**
+ * The case that broke CATH: a redirect the JDK will not follow because it
+ * changes protocol. Its body must not end up on disk under the component's name.
+ */
+ @Test
+ public void redirectBodyIsNotCached() throws IOException {
+ File cached = cacheFileFor("ATP");
+ DownloadChemCompProvider.serverBaseUrl =
+ startServer(301, "https://example.invalid/ATP.cif", "Moved Permanently");
+
+ ChemComp cc = new DownloadChemCompProvider().getChemComp("ATP");
+
+ assertFalse("the body of a redirect must never be cached as a definition", cached.exists());
+ assertNull("nothing parseable was returned, so the component must be empty", cc.getName());
+ }
+
+ /**
+ * A 200 is still cached, so the guard has not simply disabled downloading.
+ *
+ * What is under test is the download path, not the CIF parser: the response is
+ * written to the cache before anything tries to parse it, so a parse failure on
+ * this deliberately minimal body says nothing about whether the guard behaved.
+ */
+ @Test
+ public void aValidResponseIsStillCached() throws IOException {
+ File cached = cacheFileFor("ATP");
+ DownloadChemCompProvider.serverBaseUrl = startServer(200, null,
+ "data_ATP\n#\n_chem_comp.id ATP\n_chem_comp.name \"ADENOSINE-5'-TRIPHOSPHATE\"\n#\n");
+
+ try {
+ new DownloadChemCompProvider().getChemComp("ATP");
+ } catch (RuntimeException parseFailure) {
+ // see the note above
+ }
+
+ assertTrue("a 200 response should still be cached", cached.exists());
+ cached.delete();
+ }
+
+ /** A server error must not be cached either. */
+ @Test
+ public void serverErrorBodyIsNotCached() throws IOException {
+ File cached = cacheFileFor("ATP");
+ DownloadChemCompProvider.serverBaseUrl =
+ startServer(503, null, "Service Unavailable");
+
+ new DownloadChemCompProvider().getChemComp("ATP");
+
+ assertFalse("the body of a 5xx must never be cached as a definition", cached.exists());
+ }
+}
diff --git a/biojava-structure/src/test/java/org/biojava/nbio/structure/ecod/EcodParserTest.java b/biojava-structure/src/test/java/org/biojava/nbio/structure/ecod/EcodParserTest.java
new file mode 100644
index 0000000000..295806e915
--- /dev/null
+++ b/biojava-structure/src/test/java/org/biojava/nbio/structure/ecod/EcodParserTest.java
@@ -0,0 +1,323 @@
+/*
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the
+ * terms of the GNU Lesser General Public Licence. This should
+ * be distributed with the code. If you do not have a copy,
+ * see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual
+ * authors. These should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims,
+ * or to join the biojava-l mailing list, visit the home page
+ * at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.ecod;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import java.io.IOException;
+import java.io.StringReader;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.LinkedHashSet;
+import java.util.List;
+
+import org.biojava.nbio.structure.ecod.EcodInstallation.EcodParser;
+import org.junit.jupiter.api.Nested;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Checks that {@link EcodParser} reads every layout ECOD has distributed.
+ *
+ * The columns have been renamed, reordered and added to several times, and the version
+ * comment itself changed form at v294.1. Because the full distribution is 657 MB, none of
+ * that was covered by a test that could run in reasonable time, and a format change went
+ * unnoticed for months. These fixtures are taken verbatim from the real files, so the
+ * contract is pinned in milliseconds rather than by a download.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+class EcodParserTest {
+
+ /** develop204, list format 1.5: 15 columns, commented header, quoted names. */
+ private static final String DEVELOP204 = String.join("\n",
+ "#/data/ecod/database_versions/v204/ecod.develop204.domains.txt",
+ "#ECOD version develop204",
+ "#Domain list version 1.5",
+ "#Grishin lab (http://prodata.swmed.edu/ecod)",
+ "#uid\tecod_domain_id\tmanual_rep\tf_id\tpdb\tchain\tpdb_range\tseqid_range"
+ + "\tarch_name\tx_name\th_name\tt_name\tf_name\tasm_status\tligand",
+ "002137905\te6b4nA1\tAUTO_NONREP\t1.1.1\t6b4n\tA\tA:1-99\tA:1-99\tbeta barrels"
+ + "\t\"cradle loop barrel\"\t\"RIFT-related\"\t\"acid protease\""
+ + "\tF_UNCLASSIFIED\tNOT_DOMAIN_ASSEMBLY\tCL,G53,NA");
+
+ /** develop291, list format 1.6: 16 columns, f_id renamed t_id, unp_acc inserted at 9. */
+ private static final String DEVELOP291 = String.join("\n",
+ "#/data/ecod/database_versions/v291/ecod.develop291.domains.txt",
+ "#ECOD version develop291",
+ "#Domain list version 1.6",
+ "#Grishin lab (http://prodata.swmed.edu/ecod)",
+ "#uid\tecod_domain_id\tmanual_rep\tt_id\tpdb\tchain\tpdb_range\tseqid_range\tunp_acc"
+ + "\tarch_name\tx_name\th_name\tt_name\tf_name\tasm_status\tligand",
+ "000000267\te1udzA1\tMANUAL_REP\t1.1.1\t1udz\tA\tA:203-381\tA:4-182\tP12345"
+ + "\tbeta barrels\t\"cradle loop barrel\"\t\"RIFT-related\"\t\"acid protease\""
+ + "\tF_UNCLASSIFIED\tNOT_DOMAIN_ASSEMBLY\tNO_LIGANDS_4A");
+
+ private static final String V295_COLUMNS =
+ "uid\tecod_domain_id\tmanual_rep\tf_id\tpdb\tchain\tpdb_range\tseqid_range"
+ + "\tarchitecture_name\tx_name\th_name\tt_name\tf_name\tassembly_id\tdomain_id_short"
+ + "\trange_count\tarch_manual\tx_manual\th_manual\tt_manual\tf_manual"
+ + "\tvalid_structure\tligand_binding\tligand_comp_ids\tligand_pdbnum";
+
+ /** v295: 25 columns, uncommented header, True/False, empty assembly_id, moved ligands. */
+ private static final String V295 = String.join("\n",
+ "# ECOD Domain List",
+ "# Version: v295",
+ "# Generated: 2026-06-24 22:47:42",
+ "# Ligand cutoff: 4.0 A (NO_LIGANDS_4A = no contact within cutoff)",
+ "#",
+ V295_COLUMNS,
+ "0\te2nmzA1\tTrue\t1.1.1.3\t2nmz\tA\tA:1-99\tA:1-99\tbeta barrels\tcradle loop barrel"
+ + "\tRIFT-related\tacid protease\tRVP\t\t\t1\tFalse\tFalse\tFalse\tFalse\tTrue"
+ + "\tTrue\tTrue\tROC,SO4\tA:601,A:602,B:401",
+ // the last column is empty on four rows in five, so split() must keep it
+ "3\te2rspA1\tTrue\t1.1.1.3\t2rsp\tA\tA:1-124\tA:1-124\tbeta barrels\tcradle loop barrel"
+ + "\tRIFT-related\tacid protease\tRVP\t\t\t1\tFalse\tFalse\tFalse\tFalse\tTrue"
+ + "\tTrue\tFalse\tNO_LIGANDS_4A\t",
+ // a domain classified from an AlphaFold model: no PDB entry, so no EcodDomain
+ "3163557\tP44140_F1_nD2\tFalse\t2004.1.1.123\t\t\t131-315\t131-315\talpha bundles"
+ + "\tsomething\tsomething else\ta third thing\t\t\t\t1\tFalse\tFalse\tFalse"
+ + "\tFalse\tTrue\tTrue\tFalse\tNO_LIGANDS_4A\t");
+
+ private static List
+ * isosurface ID "bj_density_2fofc" delete
+ *
+ * to remove just the blue surface. Pressing Reset Display keeps the maps, since
+ * they are folded into the saved state when they are drawn.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DemoShowElectronDensity {
+
+ /**
+ * @param args an optional PDB ID to display instead of the default
+ * @throws Exception if the structure or the map could not be fetched
+ */
+ public static void main(String[] args) throws Exception {
+ String id = args.length > 0 ? args[0] : "1cbs";
+
+ Structure structure = StructureIO.getStructure(id);
+ StructureAlignmentJmol viewer = new StructureAlignmentJmol();
+ viewer.setStructure(structure);
+ viewer.evalString("select all; cartoon on; color chain; "
+ + "select ligand; wireframe 0.16; spacefill 0.4; color cpk;");
+
+ DensityMapCache cache = new DensityMapCache();
+ System.out.println("Density cache: " + cache.getCachePath());
+
+ // Clipping to the ligand keeps the surface readable and, for a large map,
+ // keeps the contouring quick enough not to stall the interface.
+ for (DensityMapKind kind : new DensityMapKind[] {DensityMapKind.TWO_FO_FC, DensityMapKind.FO_FC}) {
+ cache.findDensityMap(new PdbId(id), kind).ifPresent(map -> {
+ System.out.printf("%-8s from %-20s %s (%,d bytes)%n",
+ map.getKind(), map.getSource(), map.getFile().getName(), map.getFileSizeBytes());
+ viewer.getJmolPanel().loadDensityMap(map, "{ligand}", 5.0);
+ });
+ }
+
+ DensityMapResult any = cache.findDensityMap(new PdbId(id), DensityMapKind.AUTO).orElse(null);
+ if (any == null) {
+ System.out.println("No density is available for " + id
+ + " - try an entry with deposited structure factors, such as 1cbs.");
+ }
+ }
+}
diff --git a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MenuCreator.java b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MenuCreator.java
index 6a0e3c6e1a..197802a2c5 100644
--- a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MenuCreator.java
+++ b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MenuCreator.java
@@ -66,6 +66,8 @@ public class MenuCreator {
public static final String PAIRWISE_ALIGN = "New Pairwise Alignment";
public static final String MULTIPLE_ALIGN = "New Multiple Alignment";
public static final String PHYLOGENETIC_TREE = "Phylogenetic Tree";
+ /** @since 7.3.0 */
+ public static final String SHOW_DENSITY = "Show Electron Density";
protected static final int keyMask =
Toolkit.getDefaultToolkit().getMenuShortcutKeyMask();
@@ -169,6 +171,8 @@ public static JMenuBar initJmolMenu(JFrame frame,
distMax.setMnemonic(KeyEvent.VK_D);
distMax.addActionListener(new MyDistMaxListener(parent));
view.add(distMax);
+ //Electron density
+ view.add(getShowDensityMenuItem(parent));
//Dot Plot - only if the alignment was an afpChain
if (afpChain != null){
JMenuItem dotplot = new JMenuItem(DOT_PLOT);
@@ -229,6 +233,22 @@ public static JMenuItem getOpenPDBMenuItem() {
return openI;
}
+ /**
+ * Menu item that fetches and displays the electron density or cryo-EM map for
+ * the structure currently on screen.
+ *
+ * @param parent the viewer to draw the map into
+ * @return the menu item
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ public static JMenuItem getShowDensityMenuItem(AbstractAlignmentJmol parent) {
+ JMenuItem densityI = new JMenuItem(SHOW_DENSITY);
+ densityI.setMnemonic(KeyEvent.VK_E);
+ densityI.addActionListener(new MyShowDensityListener(parent));
+ return densityI;
+ }
+
public static JMenuItem getLoadMenuItem() {
diff --git a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MyShowDensityListener.java b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MyShowDensityListener.java
new file mode 100644
index 0000000000..b1a9cf23ff
--- /dev/null
+++ b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/MyShowDensityListener.java
@@ -0,0 +1,202 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.align.gui;
+
+import java.awt.event.ActionEvent;
+import java.awt.event.ActionListener;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ExecutionException;
+
+import javax.swing.JOptionPane;
+import javax.swing.SwingWorker;
+
+import org.biojava.nbio.structure.PdbId;
+import org.biojava.nbio.structure.Structure;
+import org.biojava.nbio.structure.align.gui.jmol.AbstractAlignmentJmol;
+import org.biojava.nbio.structure.io.density.DensityMapCache;
+import org.biojava.nbio.structure.io.density.DensityMapKind;
+import org.biojava.nbio.structure.io.density.DensityMapRequest;
+import org.biojava.nbio.structure.io.density.DensityMapResult;
+import org.biojava.nbio.structure.io.density.DensityMapSource;
+import org.biojava.nbio.structure.io.density.NoDensityMapException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Fetches the density map for the structure on screen and draws it.
+ * null before the window is built
+ * @since 7.3.0
+ */
+ public JFrame getFrame() {
+ return frame;
+ }
+
+ /**
+ * Writes a short message into the viewer's status field.
+ *
+ * @param message the message; ignored if there is no status field yet
+ * @since 7.3.0
+ */
+ public void setStatus(String message) {
+ if (status != null) {
+ status.setText(message);
+ }
+ }
+
/**
* Set the title of the AlignmentJmol window.
* @param title
diff --git a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/JmolPanel.java b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/JmolPanel.java
index 23361e62ed..9108489d29 100644
--- a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/JmolPanel.java
+++ b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/JmolPanel.java
@@ -31,12 +31,14 @@
import java.awt.event.ActionEvent;
import java.awt.event.ActionListener;
import java.io.BufferedInputStream;
+import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.PipedInputStream;
import java.io.PipedOutputStream;
import java.text.DecimalFormat;
import java.util.List;
+import java.util.Locale;
import javax.swing.JComboBox;
@@ -50,6 +52,8 @@
import org.biojava.nbio.structure.domain.pdp.Domain;
import org.biojava.nbio.structure.domain.pdp.Segment;
import org.biojava.nbio.structure.gui.util.color.ColorUtils;
+import org.biojava.nbio.structure.io.density.DensityMapKind;
+import org.biojava.nbio.structure.io.density.DensityMapResult;
import org.biojava.nbio.structure.io.mmtf.MmtfActions;
import org.biojava.nbio.structure.jama.Matrix;
import org.biojava.nbio.structure.scop.ScopDatabase;
@@ -166,6 +170,157 @@ public void setStructure(final Structure s) {
setStructure(s, false);
}
+ /** Isosurface id used for the 2mFo-DFc map, so that it can be addressed on its own. */
+ public static final String ISOSURFACE_ID_2FOFC = "bj_density_2fofc";
+
+ /** Isosurface id used for the mFo-DFc difference map, drawn as a signed pair of lobes. */
+ public static final String ISOSURFACE_ID_FOFC = "bj_density_fofc";
+
+ /** Isosurface id used for a cryo-EM map. */
+ public static final String ISOSURFACE_ID_EM = "bj_density_em";
+
+ /** Default clipping radius, in Angstroms, around the selected atoms. */
+ public static final double DEFAULT_WITHIN_RADIUS = 5.0;
+
+ /**
+ * Displays a density map fetched through
+ * {@link org.biojava.nbio.structure.io.density.DensityMapCache}, clipped to
+ * {@value #DEFAULT_WITHIN_RADIUS} Angstroms around the whole model.
+ *
+ * @param map the map to display
+ * @throws IllegalArgumentException if the map cannot be displayed without a
+ * Fourier transform first
+ * @since 7.3.0
+ */
+ public void loadDensityMap(DensityMapResult map) {
+ loadDensityMap(map, "{*}", DEFAULT_WITHIN_RADIUS);
+ }
+
+ /**
+ * Displays a density map, clipped to a distance around a selection.
+ * {*} or
+ * {ligand}, or null to contour the whole cell
+ * @param withinRadius the clipping radius in Angstroms, ignored when
+ * atomSelection is null
+ * @throws IllegalArgumentException if the map cannot be displayed without a
+ * Fourier transform first
+ * @since 7.3.0
+ */
+ public void loadDensityMap(DensityMapResult map, String atomSelection, double withinRadius) {
+ if (!map.isRenderable()) {
+ throw new IllegalArgumentException("A " + map.getFormat() + " file holds structure factors, not a "
+ + "sampled map, and cannot be displayed as it stands. Convert it first, for example with "
+ + "'gemmi sf2map', or fetch the map from a source that serves a grid.");
+ }
+
+ Double level = map.getRecommendedContourLevel();
+ if (map.getKind() == DensityMapKind.EM && level != null) {
+ // EM maps are conventionally contoured at the absolute level the
+ // depositors chose rather than at a multiple of sigma.
+ loadDensityMap(map.getFile(), map.getKind(), level, false, atomSelection, withinRadius);
+ } else {
+ double sigma = map.getKind() == DensityMapKind.TWO_FO_FC ? 1.0 : 3.0;
+ loadDensityMap(map.getFile(), map.getKind(), sigma, true, atomSelection, withinRadius);
+ }
+ }
+
+ /**
+ * Displays a density map file directly.
+ *
+ * @param mapFile the map file, in a format Jmol can contour (CCP4/MRC, or a
+ * BinaryCIF volume)
+ * @param kind what the map values mean, which decides the colouring and whether
+ * a signed pair of surfaces is drawn
+ * @param level the contour level
+ * @param levelIsSigma whether level is a multiple of the map's RMS
+ * deviation rather than an absolute value
+ * @param atomSelection a Jmol atom expression, or null to contour
+ * the whole cell
+ * @param withinRadius the clipping radius in Angstroms
+ * @since 7.3.0
+ */
+ public void loadDensityMap(File mapFile, DensityMapKind kind, double level, boolean levelIsSigma,
+ String atomSelection, double withinRadius) {
+
+ String url = toJmolFileUrl(mapFile);
+ String within = atomSelection == null ? ""
+ : String.format(Locale.US, " within %.1f %s", withinRadius, atomSelection);
+
+ if (kind == DensityMapKind.FO_FC) {
+ // One signed surface carrying both lobes, red for negative and green for
+ // positive. Drawing the negative lobe as a separate surface at "sigma -3"
+ // does not work: Jmol gives a negative sigma its own internal meaning and
+ // silently contours at the default level instead.
+ evalString(isosurfaceCommand(ISOSURFACE_ID_FOFC, "sign red green", level, levelIsSigma, within, url));
+ } else if (kind == DensityMapKind.EM) {
+ evalString(isosurfaceCommand(ISOSURFACE_ID_EM, "color grey", level, levelIsSigma, within, url));
+ } else {
+ evalString(isosurfaceCommand(ISOSURFACE_ID_2FOFC, "color blue", level, levelIsSigma, within, url));
+ }
+
+ // resetDisplay() restores "state_1", which is saved when the structure is
+ // loaded. Without re-saving here, pressing Reset Display would silently
+ // discard the map the user just asked for.
+ evalString("save STATE state_1");
+ }
+
+ /**
+ * Builds an isosurface command.
+ * mesh and
+ * nofill have to follow the file name. Placed before it, Jmol
+ * accepts the command without complaint and draws nothing at all.
+ */
+ private static String isosurfaceCommand(String id, String colouring, double level, boolean levelIsSigma,
+ String within, String url) {
+ return String.format(Locale.US,
+ "isosurface ID \"%s\" delete; isosurface ID \"%s\" %s %s %.4f%s \"%s\" mesh nofill;",
+ id, id, colouring, levelIsSigma ? "sigma" : "cutoff", level, within, url);
+ }
+
+ /**
+ * Removes any density surfaces this panel has drawn, leaving other isosurfaces
+ * alone.
+ *
+ * @since 7.3.0
+ */
+ public void clearDensityMaps() {
+ for (String id : new String[] {ISOSURFACE_ID_2FOFC, ISOSURFACE_ID_FOFC, ISOSURFACE_ID_EM}) {
+ evalString("isosurface ID \"" + id + "\" delete;");
+ }
+ evalString("save STATE state_1");
+ }
+
+ /**
+ * Converts a file to the URL form Jmol expects.
+ * file:/C:/...,
+ * which is normalised here to the usual three-slash form.
+ *
+ * @param file the file
+ * @return a URL string safe to embed in a Jmol script
+ * @since 7.3.0
+ */
+ public static String toJmolFileUrl(File file) {
+ String url = file.getAbsoluteFile().toURI().toString();
+ if (url.startsWith("file:/") && !url.startsWith("file://")) {
+ url = "file:///" + url.substring("file:/".length());
+ }
+ return url;
+ }
+
/** assign a custom color to the Jmol chains command.
*
*/
diff --git a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/RasmolCommandListener.java b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/RasmolCommandListener.java
index d1ce5c0e30..feec8b0366 100644
--- a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/RasmolCommandListener.java
+++ b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/align/gui/jmol/RasmolCommandListener.java
@@ -74,7 +74,7 @@ public void actionPerformed(ActionEvent event) {
// check last command in history
// if equivalent, don't add,
// otherwise add
- if (history.size()>0){
+ if (!history.isEmpty()){
String txt=history.get(history.size()-1);
if (! txt.equals(cmd)) {
history.add(cmd);
diff --git a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/gui/util/SequenceScalePanel.java b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/gui/util/SequenceScalePanel.java
index 06542e5271..75da6c8e28 100644
--- a/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/gui/util/SequenceScalePanel.java
+++ b/biojava-structure-gui/src/main/java/org/biojava/nbio/structure/gui/util/SequenceScalePanel.java
@@ -126,7 +126,7 @@ private void setPrefSize() {
public void setAligMap(List
+ *
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DemoFetchElectronDensity {
+
+ /**
+ * @param args ignored
+ * @throws IOException if a server could not be reached at all
+ */
+ public static void main(String[] args) throws IOException {
+ DensityMapCache cache = new DensityMapCache();
+ System.out.println("Caching under: " + cache.getCachePath());
+ System.out.println();
+
+ show(cache, "1cbs", DensityMapKind.TWO_FO_FC);
+ show(cache, "1cbs", DensityMapKind.FO_FC);
+ show(cache, "6hu9", DensityMapKind.AUTO);
+ show(cache, "4hhb", DensityMapKind.AUTO);
+ }
+
+ private static void show(DensityMapCache cache, String id, DensityMapKind kind) throws IOException {
+ System.out.printf("%s (%s)%n", id, kind);
+ try {
+ DensityMapResult result = cache.getDensityMap(new PdbId(id), kind);
+ System.out.printf(" source : %s%n", result.getSource());
+ System.out.printf(" format : %s%s%n", result.getFormat(),
+ result.isRenderable() ? "" : " (needs an FFT before display)");
+ System.out.printf(" kind : %s%n", result.getKind());
+ System.out.printf(" file : %s (%,d bytes)%n", result.getFile(), result.getFileSizeBytes());
+ System.out.printf(" cached : %s%n", result.isFromCache());
+ if (result.getEmdbId() != null) {
+ System.out.printf(" EMDB : %s%n", result.getEmdbId());
+ }
+ if (result.getRecommendedContourLevel() != null) {
+ System.out.printf(" contour : %s (author recommended)%n", result.getRecommendedContourLevel());
+ }
+ if (result.getContourInSigma() != null) {
+ System.out.printf(" in sigma : %.2f%n", result.getContourInSigma());
+ }
+ } catch (NoDensityMapException e) {
+ System.out.printf(" no map available%n");
+ e.getAttempts().forEach((source, reason) ->
+ System.out.printf(" %-24s %s%n", source, reason));
+ }
+ System.out.println();
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/Author.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/Author.java
index b0d7253507..bd5a01b885 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/Author.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/Author.java
@@ -62,7 +62,7 @@ public boolean equals(Object obj) {
if ((this.surname == null) ? (other.surname != null) : !this.surname.equals(other.surname)) {
return false;
}
- return !((this.initials == null) ? (other.initials != null) : !this.initials.equals(other.initials));
+ return (this.initials == null) ? other.initials == null : this.initials.equals(other.initials);
}
@Override
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/Element.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/Element.java
index 2f534b2828..4e2d3e340a 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/Element.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/Element.java
@@ -424,7 +424,7 @@ public boolean isHeavyAtom() {
* @return true if Element is not Hydrogen and not Carbon.
*/
public boolean isHeteroAtom() {
- return !(this == C || this == H);
+ return this != C && this != H;
}
/**
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/Site.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/Site.java
index 9158906d23..341483f31b 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/Site.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/Site.java
@@ -83,7 +83,7 @@ public String toPDB() {
@Override
public void toPDB(StringBuffer buf) {
- if (groups == null || groups.size() < 1) {
+ if (groups == null || groups.isEmpty()) {
return;
}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ClusterAltAligs.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ClusterAltAligs.java
index 373bcf1611..0933198d7b 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ClusterAltAligs.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ClusterAltAligs.java
@@ -102,7 +102,7 @@ public static void cluster(AlternativeAlignment[] aligs, int cutoff){
}
clusters.add(currentCluster);
- if ( remainList.size() == 0) {
+ if ( remainList.isEmpty()) {
break;
}
}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ce/CECalculator.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ce/CECalculator.java
index 6c045ba48e..83f16b7982 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ce/CECalculator.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/align/ce/CECalculator.java
@@ -1450,7 +1450,7 @@ private int optimizeSuperposition(AFPChain afpChain, int nse1, int nse2, int str
//afpChain.setTotalRmsdOpt(rmsd);
//System.out.println("rmsd: " + rmsd);
- if(!(nAtom
+ *
+ * StructureTools.getAtomsInContact(Chain, double)), the atom that comes first in the
+ * atom array is the first member of the pairStructureTools.getAtomsInContact(Chain, Chain, double, boolean), or a
+ * {@link StructureInterface}), the atom belonging to the first set is the first member of the pair#uid<tab>ecod_domain_id<tab>...) up to develop292 and
+ * uncommented (uid<tab>ecod_domain_id<tab>...) from v294.1
+ * onwards. Because names have also been changed between versions, lookups accept
+ * aliases and any name the file does not declare simply reads as absent.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+ private static class ColumnLayout {
+ private final Mapcb for 1cbs.
+ * 1cbs and its extended form pdb_00001cbs both
+ * give cb. Taking them from the start would file the extended form
+ * under db instead. The extended PDB identifier format is expected
+ * to keep using this same hashing scheme.
+ *
+ * @param pdbId a PDB identifier, in either the short or the extended form
+ * @return the lowercase two-character directory name
+ * @since 7.3.0
+ */
+ public static String getMiddleHash(String pdbId) {
+ int offset = pdbId.length() - 3;
+ return pdbId.substring(offset, offset + 2).toLowerCase();
+ }
+
/**
* Get the last modified time of the file in given url by retrieveing the "Last-Modified" header.
* Note that this only works for http URLs
* @param url
* @return the last modified date or null if it couldn't be retrieved (in that case a warning will be logged)
+ * @since 7.3.0 made public so that other caching code can reuse it
*/
- private Date getLastModifiedTime(URL url) {
+ public static Date getLastModifiedTime(URL url) {
// see http://stackoverflow.com/questions/2416872/how-do-you-obtain-modified-date-from-a-remote-file-java
Date date = null;
@@ -629,14 +653,12 @@ private Date getLastModifiedTime(URL url) {
protected File getDir(String pdbId, boolean obsolete) {
File dir = null;
- int offset = pdbId.length() - 3;
+ String middle = getMiddleHash(pdbId);
if (obsolete) {
// obsolete is always split
- String middle = pdbId.substring(offset, offset + 2).toLowerCase();
dir = new File(obsoleteDirPath, middle);
} else {
- String middle = pdbId.substring(offset, offset + 2).toLowerCase();
dir = new File(splitDirPath, middle);
}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/PDBFileParser.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/PDBFileParser.java
index 176459bbf2..b1d327599e 100644
--- a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/PDBFileParser.java
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/PDBFileParser.java
@@ -1406,7 +1406,7 @@ public void handleResolutionLine(String line, Pattern pR) {
try {
float res = Float.parseFloat(resString);
final float resInHeader = pdbHeader.getResolution();
- if (resInHeader!=PDBHeader.DEFAULT_RESOLUTION && resInHeader != res) {
+ if (resInHeader!=PDBHeader.DEFAULT_RESOLUTION && Math.abs(resInHeader - res) > 0.001) {
logger.warn("More than 1 resolution value present, will use last one {} and discard previous {} "
,resString, String.format("%4.2f",resInHeader));
}
@@ -1943,7 +1943,7 @@ private Group getCorrectAltLocGroup( Character altLoc,
// see if we know this altLoc already;
Listnull
+ * @param contourLevel the author-recommended contour level, or null
+ * @param sigma the map RMS deviation, or null
+ * @return the result, or null if the source has nothing for this entry
+ * @throws IOException on transport failure
+ */
+ protected DensityMapResult obtain(DensityMapRequest request, URL url, File target, DensityMapKind kind,
+ String emdbId, Double contourLevel, Double sigma) throws IOException {
+
+ FetchBehavior behavior = effectiveFetchBehavior(request);
+
+ if (isCacheUsable(target, url, behavior)) {
+ // The kind always comes from the request, never from the sidecar: a single
+ // cached file can hold more than one kind of map, so the sidecar's kind
+ // records what was asked for first, not what the caller wants now.
+ DensityMapResult cached = DensityMapResult.readMeta(target);
+ Double cachedContour = contourLevel != null || cached == null ? contourLevel
+ : cached.getRecommendedContourLevel();
+ Double cachedSigma = sigma != null || cached == null ? sigma : cached.getSigma();
+ DensityMapResult result = new DensityMapResult(target, getSource(), getFormat(), kind,
+ request.getPdbId(), emdbId, url.toString(), true, cachedContour, cachedSigma);
+ if (cached == null) {
+ // The file is good but its description was lost; write one rather than
+ // downloading several megabytes again.
+ result.writeMeta();
+ }
+ return result;
+ }
+
+ if (behavior == FetchBehavior.LOCAL_ONLY) {
+ throw new HttpStatusException(404, url.toString(),
+ "not in the local cache and downloads are disabled (FetchBehavior.LOCAL_ONLY)");
+ }
+
+ File dir = target.getAbsoluteFile().getParentFile();
+ if (!dir.isDirectory() && !dir.mkdirs()) {
+ throw new IOException("Could not create density cache directory " + dir);
+ }
+
+ enforceSizeLimit(url, effectiveMaxBytes(request), request);
+
+ logger.info("Fetching {} density map for {} from {}", kind,
+ request.getPdbId() == null ? emdbId : request.getPdbId().getId(), url);
+ FileDownloadUtils.downloadFileWithValidation(url, target, null, FileDownloadUtils.Hash.UNKNOWN,
+ FileDownloadUtils.ETagPolicy.USE_IF_HEX_DIGEST);
+
+ if (!isPlausibleMap(target)) {
+ // Do not leave a bad file behind to be picked up as a cache hit later.
+ deleteWithSidecars(target);
+ throw new IOException("The content downloaded from " + url + " is not a usable "
+ + getFormat() + " density map.");
+ }
+
+ DensityMapResult result = new DensityMapResult(target, getSource(), getFormat(), kind,
+ request.getPdbId(), emdbId, url.toString(), false, contourLevel, sigma);
+ result.writeMeta();
+ return result;
+ }
+
+ /**
+ * Whether a cached file may be used as-is.
+ *
+ * @param target the cached file
+ * @param url where it came from
+ * @param behavior the fetch behaviour in force
+ * @return true if the cached file should be served
+ */
+ protected boolean isCacheUsable(File target, URL url, FetchBehavior behavior) {
+ if (behavior == FetchBehavior.FORCE_DOWNLOAD) {
+ return false;
+ }
+ if (!target.isFile() || target.length() < MIN_DENSITY_FILE_SIZE) {
+ return false;
+ }
+ if (!FileDownloadUtils.validateFile(target)) {
+ logger.info("Cached density map [{}] failed validation and will be re-downloaded.", target);
+ return false;
+ }
+ if (!isPlausibleMap(target)) {
+ logger.info("Cached file [{}] is not a usable {} map and will be re-downloaded.", target, getFormat());
+ return false;
+ }
+ if (behavior == FetchBehavior.LOCAL_ONLY) {
+ return true;
+ }
+ if (behavior == FetchBehavior.FETCH_IF_OUTDATED) {
+ Date serverDate = LocalPDBDirectory.getLastModifiedTime(url);
+ if (serverDate == null) {
+ // Density servers generate their responses on the fly and send no
+ // Last-Modified header. Treating that as "outdated" would re-download
+ // on every single call, so an unknown timestamp keeps the cache.
+ logger.debug("No server timestamp for {}; keeping the cached copy.", url);
+ return true;
+ }
+ return target.lastModified() >= serverDate.getTime();
+ }
+ // FETCH_FILES, and FETCH_REMEDIATED which has no meaning for maps.
+ return true;
+ }
+
+ /**
+ * Checks that a file looks like the format this provider delivers. Only the
+ * CCP4/MRC formats carry a recognisable stamp; the others are accepted on size
+ * alone.
+ *
+ * @param file the file to check
+ * @return true if the file is plausibly a map of this format
+ */
+ protected boolean isPlausibleMap(File file) {
+ if (file == null || file.length() < MIN_DENSITY_FILE_SIZE) {
+ return false;
+ }
+ DensityFileFormat format = getFormat();
+ if (format == DensityFileFormat.CCP4 || format == DensityFileFormat.CCP4_GZ) {
+ return Ccp4Header.isCcp4Quietly(file);
+ }
+ return true;
+ }
+
+ /**
+ * Refuses a download whose declared size exceeds the limit, before any of the
+ * body is transferred.
+ *
+ * @param url the resource
+ * @param maxBytes the limit, or 0 for no limit
+ * @param request the request being served, used for reporting
+ * @throws DensityMapTooLargeException if the resource is too large
+ */
+ protected void enforceSizeLimit(URL url, long maxBytes, DensityMapRequest request)
+ throws DensityMapTooLargeException {
+ if (maxBytes <= 0) {
+ return;
+ }
+ long size = declaredSize(url);
+ if (size > maxBytes) {
+ reportTooLarge(url, size, maxBytes, request);
+ throw new DensityMapTooLargeException(url.toString(), size, maxBytes);
+ }
+ }
+
+ /**
+ * Announces that a map was skipped for being too large.
+ * Content-Length a server declares for a resource.
+ *
+ * @param url the resource
+ * @return the size in bytes, or a negative value if the server did not say
+ */
+ protected long declaredSize(URL url) {
+ try {
+ URLConnection connection = FileDownloadUtils.prepareURLConnection(url.toString(), 30000);
+ if (connection instanceof java.net.HttpURLConnection) {
+ ((java.net.HttpURLConnection) connection).setRequestMethod("HEAD");
+ }
+ connection.connect();
+ try {
+ return connection.getContentLengthLong();
+ } finally {
+ if (connection instanceof java.net.HttpURLConnection) {
+ ((java.net.HttpURLConnection) connection).disconnect();
+ }
+ }
+ } catch (IOException e) {
+ logger.debug("Could not determine the size of {}: {}", url, e.getMessage());
+ return -1;
+ }
+ }
+
+ /**
+ * Deletes a cached map along with its validation and metadata sidecars.
+ *
+ * @param target the cached map
+ */
+ protected void deleteWithSidecars(File target) {
+ File dir = target.getAbsoluteFile().getParentFile();
+ if (dir == null) {
+ return;
+ }
+ File[] siblings = dir.listFiles((d, name) -> name.startsWith(target.getName()));
+ if (siblings == null) {
+ return;
+ }
+ for (File f : siblings) {
+ if (!f.delete()) {
+ logger.debug("Could not delete [{}]", f);
+ }
+ }
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/Ccp4Header.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/Ccp4Header.java
new file mode 100644
index 0000000000..594ad91927
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/Ccp4Header.java
@@ -0,0 +1,133 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.util.zip.GZIPInputStream;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Recognises CCP4/MRC map files by their header.
+ * MAP
+ * stamp does, turning a silent failure into a clean cache miss.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class Ccp4Header {
+
+ private static final Logger logger = LoggerFactory.getLogger(Ccp4Header.class);
+
+ /**
+ * Byte offset of the four-character format stamp within a CCP4/MRC header. It
+ * sits in word 53 of the 256-word header.
+ */
+ public static final int MAP_STAMP_OFFSET = 208;
+
+ /** The stamp itself: the three letters of "MAP" followed by a space. */
+ public static final String MAP_STAMP = "MAP ";
+
+ /** Number of header bytes that must be readable for the check to be possible. */
+ private static final int HEADER_BYTES = MAP_STAMP_OFFSET + 4;
+
+ private Ccp4Header() {
+ }
+
+ /**
+ * Checks whether a file is a CCP4/MRC map. Gzipped files are decompressed on
+ * the fly, so .map.gz works as well as .ccp4.
+ *
+ * @param file the file to check
+ * @return true if the CCP4 stamp is present
+ * @throws IOException if the file could not be read
+ */
+ public static boolean isCcp4(File file) throws IOException {
+ if (file == null || !file.isFile()) {
+ return false;
+ }
+ // Deliberately no shortcut on file.length(): a gzipped map compresses to far
+ // less than the size of the header it contains, so a length test here would
+ // reject perfectly good small maps. Reading decides it instead.
+ try (InputStream in = openPossiblyGzipped(file)) {
+ return isCcp4(in);
+ }
+ }
+
+ /**
+ * Checks whether a stream carries a CCP4/MRC map header. The stream is read
+ * from its current position and is not closed; it is not un-read afterwards,
+ * so pass a fresh stream or one that supports marking.
+ *
+ * @param in the stream to check, already decompressed
+ * @return true if the CCP4 stamp is present
+ * @throws IOException if the stream could not be read
+ */
+ public static boolean isCcp4(InputStream in) throws IOException {
+ byte[] header = new byte[HEADER_BYTES];
+ int read = 0;
+ while (read < HEADER_BYTES) {
+ int n = in.read(header, read, HEADER_BYTES - read);
+ if (n < 0) {
+ return false; // shorter than a CCP4 header, so certainly not one
+ }
+ read += n;
+ }
+ String stamp = new String(header, MAP_STAMP_OFFSET, 4, StandardCharsets.US_ASCII);
+ return MAP_STAMP.equals(stamp);
+ }
+
+ /**
+ * Same as {@link #isCcp4(File)} but reports a problem rather than propagating
+ * it, for use in cache-validity checks where an unreadable file and an invalid
+ * one lead to the same action.
+ *
+ * @param file the file to check
+ * @return true if the file is readable and carries the CCP4 stamp
+ */
+ public static boolean isCcp4Quietly(File file) {
+ try {
+ return isCcp4(file);
+ } catch (IOException e) {
+ logger.debug("Could not read [{}] to check for a CCP4 header: {}", file, e.getMessage());
+ return false;
+ }
+ }
+
+ private static InputStream openPossiblyGzipped(File file) throws IOException {
+ InputStream in = new BufferedInputStream(Files.newInputStream(file.toPath()));
+ in.mark(2);
+ int b1 = in.read();
+ int b2 = in.read();
+ in.reset();
+ if (b1 == 0x1F && b2 == 0x8B) {
+ return new GZIPInputStream(in);
+ }
+ return in;
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityCacheLayout.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityCacheLayout.java
new file mode 100644
index 0000000000..60e6ae7b61
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityCacheLayout.java
@@ -0,0 +1,242 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.File;
+
+import org.biojava.nbio.structure.PdbId;
+import org.biojava.nbio.structure.io.LocalPDBDirectory;
+
+/**
+ * Where cached density files live on disk.
+ *
+ * <cache>/density/cb/1cbs_2fofc_pdbe.ccp4
+ * <cache>/density/cb/1cbs_fofc_pdbe.ccp4
+ * <cache>/density/cb/1cbs_2fofc_rcsbvs_d0.bcif
+ * <cache>/density/cb/1cbs_2fofc_wwpdb.cif.gz
+ *
+ * Both the kind and the source appear in the file name, so no two combinations
+ * can collide.
+ *
+ * <cache>/density/emd/EMD-0262/emd_0262.map.gz
+ *
+ * Several PDB entries are often fitted into a single EM map, and those maps can
+ * be hundreds of megabytes, so keying them by PDB entry would cache the same
+ * enormous file many times over.
+ * 1cbs and pdb_00001cbs both land
+ * in cb — which is also the rule the wwPDB documents for the
+ * new archive.
+ * rsync --delete. Should
+ * that judgement ever be revisited, every cached path is computed here and
+ * nowhere else, so a different layout is a change to this class plus a fallback
+ * probe for files in the old places — not a cache that everyone has to
+ * discard.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DensityCacheLayout {
+
+ /** Name of the density sub-directory within the BioJava cache directory. */
+ public static final String DENSITY_DIR = "density";
+
+ /** Name of the sub-directory holding EMDB-keyed maps. */
+ public static final String EMDB_DIR = "emd";
+
+ /** Name of the sub-directory holding cached PDB-to-EMDB mappings. */
+ public static final String EMDB_MAPPING_DIR = "emdb-mapping";
+
+ /**
+ * File-name token for a file that holds more than one kind of map, as a density
+ * server response does.
+ */
+ public static final String BOTH_KINDS_TOKEN = "both";
+
+ private DensityCacheLayout() {
+ }
+
+ /**
+ * The root density directory inside a cache directory.
+ *
+ * @param cacheRoot the BioJava cache directory
+ * @return the density directory, which need not exist yet
+ */
+ public static File densityRoot(File cacheRoot) {
+ return new File(cacheRoot, DENSITY_DIR);
+ }
+
+ /**
+ * The cache file for a PDB-keyed map.
+ *
+ * @param cacheRoot the BioJava cache directory
+ * @param pdbId the entry
+ * @param kind the kind of map
+ * @param source the service it came from
+ * @param format the file format
+ * @param qualifier an extra discriminator such as a detail level, or
+ * null. Anything that changes the content but not the
+ * entry, kind or source belongs here.
+ * @return the file, which need not exist
+ */
+ public static File pdbMapFile(File cacheRoot, PdbId pdbId, DensityMapKind kind, DensityMapSource source,
+ DensityFileFormat format, String qualifier) {
+ return pdbMapFile(cacheRoot, pdbId, kind.getFileToken(), source, format, qualifier);
+ }
+
+ /**
+ * The cache file for a PDB-keyed map, naming the kind explicitly.
+ * null
+ * @return the file, which need not exist
+ */
+ public static File pdbMapFile(File cacheRoot, PdbId pdbId, String kindToken, DensityMapSource source,
+ DensityFileFormat format, String qualifier) {
+ String id = shortIdOrFull(pdbId).toLowerCase();
+ File dir = new File(densityRoot(cacheRoot), LocalPDBDirectory.getMiddleHash(id));
+ StringBuilder name = new StringBuilder(id)
+ .append('_').append(kindToken)
+ .append('_').append(source.getFileToken());
+ if (qualifier != null && !qualifier.isEmpty()) {
+ name.append('_').append(qualifier);
+ }
+ name.append(format.getExtension());
+ return new File(dir, name.toString());
+ }
+
+ /**
+ * The cache file for an EMDB-keyed map.
+ *
+ * @param cacheRoot the BioJava cache directory
+ * @param emdbId the EMDB entry, in any accepted form
+ * @param source the service it came from
+ * @param format the file format
+ * @param qualifier an extra discriminator such as a detail level, or null
+ * @return the file, which need not exist
+ */
+ public static File emdbMapFile(File cacheRoot, String emdbId, DensityMapSource source,
+ DensityFileFormat format, String qualifier) {
+ String canonical = DensityMapRequest.normalizeEmdbId(emdbId);
+ String number = DensityMapRequest.emdbNumber(emdbId);
+ File dir = new File(new File(densityRoot(cacheRoot), EMDB_DIR), canonical);
+ StringBuilder name = new StringBuilder("emd_").append(number);
+ if (source != DensityMapSource.EMDB_MAP) {
+ name.append('_').append(source.getFileToken());
+ }
+ if (qualifier != null && !qualifier.isEmpty()) {
+ name.append('_').append(qualifier);
+ }
+ name.append(format.getExtension());
+ return new File(dir, name.toString());
+ }
+
+ /**
+ * The file caching the EMDB identifiers and author contour level associated
+ * with a PDB entry.
+ *
+ * @param cacheRoot the BioJava cache directory
+ * @param pdbId the entry
+ * @return the file, which need not exist
+ */
+ public static File emdbMappingFile(File cacheRoot, PdbId pdbId) {
+ String id = shortIdOrFull(pdbId).toLowerCase();
+ File dir = new File(new File(densityRoot(cacheRoot), EMDB_MAPPING_DIR), LocalPDBDirectory.getMiddleHash(id));
+ return new File(dir, id + ".emdb.properties");
+ }
+
+ /**
+ * The file caching an EMDB entry's map metadata as served by the EMDB API.
+ *
+ * @param cacheRoot the BioJava cache directory
+ * @param emdbId the EMDB entry, in any accepted form
+ * @return the file, which need not exist
+ */
+ public static File emdbMapInfoFile(File cacheRoot, String emdbId) {
+ String canonical = DensityMapRequest.normalizeEmdbId(emdbId);
+ File dir = new File(new File(densityRoot(cacheRoot), EMDB_DIR), canonical);
+ return new File(dir, canonical + ".map-info.json");
+ }
+
+ /**
+ * The companion name a density-server file must have for Jmol to read its
+ * difference-map block.
+ * 2FO-FC and an
+ * FO-FC data block, and Jmol's reader chooses between them by
+ * testing whether the file name contains the literal text
+ * &diff=1. That works for a URL fetched straight from the
+ * server, where the marker rides along in the query string, but a cached local
+ * file has no query string; appending the marker to the file URL only makes
+ * Jmol look for a file that does not exist. Putting the marker into the name
+ * itself is what actually selects the block.
+ * BCifDensityReader still carries the author's
+ * "what about cached data" to-do beside it. Should Jmol gain a cleaner way to
+ * choose the block, this can be retired.
+ *
+ * @param mapFile the cached density-server file
+ * @return the sibling path that selects the difference map
+ */
+ public static File differenceMarkerFile(File mapFile) {
+ String name = mapFile.getName();
+ int dot = name.lastIndexOf('.');
+ String stem = dot < 0 ? name : name.substring(0, dot);
+ String ext = dot < 0 ? "" : name.substring(dot);
+ return new File(mapFile.getAbsoluteFile().getParentFile(), stem + "&diff=1" + ext);
+ }
+
+ /**
+ * The short four-character spelling of an identifier where one exists.
+ * gemmi sf2map, or CCP4's fft after
+ * cif2mtz) is needed before these can be displayed.
+ */
+ MAP_COEFFICIENTS_CIF_GZ(".cif.gz", false);
+
+ private final String extension;
+ private final boolean jmolLoadable;
+
+ DensityFileFormat(String extension, boolean jmolLoadable) {
+ this.extension = extension;
+ this.jmolLoadable = jmolLoadable;
+ }
+
+ /**
+ * @return the file extension used for cached files of this format, including
+ * the leading dot
+ */
+ public String getExtension() {
+ return extension;
+ }
+
+ /**
+ * Whether a viewer can contour this file as it stands.
+ *
+ * @return false for {@link #MAP_COEFFICIENTS_CIF_GZ}, which needs
+ * an FFT first; true for the sampled grid formats
+ */
+ public boolean isJmolLoadable() {
+ return jmolLoadable;
+ }
+
+ /**
+ * Whether files of this format are gzip-compressed on disk. Jmol detects gzip
+ * from the magic bytes, so such files do not need decompressing before display.
+ *
+ * @return true for the compressed formats
+ */
+ public boolean isCompressed() {
+ return this == CCP4_GZ || this == MAP_COEFFICIENTS_CIF_GZ;
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapCache.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapCache.java
new file mode 100644
index 0000000000..adbbd917a8
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapCache.java
@@ -0,0 +1,526 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.EnumSet;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+import org.biojava.nbio.core.util.FileDownloadUtils;
+import org.biojava.nbio.core.util.HttpStatusException;
+import org.biojava.nbio.structure.ExperimentalTechnique;
+import org.biojava.nbio.structure.PdbId;
+import org.biojava.nbio.structure.Structure;
+import org.biojava.nbio.structure.align.util.UserConfiguration;
+import org.biojava.nbio.structure.io.LocalPDBDirectory.FetchBehavior;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Downloads and caches electron density and cryo-EM maps, trying several sources
+ * in turn until one produces a map.
+ * PDB_CACHE_DIR),
+ * laid out as described in {@link DensityCacheLayout}. Typical use is simply:
+ *
+ * DensityMapCache cache = new DensityMapCache();
+ * DensityMapResult map = cache.getDensityMap(new PdbId("1cbs"), DensityMapKind.TWO_FO_FC);
+ * File file = map.getFile();
+ *
+ * PDB_CACHE_DIR falling back to PDB_DIR.
+ */
+ public DensityMapCache() {
+ this(new UserConfiguration().getCacheFilePath());
+ }
+
+ /**
+ * @param cachePath the directory to cache under
+ */
+ public DensityMapCache(String cachePath) {
+ this.cacheRoot = new File(FileDownloadUtils.expandUserHome(cachePath));
+ this.emdbResolver = new EmdbEntryResolver(cacheRoot);
+ buildDefaultProviders();
+ }
+
+ /**
+ * A lazily created shared instance, for callers that do not want to manage one.
+ *
+ * @return the shared cache
+ */
+ public static synchronized DensityMapCache getInstance() {
+ if (instance == null) {
+ instance = new DensityMapCache();
+ }
+ return instance;
+ }
+
+ private void buildDefaultProviders() {
+ providers.clear();
+ register(new VolumeServerProvider(cacheRoot, VolumeServerProvider.Host.RCSB));
+ register(new VolumeServerProvider(cacheRoot, VolumeServerProvider.Host.PDBE));
+ register(new PdbeCcp4MapProvider(cacheRoot));
+ register(new EmdbMapProvider(cacheRoot, emdbResolver));
+ register(new WwpdbMapCoefficientsProvider(cacheRoot));
+ applySettingsToProviders();
+ }
+
+ private void register(DensityMapProvider provider) {
+ providers.put(provider.getSource(), provider);
+ }
+
+ private void applySettingsToProviders() {
+ for (DensityMapProvider p : providers.values()) {
+ if (p instanceof AbstractDensityMapProvider) {
+ AbstractDensityMapProvider a = (AbstractDensityMapProvider) p;
+ a.setCacheRoot(cacheRoot);
+ a.setFetchBehavior(fetchBehavior);
+ a.setMaxDownloadBytes(maxDownloadBytes);
+ }
+ }
+ emdbResolver.setCacheRoot(cacheRoot);
+ emdbResolver.setFetchBehavior(fetchBehavior);
+ }
+
+ /** @return the directory maps are cached under */
+ public String getCachePath() {
+ return cacheRoot.getAbsolutePath();
+ }
+
+ /** @param cachePath the directory to cache under */
+ public void setCachePath(String cachePath) {
+ this.cacheRoot = new File(FileDownloadUtils.expandUserHome(cachePath));
+ applySettingsToProviders();
+ }
+
+ /** @return how aggressively cached maps are re-fetched */
+ public FetchBehavior getFetchBehavior() {
+ return fetchBehavior;
+ }
+
+ /**
+ * @param fetchBehavior how aggressively to re-fetch. {@link FetchBehavior#FETCH_REMEDIATED}
+ * behaves as {@link FetchBehavior#FETCH_FILES}: the 2011 remediation date
+ * is a coordinate-file concept with no meaning for maps.
+ */
+ public void setFetchBehavior(FetchBehavior fetchBehavior) {
+ this.fetchBehavior = fetchBehavior == null ? FetchBehavior.FETCH_FILES : fetchBehavior;
+ applySettingsToProviders();
+ }
+
+ /** @return the ceiling on a single download in bytes, or 0 for no limit */
+ public long getMaxDownloadBytes() {
+ return maxDownloadBytes;
+ }
+
+ /** @param maxDownloadBytes the ceiling on a single download in bytes, or 0 for no limit */
+ public void setMaxDownloadBytes(long maxDownloadBytes) {
+ this.maxDownloadBytes = maxDownloadBytes;
+ applySettingsToProviders();
+ }
+
+ /**
+ * The order sources are tried in for a given kind of map.
+ *
+ * @param kind the kind of map
+ * @return the source order
+ */
+ public ListEMD-0262
+ * @param kind the kind of map wanted
+ * @return the map
+ * @throws NoDensityMapException if no enabled source has a map for this entry
+ * @throws IOException on transport failure
+ */
+ public DensityMapResult getDensityMap(String id, DensityMapKind kind) throws IOException {
+ return getDensityMap(DensityMapRequest.builder(id).kind(kind).build());
+ }
+
+ /**
+ * Fetches a density map for an already-loaded structure.
+ * null if it is not cached
+ */
+ public DensityMapResult getCached(PdbId pdbId, DensityMapKind kind, DensityMapSource source) {
+ DensityMapProvider provider = providers.get(source);
+ if (provider == null) {
+ return null;
+ }
+ File file = DensityCacheLayout.pdbMapFile(cacheRoot, pdbId, kind, source, provider.getFormat(), null);
+ return file.isFile() ? DensityMapResult.readMeta(file) : null;
+ }
+
+ /**
+ * Removes every cached density file for an entry, including the sidecars.
+ *
+ * @param pdbId the entry
+ * @return the number of files deleted
+ */
+ public int deleteDensityMaps(PdbId pdbId) {
+ String id = DensityCacheLayout.shortIdOrFull(pdbId).toLowerCase();
+ File dir = DensityCacheLayout.pdbMapFile(cacheRoot, pdbId, DensityMapKind.TWO_FO_FC,
+ DensityMapSource.PDBE_CCP4, DensityFileFormat.CCP4, null).getParentFile();
+ File[] files = dir == null ? null : dir.listFiles((d, name) -> name.startsWith(id + "_"));
+ if (files == null) {
+ return 0;
+ }
+ int deleted = 0;
+ for (File f : files) {
+ if (f.delete()) {
+ deleted++;
+ }
+ }
+ return deleted;
+ }
+
+ private DensityMapResult getDensityMap(DensityMapRequest request, Listnull for {@link #AUTO}, which is never
+ * itself cached
+ */
+ public String getFileToken() {
+ return fileToken;
+ }
+
+ /**
+ * Expands this kind into the concrete kinds to try, in order.
+ *
+ * @return a single-element list for a concrete kind, or the X-ray-then-EM order
+ * for {@link #AUTO}
+ */
+ public Listtrue only for {@link #FO_FC}
+ */
+ public boolean isDifferenceMap() {
+ return this == FO_FC;
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapProvider.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapProvider.java
new file mode 100644
index 0000000000..c628387ffd
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapProvider.java
@@ -0,0 +1,78 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.IOException;
+
+import org.biojava.nbio.core.util.HttpStatusException;
+
+/**
+ * Fetches density maps from one particular service.
+ *
+ *
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public interface DensityMapProvider {
+
+ /**
+ * @return which service this provider talks to
+ */
+ DensityMapSource getSource();
+
+ /**
+ * @return the format this provider delivers
+ */
+ DensityFileFormat getFormat();
+
+ /**
+ * Whether this provider can serve a given kind of map at all. Used to skip
+ * requests that could not possibly succeed, such as asking an X-ray map service
+ * for a cryo-EM map.
+ *
+ * @param kind the kind of map wanted
+ * @return null — when this service simply has nothing
+ * for the entry. The chain moves on to the next source.true if it is worth trying
+ */
+ boolean supports(DensityMapKind kind);
+
+ /**
+ * Fetches a map, using the cache if the request's fetch behaviour allows.
+ *
+ * @param request what is wanted. Its kind is always concrete, never
+ * {@link DensityMapKind#AUTO}.
+ * @return the map, or null if this service has nothing for the entry
+ * @throws DensityMapTooLargeException if the map exceeds the request's size limit
+ * @throws IOException on transport failure; use {@link HttpStatusException} so
+ * that a missing resource can be told apart from a broken connection
+ */
+ DensityMapResult fetch(DensityMapRequest request) throws IOException;
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapRequest.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapRequest.java
new file mode 100644
index 0000000000..d6677a3835
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapRequest.java
@@ -0,0 +1,275 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.File;
+import java.util.Collections;
+import java.util.List;
+
+import org.biojava.nbio.structure.PdbId;
+import org.biojava.nbio.structure.io.LocalPDBDirectory.FetchBehavior;
+
+/**
+ * A request for a density map, describing what is wanted and how hard to look
+ * for it.
+ * EMD- (case-insensitively) is taken as an EMDB entry, anything
+ * else as a PDB entry.
+ *
+ * @param id a PDB or EMDB identifier
+ * @return a new builder
+ */
+ public static Builder builder(String id) {
+ if (id == null) {
+ throw new IllegalArgumentException("Identifier must not be null");
+ }
+ String trimmed = id.trim();
+ if (trimmed.toUpperCase().startsWith("EMD-") || trimmed.toUpperCase().startsWith("EMD_")) {
+ return new Builder(null, normalizeEmdbId(trimmed)).kind(DensityMapKind.EM);
+ }
+ return new Builder(new PdbId(trimmed), null);
+ }
+
+ /**
+ * Normalises an EMDB identifier to the canonical EMD-1234 form.
+ *
+ * @param emdbId an identifier such as emd-1234, EMD_1234
+ * or a bare number
+ * @return the canonical form
+ */
+ public static String normalizeEmdbId(String emdbId) {
+ if (emdbId == null) {
+ return null;
+ }
+ String digits = emdbId.trim().toUpperCase().replaceFirst("^EMD[-_]?", "");
+ return "EMD-" + digits;
+ }
+
+ /**
+ * Extracts the numeric part of an EMDB identifier, as used in file names and
+ * some URL templates.
+ *
+ * @param emdbId an EMDB identifier in any accepted form
+ * @return the digits, without the EMD- prefix
+ */
+ public static String emdbNumber(String emdbId) {
+ return normalizeEmdbId(emdbId).substring("EMD-".length());
+ }
+
+ /** @return the PDB entry requested, or null for an EMDB-only request */
+ public PdbId getPdbId() {
+ return pdbId;
+ }
+
+ /** @return the EMDB entry, if known or explicitly requested; otherwise null */
+ public String getEmdbId() {
+ return emdbId;
+ }
+
+ /** @return the kind of map wanted; never null */
+ public DensityMapKind getKind() {
+ return kind;
+ }
+
+ /** @return how aggressively to re-fetch, or null to use the cache's setting */
+ public FetchBehavior getFetchBehavior() {
+ return fetchBehavior;
+ }
+
+ /** @return an override for the cache directory, or null to use the cache's own */
+ public File getCacheDir() {
+ return cacheDir;
+ }
+
+ /**
+ * Whether sources that deliver something a viewer cannot contour directly
+ * — currently only map coefficients — may be used.
+ *
+ * @return true by default; a viewer should set it to
+ * false
+ */
+ public boolean isAllowNonRenderableFormats() {
+ return allowNonRenderableFormats;
+ }
+
+ /** @return the download size limit in bytes, or 0 for no limit */
+ public long getMaxDownloadBytes() {
+ return maxDownloadBytes;
+ }
+
+ /** @return an explicit source order, or null to let the cache choose */
+ public Listnull means {@link DensityMapKind#AUTO} */
+ public Builder kind(DensityMapKind kind) {
+ this.kind = kind == null ? DensityMapKind.AUTO : kind;
+ return this;
+ }
+
+ /** @param emdbId the EMDB entry to use, skipping the PDB-to-EMDB lookup */
+ public Builder emdbId(String emdbId) {
+ this.emdbId = emdbId == null ? null : normalizeEmdbId(emdbId);
+ return this;
+ }
+
+ /** @param fetchBehavior how aggressively to re-fetch */
+ public Builder fetchBehavior(FetchBehavior fetchBehavior) {
+ this.fetchBehavior = fetchBehavior;
+ return this;
+ }
+
+ /** @param cacheDir an override for the cache directory */
+ public Builder cacheDir(File cacheDir) {
+ this.cacheDir = cacheDir;
+ return this;
+ }
+
+ /**
+ * @param allow whether formats that cannot be contoured directly may be used.
+ * A viewer should pass false.
+ */
+ public Builder allowNonRenderableFormats(boolean allow) {
+ this.allowNonRenderableFormats = allow;
+ return this;
+ }
+
+ /** @param bytes the download size limit, or 0 for no limit, or a negative value to use the cache's setting */
+ public Builder maxDownloadBytes(long bytes) {
+ this.maxDownloadBytes = bytes;
+ return this;
+ }
+
+ /** @param chain an explicit source order, overriding the cache's choice */
+ public Builder sourceChain(List.meta sidecar beside the cached
+ * file, so a result can be reconstructed later without contacting any server.
+ * That is what lets the cache serve
+ * {@link org.biojava.nbio.structure.io.LocalPDBDirectory.FetchBehavior#LOCAL_ONLY}
+ * requests fully offline.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class DensityMapResult {
+
+ private static final Logger logger = LoggerFactory.getLogger(DensityMapResult.class);
+
+ /** Extension of the metadata sidecar written beside every cached map. */
+ public static final String META_EXT = ".meta";
+
+ private final File file;
+ private final DensityMapSource source;
+ private final DensityFileFormat format;
+ private final DensityMapKind kind;
+ private final PdbId pdbId;
+ private final String emdbId;
+ private final String sourceUrl;
+ private final boolean fromCache;
+ private final Double recommendedContourLevel;
+ private final Double sigma;
+
+ /**
+ * @param file the cached map file
+ * @param source which service supplied it
+ * @param format the file format
+ * @param kind what the values mean; never {@link DensityMapKind#AUTO}
+ * @param pdbId the PDB entry, may be null
+ * @param emdbId the EMDB entry, may be null
+ * @param sourceUrl the URL it came from
+ * @param fromCache whether it was already on disk rather than freshly downloaded
+ * @param recommendedContourLevel the author-recommended contour level in absolute
+ * map units, or null if unknown
+ * @param sigma the RMS deviation of the map, or null if unknown
+ */
+ public DensityMapResult(File file, DensityMapSource source, DensityFileFormat format, DensityMapKind kind,
+ PdbId pdbId, String emdbId, String sourceUrl, boolean fromCache,
+ Double recommendedContourLevel, Double sigma) {
+ this.file = file;
+ this.source = source;
+ this.format = format;
+ this.kind = kind;
+ this.pdbId = pdbId;
+ this.emdbId = emdbId;
+ this.sourceUrl = sourceUrl;
+ this.fromCache = fromCache;
+ this.recommendedContourLevel = recommendedContourLevel;
+ this.sigma = sigma;
+ }
+
+ /** @return the cached map file */
+ public File getFile() {
+ return file;
+ }
+
+ /** @return which service supplied the map */
+ public DensityMapSource getSource() {
+ return source;
+ }
+
+ /** @return the file format */
+ public DensityFileFormat getFormat() {
+ return format;
+ }
+
+ /** @return what the map values mean; never {@link DensityMapKind#AUTO} */
+ public DensityMapKind getKind() {
+ return kind;
+ }
+
+ /** @return the PDB entry, or null */
+ public PdbId getPdbId() {
+ return pdbId;
+ }
+
+ /** @return the EMDB entry, or null */
+ public String getEmdbId() {
+ return emdbId;
+ }
+
+ /** @return the URL the map was fetched from */
+ public String getSourceUrl() {
+ return sourceUrl;
+ }
+
+ /** @return true if the file was already cached rather than downloaded now */
+ public boolean isFromCache() {
+ return fromCache;
+ }
+
+ /**
+ * Whether a viewer can contour this file as it stands.
+ *
+ * @return false for map coefficients, which need a Fourier
+ * transform first
+ * @see DensityFileFormat#isJmolLoadable()
+ */
+ public boolean isRenderable() {
+ return format != null && format.isJmolLoadable();
+ }
+
+ /**
+ * The contour level recommended by the depositors, in absolute map units.
+ * Normally only available for EMDB maps, where it is the conventional way to
+ * contour rather than a multiple of sigma.
+ *
+ * @return the level, or null if unknown
+ */
+ public Double getRecommendedContourLevel() {
+ return recommendedContourLevel;
+ }
+
+ /**
+ * @return the RMS deviation of the map values, or null if unknown
+ */
+ public Double getSigma() {
+ return sigma;
+ }
+
+ /**
+ * The recommended contour expressed in multiples of sigma, for viewers that
+ * prefer to work that way.
+ *
+ * @return the level divided by sigma, or null if either is unknown
+ * or sigma is zero
+ */
+ public Double getContourInSigma() {
+ if (recommendedContourLevel == null || sigma == null || sigma == 0.0) {
+ return null;
+ }
+ return recommendedContourLevel / sigma;
+ }
+
+ /** @return the size of the cached file in bytes, or 0 if it is missing */
+ public long getFileSizeBytes() {
+ return file == null ? 0 : file.length();
+ }
+
+ /**
+ * The metadata sidecar file for a cached map.
+ *
+ * @param mapFile the cached map
+ * @return the sidecar path, which need not exist
+ */
+ public static File metaFileFor(File mapFile) {
+ return new File(mapFile.getAbsoluteFile().getParentFile(), mapFile.getName() + META_EXT);
+ }
+
+ /**
+ * Writes this result's metadata beside the cached file, so that it can be
+ * reconstructed later without any network access.
+ */
+ public void writeMeta() {
+ Properties p = new Properties();
+ put(p, "source", source);
+ put(p, "format", format);
+ put(p, "kind", kind);
+ put(p, "pdbId", pdbId == null ? null : pdbId.getId());
+ put(p, "emdbId", emdbId);
+ put(p, "url", sourceUrl);
+ put(p, "downloaded", Instant.now().toString());
+ put(p, "bytes", getFileSizeBytes());
+ put(p, "contourLevel", recommendedContourLevel);
+ put(p, "sigma", sigma);
+ File meta = metaFileFor(file);
+ try (OutputStream out = Files.newOutputStream(meta.toPath())) {
+ p.store(out, "BioJava density map metadata");
+ } catch (IOException e) {
+ // Losing the sidecar costs us the offline description, not the map itself.
+ logger.warn("Could not write density metadata [{}]: {}", meta, e.getMessage());
+ }
+ }
+
+ /**
+ * Reconstructs a result from a cached file and its metadata sidecar.
+ *
+ * @param mapFile the cached map
+ * @return the reconstructed result, or null if the sidecar is
+ * missing or unusable
+ */
+ public static DensityMapResult readMeta(File mapFile) {
+ File meta = metaFileFor(mapFile);
+ if (!meta.isFile()) {
+ return null;
+ }
+ Properties p = new Properties();
+ try (InputStream in = Files.newInputStream(meta.toPath())) {
+ p.load(in);
+ } catch (IOException e) {
+ logger.warn("Could not read density metadata [{}]: {}", meta, e.getMessage());
+ return null;
+ }
+ try {
+ String pdb = p.getProperty("pdbId");
+ return new DensityMapResult(mapFile,
+ DensityMapSource.valueOf(p.getProperty("source")),
+ DensityFileFormat.valueOf(p.getProperty("format")),
+ DensityMapKind.valueOf(p.getProperty("kind")),
+ pdb == null || pdb.isEmpty() ? null : new PdbId(pdb),
+ emptyToNull(p.getProperty("emdbId")),
+ p.getProperty("url"),
+ true,
+ parseDouble(p.getProperty("contourLevel")),
+ parseDouble(p.getProperty("sigma")));
+ } catch (RuntimeException e) {
+ logger.warn("Density metadata [{}] is unusable: {}", meta, e.getMessage());
+ return null;
+ }
+ }
+
+ private static void put(Properties p, String key, Object value) {
+ p.setProperty(key, value == null ? "" : String.valueOf(value));
+ }
+
+ private static String emptyToNull(String s) {
+ return s == null || s.isEmpty() ? null : s;
+ }
+
+ private static Double parseDouble(String s) {
+ if (s == null || s.isEmpty()) {
+ return null;
+ }
+ try {
+ return Double.valueOf(s);
+ } catch (NumberFormatException e) {
+ return null;
+ }
+ }
+
+ @Override
+ public String toString() {
+ return String.format("%s %s map from %s (%s, %d bytes)%s",
+ pdbId == null ? emdbId : pdbId.getId(), kind, source, format, getFileSizeBytes(),
+ isRenderable() ? "" : " [needs an FFT before it can be displayed]");
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapSource.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapSource.java
new file mode 100644
index 0000000000..a0f2e1a365
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapSource.java
@@ -0,0 +1,84 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+/**
+ * A remote service that density data can be fetched from.
+ * edmaps.rcsb.org service, which used to serve
+ * DSN6 and MTZ files, was shut down in October 2024. The map coefficients
+ * published with the wwPDB validation reports replaced it, but those are
+ * structure factors rather than a sampled grid; see
+ * {@link DensityFileFormat#MAP_COEFFICIENTS_CIF_GZ}.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public enum DensityMapSource {
+
+ /**
+ * RCSB's Mol* density server at maps.rcsb.org, which serves
+ * downsampled BinaryCIF volume slices for both X-ray and EM entries.
+ */
+ RCSB_VOLUME_SERVER("rcsbvs"),
+
+ /**
+ * PDBe's Mol* density server, equivalent to {@link #RCSB_VOLUME_SERVER}.
+ */
+ PDBE_VOLUME_SERVER("pdbevs"),
+
+ /**
+ * PDBe's pre-computed full CCP4 maps. This is the source Jmol itself uses for
+ * its built-in map-loading shortcuts.
+ */
+ PDBE_CCP4("pdbe"),
+
+ /**
+ * The primary map of an EMDB entry, at full resolution. Can be very large.
+ */
+ EMDB_MAP("emdb"),
+
+ /**
+ * Map coefficients from the wwPDB validation reports. Archival only: these
+ * cannot be displayed without an FFT.
+ */
+ WWPDB_MAP_COEFFICIENTS("wwpdb");
+
+ private final String fileToken;
+
+ DensityMapSource(String fileToken) {
+ this.fileToken = fileToken;
+ }
+
+ /**
+ * The short token used to distinguish this source in a cached file name, so
+ * that maps of the same kind from different sources never collide.
+ *
+ * @return the token
+ */
+ public String getFileToken() {
+ return fileToken;
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapTooLargeException.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapTooLargeException.java
new file mode 100644
index 0000000000..ce62e36d1d
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/DensityMapTooLargeException.java
@@ -0,0 +1,84 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.IOException;
+
+/**
+ * Thrown when a density map exceeds the configured download size limit.
+ * null
+ * @param sigma the RMS deviation of the map values, or null
+ * @param mapSizeBytes the size of the primary map file, or null
+ */
+ public EmdbEntryInfo(String emdbId, Double recommendedContourLevel, Double sigma, Long mapSizeBytes) {
+ this.emdbId = emdbId;
+ this.recommendedContourLevel = recommendedContourLevel;
+ this.sigma = sigma;
+ this.mapSizeBytes = mapSizeBytes;
+ }
+
+ /** @return the entry identifier in canonical EMD-1234 form */
+ public String getEmdbId() {
+ return emdbId;
+ }
+
+ /**
+ * The contour level the depositors recommend, in absolute map units. EM maps
+ * are conventionally displayed at this level rather than at a multiple of
+ * sigma, so it is the right default for a viewer.
+ *
+ * @return the level, or null if the entry does not state one
+ */
+ public Double getRecommendedContourLevel() {
+ return recommendedContourLevel;
+ }
+
+ /**
+ * @return the RMS deviation of the map values, which converts between absolute
+ * and sigma-relative contour levels, or null if unknown
+ */
+ public Double getSigma() {
+ return sigma;
+ }
+
+ /**
+ * @return the size of the full primary map in bytes, or null if
+ * unknown. Used to decline a download before it starts.
+ */
+ public Long getMapSizeBytes() {
+ return mapSizeBytes;
+ }
+
+ @Override
+ public String toString() {
+ return String.format("%s[contour=%s, sigma=%s, %s bytes]",
+ emdbId, recommendedContourLevel, sigma, mapSizeBytes);
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/EmdbEntryResolver.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/EmdbEntryResolver.java
new file mode 100644
index 0000000000..fab0101b34
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/EmdbEntryResolver.java
@@ -0,0 +1,382 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+import java.nio.file.Files;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Properties;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+import org.biojava.nbio.structure.PdbId;
+import org.biojava.nbio.structure.align.util.URLConnectionTools;
+import org.biojava.nbio.structure.io.LocalPDBDirectory.FetchBehavior;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Finds the EMDB entry, if any, associated with a PDB entry, and reads the few
+ * facts about its map that are needed to fetch and display it.
+ * null if unknown
+ */
+ public Double getContourLevelFor(PdbId pdbId) {
+ Mapping cached = readMapping(pdbId);
+ if (cached != null && (fetchBehavior == FetchBehavior.LOCAL_ONLY || cached.isFresh(mappingMaxAgeDays))) {
+ return cached.contourLevel;
+ }
+ getEmdbIds(pdbId);
+ Mapping refreshed = readMapping(pdbId);
+ return refreshed == null ? null : refreshed.contourLevel;
+ }
+
+ /**
+ * Reads an EMDB entry's map metadata: contour level, RMS deviation and the size
+ * of the primary map file.
+ * null if it could not be obtained
+ */
+ public EmdbEntryInfo getEntryInfo(String emdbId) {
+ String canonical = DensityMapRequest.normalizeEmdbId(emdbId);
+ File cacheFile = DensityCacheLayout.emdbMapInfoFile(cacheRoot, canonical);
+
+ String json = null;
+ if (cacheFile.isFile()) {
+ try {
+ json = new String(Files.readAllBytes(cacheFile.toPath()), StandardCharsets.UTF_8);
+ } catch (IOException e) {
+ logger.debug("Could not read cached EMDB metadata [{}]: {}", cacheFile, e.getMessage());
+ }
+ }
+ if (json == null) {
+ if (fetchBehavior == FetchBehavior.LOCAL_ONLY) {
+ return null;
+ }
+ String url = UrlTemplates.expand(mapInfoUrlTemplate, UrlTemplates.values(null, canonical, -1));
+ try {
+ json = read(new URL(url));
+ } catch (IOException e) {
+ logger.warn("Could not read EMDB metadata for {}: {}", canonical, e.getMessage());
+ return null;
+ }
+ writeCache(cacheFile, json);
+ }
+
+ try {
+ JsonNode map = MAPPER.readTree(json).path("map");
+ Double contour = null;
+ JsonNode contours = map.path("contour_list").path("contour");
+ for (JsonNode c : contours) {
+ if (contour == null || c.path("primary").asBoolean(false)) {
+ contour = c.path("level").isNumber() ? c.path("level").asDouble() : contour;
+ }
+ }
+ Double sigma = map.path("statistics").path("std").isNumber()
+ ? map.path("statistics").path("std").asDouble() : null;
+ Long bytes = map.path("size_kbytes").isNumber()
+ ? map.path("size_kbytes").asLong() * 1024L : null;
+ return new EmdbEntryInfo(canonical, contour, sigma, bytes);
+ } catch (IOException | RuntimeException e) {
+ logger.warn("Could not parse EMDB metadata for {}: {}", canonical, e.getMessage());
+ return null;
+ }
+ }
+
+ private Mapping searchEmdb(PdbId pdbId) {
+ String url = UrlTemplates.expand(searchUrlTemplate,
+ UrlTemplates.values(DensityCacheLayout.shortIdOrFull(pdbId), null, -1));
+ try {
+ String csv = read(new URL(url));
+ List.map.gz can be handed to it without decompressing first.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class EmdbMapProvider extends AbstractDensityMapProvider {
+
+ /** Default base URL of the EMDB archive. */
+ public static final String DEFAULT_SERVER_URL = "https://ftp.ebi.ac.uk/pub/databases/emdb/structures/";
+
+ /** Default path template for an entry's primary map. */
+ public static final String DEFAULT_MAP_TEMPLATE = "{emdb_id}/map/emd_{emdb_num}.map.gz";
+
+ private static String serverBaseUrl = DEFAULT_SERVER_URL;
+ private static String mapTemplate = DEFAULT_MAP_TEMPLATE;
+
+ private final EmdbEntryResolver resolver;
+
+ /**
+ * @param cacheRoot the BioJava cache directory
+ * @param resolver used to find the EMDB entry for a PDB entry and to read its
+ * contour level and size
+ */
+ public EmdbMapProvider(File cacheRoot, EmdbEntryResolver resolver) {
+ super(cacheRoot);
+ this.resolver = resolver;
+ }
+
+ /** @return the base URL of the EMDB archive */
+ public static String getServerBaseUrl() {
+ return serverBaseUrl;
+ }
+
+ /** @param url the base URL; a trailing slash is added if missing */
+ public static void setServerBaseUrl(String url) {
+ serverBaseUrl = url == null ? DEFAULT_SERVER_URL : (url.endsWith("/") ? url : url + "/");
+ }
+
+ /** @param template the path template for an entry's primary map */
+ public static void setMapUrlTemplate(String template) {
+ mapTemplate = template == null ? DEFAULT_MAP_TEMPLATE : template;
+ }
+
+ /** Restores the default server and template. */
+ public static void resetToDefaults() {
+ serverBaseUrl = DEFAULT_SERVER_URL;
+ mapTemplate = DEFAULT_MAP_TEMPLATE;
+ }
+
+ @Override
+ public DensityMapSource getSource() {
+ return DensityMapSource.EMDB_MAP;
+ }
+
+ @Override
+ public DensityFileFormat getFormat() {
+ return DensityFileFormat.CCP4_GZ;
+ }
+
+ @Override
+ public boolean supports(DensityMapKind kind) {
+ return kind == DensityMapKind.EM;
+ }
+
+ /**
+ * Builds the URL of an entry's primary map without fetching it.
+ *
+ * @param emdbId the EMDB entry, in any accepted form
+ * @return the URL as a string
+ */
+ public String buildUrl(String emdbId) {
+ return serverBaseUrl + UrlTemplates.expand(mapTemplate, UrlTemplates.values(null, emdbId, -1));
+ }
+
+ @Override
+ public DensityMapResult fetch(DensityMapRequest request) throws IOException {
+ if (!supports(request.getKind())) {
+ return null;
+ }
+ String emdbId = request.getEmdbId();
+ if (emdbId == null) {
+ return null;
+ }
+
+ EmdbEntryInfo info = resolver == null ? null : resolver.getEntryInfo(emdbId);
+
+ // Decline before transferring anything: EMDB reports the map size in its
+ // metadata, so an oversized map costs one small request rather than a
+ // partial multi-hundred-megabyte download.
+ long limit = effectiveMaxBytes(request);
+ if (limit > 0 && info != null && info.getMapSizeBytes() != null && info.getMapSizeBytes() > limit) {
+ URL url = new URL(buildUrl(emdbId));
+ reportTooLarge(url, info.getMapSizeBytes(), limit, request);
+ throw new DensityMapTooLargeException(url.toString(), info.getMapSizeBytes(), limit);
+ }
+
+ URL url = new URL(buildUrl(emdbId));
+ File target = DensityCacheLayout.emdbMapFile(effectiveCacheRoot(request), emdbId,
+ getSource(), getFormat(), null);
+ return obtain(request, url, target, DensityMapKind.EM, emdbId,
+ info == null ? null : info.getRecommendedContourLevel(),
+ info == null ? null : info.getSigma());
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/NoDensityMapException.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/NoDensityMapException.java
new file mode 100644
index 0000000000..6691b584d9
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/NoDensityMapException.java
@@ -0,0 +1,99 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.IOException;
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+import org.biojava.nbio.structure.PdbId;
+
+/**
+ * Thrown when no enabled source could supply a density map for an entry.
+ * null for a
+ * request made by EMDB identifier
+ * @param kind the kind of map that was requested
+ * @param attempts what happened at each source that was tried, in the order tried
+ */
+ public NoDensityMapException(PdbId pdbId, DensityMapKind kind, Mapnull
+ */
+ public PdbId getPdbId() {
+ return pdbId;
+ }
+
+ /**
+ * @return the kind of map that was requested
+ */
+ public DensityMapKind getKind() {
+ return kind;
+ }
+
+ /**
+ * What happened at each source, in the order they were tried. Suitable for
+ * building a human-readable explanation.
+ *
+ * @return an unmodifiable map from source to reason
+ */
+ public Map1CBS.ccp4 and pdb_00001cbs.ccp4 both
+ * return HTTP 404. Entries deposited without structure factors, and cryo-EM
+ * entries, have no maps here at all.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class PdbeCcp4MapProvider extends AbstractDensityMapProvider {
+
+ /** Default base URL of the PDBe map service. */
+ public static final String DEFAULT_SERVER_URL = "https://www.ebi.ac.uk/pdbe/coordinates/files/";
+
+ /**
+ * An equivalent base URL serving the same files, kept in the documentation as a
+ * ready alternative should the primary one change.
+ */
+ public static final String ALTERNATIVE_SERVER_URL = "https://www.ebi.ac.uk/pdbe/entry-files/";
+
+ /** Default path template for the 2mFo-DFc map. */
+ public static final String DEFAULT_TWO_FO_FC_TEMPLATE = "{pdbid_lc}.ccp4";
+
+ /** Default path template for the mFo-DFc difference map. */
+ public static final String DEFAULT_FO_FC_TEMPLATE = "{pdbid_lc}_diff.ccp4";
+
+ private static String serverBaseUrl = DEFAULT_SERVER_URL;
+ private static String twoFoFcTemplate = DEFAULT_TWO_FO_FC_TEMPLATE;
+ private static String foFcTemplate = DEFAULT_FO_FC_TEMPLATE;
+
+ /**
+ * @param cacheRoot the BioJava cache directory
+ */
+ public PdbeCcp4MapProvider(File cacheRoot) {
+ super(cacheRoot);
+ }
+
+ /** @return the base URL of the map service */
+ public static String getServerBaseUrl() {
+ return serverBaseUrl;
+ }
+
+ /**
+ * @param url the base URL of the map service; a trailing slash is added if missing
+ */
+ public static void setServerBaseUrl(String url) {
+ serverBaseUrl = url == null ? DEFAULT_SERVER_URL : (url.endsWith("/") ? url : url + "/");
+ }
+
+ /**
+ * Overrides the path template for a map kind.
+ *
+ * @param kind {@link DensityMapKind#TWO_FO_FC} or {@link DensityMapKind#FO_FC}
+ * @param template a template understood by {@link UrlTemplates}
+ */
+ public static void setPathUrlTemplate(DensityMapKind kind, String template) {
+ if (kind == DensityMapKind.TWO_FO_FC) {
+ twoFoFcTemplate = template == null ? DEFAULT_TWO_FO_FC_TEMPLATE : template;
+ } else if (kind == DensityMapKind.FO_FC) {
+ foFcTemplate = template == null ? DEFAULT_FO_FC_TEMPLATE : template;
+ } else {
+ throw new IllegalArgumentException("PDBe CCP4 maps exist only for 2Fo-Fc and Fo-Fc, not " + kind);
+ }
+ }
+
+ /** Restores the default server and templates. */
+ public static void resetToDefaults() {
+ serverBaseUrl = DEFAULT_SERVER_URL;
+ twoFoFcTemplate = DEFAULT_TWO_FO_FC_TEMPLATE;
+ foFcTemplate = DEFAULT_FO_FC_TEMPLATE;
+ }
+
+ @Override
+ public DensityMapSource getSource() {
+ return DensityMapSource.PDBE_CCP4;
+ }
+
+ @Override
+ public DensityFileFormat getFormat() {
+ return DensityFileFormat.CCP4;
+ }
+
+ @Override
+ public boolean supports(DensityMapKind kind) {
+ return kind == DensityMapKind.TWO_FO_FC || kind == DensityMapKind.FO_FC;
+ }
+
+ /**
+ * Builds the URL for a map without fetching it.
+ *
+ * @param pdbId the entry
+ * @param kind the kind of map
+ * @return the URL as a string
+ */
+ public String buildUrl(PdbId pdbId, DensityMapKind kind) {
+ String template = kind == DensityMapKind.FO_FC ? foFcTemplate : twoFoFcTemplate;
+ return serverBaseUrl + UrlTemplates.expand(template, UrlTemplates.values(urlId(pdbId), null, -1));
+ }
+
+ @Override
+ public DensityMapResult fetch(DensityMapRequest request) throws IOException {
+ if (request.getPdbId() == null || !supports(request.getKind())) {
+ return null;
+ }
+ URL url = new URL(buildUrl(request.getPdbId(), request.getKind()));
+ File target = DensityCacheLayout.pdbMapFile(effectiveCacheRoot(request), request.getPdbId(),
+ request.getKind(), getSource(), getFormat(), null);
+ return obtain(request, url, target, request.getKind(), null, null, null);
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/UrlTemplates.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/UrlTemplates.java
new file mode 100644
index 0000000000..0518f1d4a4
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/UrlTemplates.java
@@ -0,0 +1,152 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.util.LinkedHashMap;
+import java.util.Map;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+import org.biojava.nbio.structure.PdbId;
+import org.biojava.nbio.structure.StructureException;
+
+/**
+ * Expands the named placeholders used in the configurable density-server URL
+ * templates.
+ *
+ *
+ * A placeholder with no supplied value is left in place rather than replaced by
+ * an empty string, so that a misconfigured template produces an obviously wrong
+ * URL instead of a subtly wrong one.
+ * {pdbid}{pdbid_lc}{pdbid_uc}{mid}cb for 1cbs{extid}pdb_00001cbs for
+ * 1cbs{emdb_id}EMD-0262{emdb_num}0262{detail}{mid} and {extid} exist for mirrors rather than for
+ * the default URLs. The services BioJava fetches from resolve an entry by name
+ * alone, but a site pointing this code at its own copy of the archive has to
+ * spell out a directory path — and so does any mirror that publishes one,
+ * such as EBI. Both the present divided layout and the per-entry layout that
+ * replaces it in July 2027 are directory paths, and between them the two
+ * placeholders express either without a code change:
+ *
+ * {mid}/{pdbid_lc}/{pdbid_lc}_validation_2fo-fc_map_coef.cif.gz
+ * entries/{mid}/{extid}/validation_reports/{extid}_validation_2fo-fc_map_coef.cif.gz
+ *
+ * null
+ * @param emdbId the EMDB identifier in any accepted form, may be null
+ * @param detail the density-server detail level, or a negative value to omit it
+ * @return a map suitable for {@link #expand(String, Map)}
+ */
+ public static Mapnull if the argument is
+ * neither a short nor an extended identifier, in which case
+ * {extid} is left unexpanded rather than guessed at
+ */
+ private static String extendedId(String pdbId) {
+ try {
+ return PdbId.toExtendedId(pdbId).toLowerCase();
+ } catch (StructureException e) {
+ return null;
+ }
+ }
+}
diff --git a/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/VolumeServerProvider.java b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/VolumeServerProvider.java
new file mode 100644
index 0000000000..1700f3a472
--- /dev/null
+++ b/biojava-structure/src/main/java/org/biojava/nbio/structure/io/density/VolumeServerProvider.java
@@ -0,0 +1,246 @@
+/**
+ * BioJava development code
+ *
+ * This code may be freely distributed and modified under the terms of the GNU
+ * Lesser General Public Licence. This should be distributed with the code. If
+ * you do not have a copy, see:
+ *
+ * http://www.gnu.org/copyleft/lesser.html
+ *
+ * Copyright for this code is held jointly by the individual authors. These
+ * should be listed in @author doc comments.
+ *
+ * For more information on the BioJava project and its aims, or to join the
+ * biojava-l mailing list, visit the home page at:
+ *
+ * http://www.biojava.org/
+ */
+package org.biojava.nbio.structure.io.density;
+
+import java.io.File;
+import java.io.IOException;
+import java.net.URL;
+import java.nio.file.Files;
+import java.nio.file.StandardCopyOption;
+
+/**
+ * Fetches downsampled volume slices from a Mol* density server.
+ * Content-Length, no ETag and no
+ * Last-Modified, so the usual server-provided validation metadata is
+ * unavailable; the size recorded for a cached slice is the byte count observed
+ * during our own download instead. Detail levels are not linear either —
+ * for a small entry the response stops growing past detail 1, while for a large
+ * EM map the step from detail 2 to 3 multiplies the size several-fold.
+ *
+ * @author Amr ALHOSSARY
+ * @since 7.3.0
+ */
+public class VolumeServerProvider extends AbstractDensityMapProvider {
+
+ /**
+ * Which density server to talk to.
+ */
+ public enum Host {
+ /** RCSB's server at maps.rcsb.org. */
+ RCSB(DensityMapSource.RCSB_VOLUME_SERVER, "https://maps.rcsb.org/", 3),
+ /** PDBe's server at www.ebi.ac.uk/pdbe/volume-server. */
+ PDBE(DensityMapSource.PDBE_VOLUME_SERVER, "https://www.ebi.ac.uk/pdbe/volume-server/", 6);
+
+ private final DensityMapSource source;
+ private final String defaultBaseUrl;
+ private final int defaultDetail;
+
+ Host(DensityMapSource source, String defaultBaseUrl, int defaultDetail) {
+ this.source = source;
+ this.defaultBaseUrl = defaultBaseUrl;
+ this.defaultDetail = defaultDetail;
+ }
+
+ /** @return the source constant this host corresponds to */
+ public DensityMapSource getSource() {
+ return source;
+ }
+
+ /** @return the default base URL */
+ public String getDefaultBaseUrl() {
+ return defaultBaseUrl;
+ }
+
+ /** @return the detail level this host's own clients use by default */
+ public int getDefaultDetail() {
+ return defaultDetail;
+ }
+ }
+
+ /** Default path template for an X-ray entry's whole unit cell. */
+ public static final String DEFAULT_XRAY_CELL_TEMPLATE = "x-ray/{pdbid_lc}/cell?detail={detail}&encoding={encoding}";
+
+ /** Default path template for an EM entry's whole cell. */
+ public static final String DEFAULT_EM_CELL_TEMPLATE = "em/emd-{emdb_num}/cell?detail={detail}&encoding={encoding}";
+
+ private final Host host;
+ private String baseUrl;
+ private int detail;
+ private String encoding = "bcif";
+
+ /**
+ * @param cacheRoot the BioJava cache directory
+ * @param host which density server to use
+ */
+ public VolumeServerProvider(File cacheRoot, Host host) {
+ super(cacheRoot);
+ this.host = host;
+ this.baseUrl = host.getDefaultBaseUrl();
+ this.detail = host.getDefaultDetail();
+ }
+
+ /** @return which density server this instance uses */
+ public Host getHost() {
+ return host;
+ }
+
+ /** @return the base URL in use */
+ public String getBaseUrl() {
+ return baseUrl;
+ }
+
+ /** @param baseUrl the base URL; a trailing slash is added if missing */
+ public void setBaseUrl(String baseUrl) {
+ this.baseUrl = baseUrl == null ? host.getDefaultBaseUrl()
+ : (baseUrl.endsWith("/") ? baseUrl : baseUrl + "/");
+ }
+
+ /** @return the detail level requested from the server */
+ public int getDetail() {
+ return detail;
+ }
+
+ /**
+ * Sets the detail level. Higher means a finer grid and a larger download; the
+ * relationship is neither linear nor the same for every entry.
+ *
+ * @param detail the level, normally 0 to 6
+ */
+ public void setDetail(int detail) {
+ this.detail = detail;
+ }
+
+ /** @return the encoding requested, either bcif or cif */
+ public String getEncoding() {
+ return encoding;
+ }
+
+ /**
+ * @param encoding bcif for BinaryCIF (default, and much smaller) or
+ * cif for text
+ */
+ public void setEncoding(String encoding) {
+ this.encoding = encoding == null ? "bcif" : encoding;
+ }
+
+ @Override
+ public DensityMapSource getSource() {
+ return host.getSource();
+ }
+
+ @Override
+ public DensityFileFormat getFormat() {
+ return "cif".equalsIgnoreCase(encoding) ? DensityFileFormat.CIF_VOLUME : DensityFileFormat.BCIF_VOLUME;
+ }
+
+ @Override
+ public boolean supports(DensityMapKind kind) {
+ return kind == DensityMapKind.TWO_FO_FC || kind == DensityMapKind.FO_FC || kind == DensityMapKind.EM;
+ }
+
+ /**
+ * Builds the URL for a request without fetching it.
+ *
+ * @param request the request; its kind must be concrete
+ * @return the URL as a string, or null if the request cannot be served
+ */
+ public String buildUrl(DensityMapRequest request) {
+ if (request.getKind() == DensityMapKind.EM) {
+ if (request.getEmdbId() == null) {
+ return null;
+ }
+ return baseUrl + UrlTemplates.expand(withEncoding(DEFAULT_EM_CELL_TEMPLATE),
+ UrlTemplates.values(null, request.getEmdbId(), detail));
+ }
+ if (request.getPdbId() == null) {
+ return null;
+ }
+ return baseUrl + UrlTemplates.expand(withEncoding(DEFAULT_XRAY_CELL_TEMPLATE),
+ UrlTemplates.values(urlId(request.getPdbId()), null, detail));
+ }
+
+ private String withEncoding(String template) {
+ return template.replace("{encoding}", encoding);
+ }
+
+ @Override
+ public DensityMapResult fetch(DensityMapRequest request) throws IOException {
+ if (!supports(request.getKind())) {
+ return null;
+ }
+ String urlString = buildUrl(request);
+ if (urlString == null) {
+ return null;
+ }
+ URL url = new URL(urlString);
+
+ // The detail level changes the content, so it has to be part of the cache key.
+ String qualifier = "d" + detail;
+ File target;
+ if (request.getKind() == DensityMapKind.EM) {
+ target = DensityCacheLayout.emdbMapFile(effectiveCacheRoot(request), request.getEmdbId(),
+ getSource(), getFormat(), qualifier);
+ } else {
+ // One response carries both the 2Fo-Fc and the Fo-Fc blocks, so the two
+ // kinds share a cache entry: asking for each separately would otherwise
+ // download and store the identical file twice. Which block to read is
+ // decided at display time, not here.
+ target = DensityCacheLayout.pdbMapFile(effectiveCacheRoot(request), request.getPdbId(),
+ DensityCacheLayout.BOTH_KINDS_TOKEN, getSource(), getFormat(), qualifier);
+ }
+ DensityMapResult result = obtain(request, url, target, request.getKind(), request.getEmdbId(), null, null);
+
+ if (request.getKind() == DensityMapKind.FO_FC) {
+ return presentAsDifferenceMap(result);
+ }
+ return result;
+ }
+
+ /**
+ * Points the result at the companion file name that makes Jmol read the
+ * difference-map block. See
+ * {@link DensityCacheLayout#differenceMarkerFile(File)} for why the marker has
+ * to be in the name.
+ * gemmi sf2map, or cif2mtz followed by
+ * CCP4's fft. Nothing in BioJava or Jmol will render them.
+ * edmaps.rcsb.org was shut down in October 2024, and because they
+ * are the authoritative archival form. For anything that needs to be displayed,
+ * prefer {@link PdbeCcp4MapProvider} or {@link VolumeServerProvider}. Accordingly
+ * this source is disabled by default in {@link DensityMapCache}.
+ * ETag, so downloads from here are checksum-verified automatically.
+ *
+ * setServerBaseUrl(EBI_MIRROR_URL);
+ * setPathUrlTemplate(DensityMapKind.TWO_FO_FC, DIVIDED_TWO_FO_FC_TEMPLATE);
+ * setPathUrlTemplate(DensityMapKind.FO_FC, DIVIDED_FO_FC_TEMPLATE);
+ *
+ * Setting the base alone yields 404s, because the flat file names do not exist
+ * there.
+ */
+ public static final String EBI_MIRROR_URL = "https://ftp.ebi.ac.uk/pub/databases/pdb/validation_reports/";
+
+ /**
+ * Default path template for the 2mFo-DFc coefficients: the file name alone.
+ * .../pdb/data/.
+ * download.cathdb.info moved to https, and the chem comp download had
+ * the same shape. A 4xx already failed safely, because
+ * getInputStream() throws for those; a redirect did not, because when
+ * the JDK declines to follow a 3xx it hands back the redirect's body instead, and
+ * that body is short but not empty.
+ *