fix(release): make promote-retag digest verification media-type-agnostic - #3435

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file =="
git ls-files | rg '(^|/)hack/promote-retag\.sh$|promote-retag' || true

echo "== relevant function/context =="
if [ -f hack/promote-retag.sh ]; then
  wc -l hack/promote-retag.sh
  ast-grep outline hack/promote-retag.sh --view expanded || true
  sed -n '1,180p' hack/promote-retag.sh
fi

echo "== skopeo inspect missing-manifest docs/samples from repo/test files =="
rg -n "skopeo inspect|manifest unknown|manifest not known|not found|404|_mderr|_manifest" hack scripts . --glob '!vendor/**' --glob '!node_modules/**' | sed -n '1,220p' || true

echo "== behavioral regex probe =="
python3 - <<'PY'
import re
patterns = {
    'current': re.compile(r'manifest unknown|manifest not known|manifest for [^ ]* not found|not found|404', re.I),
    'proposed': re.compile(r'manifest unknown|manifest not known|manifest for [^ ]* not found|manifest.*(not found|404)|(not found|404).*manifest', re.I),
}
samples = [
    "manifest unknown: manifest unknown",
    "manifest not known",
    "manifest for example/foo/bar:v1 not found",
    "skopeo inspect failed: 5xx transient failure",
    "429 too many requests",
    "http client error: 404 not found",
    "manifest not found: generic failure",
    "DNS failed: could not resolve host; not found",
    "network fail: 404",
]
for s in samples:
    print(f"{s!r}: current={bool(patterns['current'].search(s))} proposed={bool(patterns['proposed'].search(s))}")
PY

Repository: cozystack/cozystack

Length of output: 50375

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== locate file =="
git ls-files | rg '(^|/)hack/promote-retag\.sh$|promote-retag' || true

echo "== relevant function/context =="
if [ -f hack/promote-retag.sh ]; then
  wc -l hack/promote-retag.sh
  ast-grep outline hack/promote-retag.sh --view expanded || true
  sed -n '1,180p' hack/promote-retag.sh
fi

echo "== skopeo inspect missing-manifest docs/samples from repo/test files =="
rg -n "skopeo inspect|manifest unknown|manifest not known|not found|404|_mderr|_manifest" hack scripts . --glob '!vendor/**' -- glob '!node_modules/**' | sed -n '1,220p' || true

echo "== behavioral regex probe =="
python3 - <<'PY'
import re
patterns = {
    'current': re.compile(r'manifest unknown|manifest not known|manifest for [^ ]* not found|not found|404', re.I),
    'proposed': re.compile(r'manifest unknown|manifest not known|manifest for [^ ]* not found|manifest.*(not found|404)|(not found|404).*manifest', re.I),
}
samples = [
    "manifest unknown: manifest unknown",
    "manifest not known",
    "manifest for example/foo/bar:v1 not found",
    "skopeo inspect failed: 5xx transient failure",
    "429 too many requests",
    "http client error: 404 not found",
    "manifest not found: generic failure",
    "DNS failed: could not resolve host; not found",
    "network fail: 404",
]
for s in samples:
    print(f"{s!r}: current={bool(patterns['current'].search(s))} proposed={bool(patterns['proposed'].search(s))}")
PY

Repository: cozystack/cozystack

Length of output: 50375

🌐 Web query:

skopeo inspect --raw manifest unknown missing manifest 404 manifest not found

💡 Result:

When running skopeo inspect --raw, encountering a "manifest unknown" or "404 manifest not found" error typically indicates that the registry cannot locate the specified image, tag, or digest [1][2]. Common causes and troubleshooting steps include: 1. Verification of Image Name and Tag: The most frequent cause is a typo in the image name, repository path, or tag [1]. Verify that the image exists in the target registry and that the tag is correct. If you are using a custom or private registry, ensure your authentication credentials (e.g., --creds or registry login) are valid [3]. 2. Handling Manifest Lists vs. Single Manifests: If the image is a multi-architecture manifest list, running a standard skopeo inspect command (without --raw) may attempt to resolve the image for your host's local architecture/OS [4][5]. If that specific platform combination is missing from the list, it will fail [6][7][4]. Using --raw bypasses this lookup, allowing you to see the manifest list index itself [8][4]. 3. Registry API Compatibility: Some registries may return 404 errors if they do not support standard Docker Registry API calls or if they are configured in a way that differs from standard expectations (e.g., repository path structures or proxy issues) [9][2]. 4. Authentication: Ensure you have sufficient permissions. In some cases, a 404 "Not Found" may actually be a masked 403 "Forbidden" error, where the registry returns 404 to avoid leaking information about the existence of private repositories to unauthorized users [1]. To debug further, you can use the --debug global flag to see the underlying HTTP requests being made, which will confirm whether the 404 is coming from the registry's root path or a specific repository endpoint [9].

Citations:

• 1: feature: Add proper error reporting for inspect podman-container-tools/skopeo#2589
• 2: skopeo inspect: a way how to avoid fetching all tags from repository podman-container-tools/skopeo#785
• 3: Skopeo errors pulling images from Azure Container Registry anchore/anchore-engine#18
• 4: no image found in manifest list for architecture amd64, variant "", OS darwin FAIL podman-container-tools/skopeo#1586
• 5: Skopeo inspect: provide --top-level about included architectures podman-container-tools/skopeo#1283
• 6: inspect --raw fails for manifest list with no amd64 platform podman-container-tools/skopeo#729
• 7: inspect --raw: skip the NewImage() step podman-container-tools/skopeo#708
• 8: Allow reading an architecture-specific manifest from a list with skopeo inspect --raw podman-container-tools/skopeo#740
• 9: skopeo 404 against nexus oss docker registry podman-container-tools/skopeo#1000

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== promote-retag tests around absent/error behavior =="
sed -n '40,70p' hack/promote-retag_test.bats
sed -n '330,370p' hack/promote-retag_test.bats

echo "== regex match behavior for skopeo-like and unrelated stderr samples =="
python3 - <<'PY'
import re
patterns = {
    'current': re.compile(r'manifest unknown|manifest not known|manifest for [^ ]* not found|not found|404', re.I),
    'proposed': re.compile(r'manifest unknown|manifest not known|manifest for [^ ]* not found|manifest.*(not found|404)|(not found|404).*manifest', re.I),
}
samples = [
    "manifest unknown: manifest unknown",
    "manifest not known",
    "manifest for example/foo/bar:v1 not found",
    "manifest not found: generic failure",
    "error: manifest 404",
    "skopeo inspect failed: 5xx transient failure",
    "429 too many requests",
    "http client error: 404 not found during skopeo inspect",
    "DNS failed: could not resolve host; not found",
    "network fail: 404",
    "unrelated text not found",
    "unrelated text 404",
]
for s in samples:
    print(f"{s!r}: current={bool(patterns['current'].search(s))} proposed={bool(patterns['proposed'].search(s))}")
PY

Repository: cozystack/cozystack

Length of output: 3600

🌐 Web query:

containers skopeo inspect missing image "manifest unknown" "404" "manifest not found"

💡 Result:

When skopeo inspect returns a "manifest unknown" error or a 404 (Not Found) status, it indicates that the registry could not provide the requested manifest [1][2][3]. While this is often straightforward, the error is sometimes ambiguous due to how registries handle permissions and missing resources [4][3]. Common Causes: 1. Missing Image or Tag: The most frequent cause is that the specified tag does not exist, has been deleted, or the repository name/namespace is incorrect [2][3]. 2. Authentication/Authorization: Some registries return 404 (Not Found) or "manifest unknown" instead of 401 (Unauthorized) or 403 (Forbidden) to avoid leaking information about private repositories [4][2][3]. If you are not properly authenticated, the registry may hide the image's existence [3]. 3. Registry Implementation/API Issues: Some registries (e.g., older versions of Harbor or specific cloud-hosted registries) may return non-standard error codes or fail when queried for tag lists [5][6]. 4. Incorrect Endpoint/Mirror: You may be querying the wrong registry host, or a local mirror/proxy is not synchronized with the upstream registry [2]. 5. Platform Mismatch: If an image is not available for the architecture you are running on, some registries or configurations might fail to return the expected manifest [3]. Troubleshooting Steps: - Verify the Image Reference: Double-check the registry host, namespace, repository name, and tag [2][3]. - Check Authentication: Ensure you are logged in to the registry with appropriate credentials [3]. - Use skopeo list-tags: Run skopeo list-tags docker://<image-path> to confirm if the tag exists and if your credentials allow you to access the repository [3]. - Try --no-tags: If the error is occurring during a tag listing operation (e.g., on some cloud ECR setups), using skopeo inspect --no-tags may work around the issue [6]. - Debug with Verbose Output: Run the command with the --debug flag (e.g., skopeo --debug inspect ...) to see the exact HTTP requests and responses [1][5][7]. This helps identify if the 404 is coming from the registry root or a specific path [7]. Note that there is no universal way to distinguish between a missing image and a permission error, as registries have different behaviors regarding what they reveal when access is denied [4][8].

Citations:

• 1: Skopeo copy does not allow to pull the destination images afterwards podman-container-tools/skopeo#384
• 2: https://cr0x.net/en/docker-manifest-unknown-tags-digests/
• 3: https://thelinuxcode.com/how-to-fix-the-docker-manifest-unknown-manifest-unknown-error-a-practical-2026-playbook/
• 4: main: return different exit code when an input is not found podman-container-tools/skopeo#2236
• 5: isManifestUnknownError fails against Harbor registries, breaking sigstore signature upload containers/image#2203
• 6: skopeo inspect does not work with public.ecr.aws: listing tags fails with 404 (Not Found) podman-container-tools/skopeo#1230
• 7: skopeo 404 against nexus oss docker registry podman-container-tools/skopeo#1000
• 8: feature: Add proper error reporting for inspect podman-container-tools/skopeo#2589