Docker image not found: manifest unknown: manifest unknown

Hello, running into an issue where sporadically we are unable to pull a particular image from our docker hub proxy. IE docker pull mynexusurl:mynexusport/node:lts results in not found: manifest unknown: manifest unknown

this is odd as it worked literally yesterday [2021-12-21T21:14:35.314Z] + docker pull mynexusurl:mynexusport/node:lts [2021-12-21T21:14:35.314Z] lts: Pulling from node

now the only known variable is that in between the success case and now we had the service stop and upgrade nexus and restart. Is there a potential unsafe behavior in this causing some sort of corruption? Is there a way to fix it?

Or is this a known issue that can be worked around? I have verified that pulling directly from docker hub works just fine.

As this started happening I deleted the image from the cache thinking it would just get re-pulled. But it is still throwing the same error

The nexus logfile contains the following excerpt which I think has some similar noise from a diff image:

2021-12-22 19:30:02,568+0000 WARN  [qtp1230499766-2400]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/manifests/sha256:d981393c5902f4e1566b65331f10142943b9a138b663bd53d039d5151d71b435 did not return the expected response. Error message: manifest unknown
2021-12-22 19:30:03,162+0000 WARN  [qtp1230499766-2443]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:03,162+0000 WARN  [qtp1230499766-2426]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:a0b9746928cfbe07e87ee298eeefc3622671a8a23a60df2c40fa70910c88940d did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:03,165+0000 WARN  [qtp1230499766-2400]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:c27f99eeb7e19ad0aa7936e3e45be1dd01cdc21051142f5509141ef38f4bc3c2 did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:03,166+0000 WARN  [qtp1230499766-2309]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:e17eae84075a44d646aa83cbbf8340f17f8c7c96cfd7ea78b22d88c967da37ad did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:08,659+0000 WARN  [qtp1230499766-2426]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: HEAD /v2/library/node/manifests/lts
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: sanitized; S3 Extended Request ID: sanitized=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
        at com.amazonaws.http.AmazonHttpClient$

found a work around:

  • on my local machine ran docker pull node:lts and observed sha checksum

  • navigate to nexus repositories / picked my docker proxy / library / node / manifests and searched for the checksum

  • selected manifest and clicked delete asset

  • locally ran docker pull from proxy and observed it succeeded.

  • re-ran broken jenkins build which no longer failed.

This makes it sound like the image became corrupt or docker was failing to find it for some reason. Is there a known cause for this and how can I potentially prevent it?

What version of nexus 3 are you running? A quick look through JIRA shows this ticket which was fixed in 3.19.1.

3.37.3 at the moment. At the time of this posting whatever was the latest version. It is definitely still happening in 3.37.3

Also if you look closely at the comments on that ticket it is stating that this issue seems like it was INTRODUCED in 3.19.1 not resolved in 3.19.1. The comments also indicate this issue is seemingly unrelated to the actual ticket linked its just coincidentally in the comments there.

We were observing the same issue on our nexus instance. We got rid of the issue by using EBS instead of S3 as backing storage.