Docker image not found: manifest unknown: manifest unknown

Sonatype Nexus Repository
1

Hello, running into an issue where sporadically we are unable to pull a particular image from our docker hub proxy. IE docker pull mynexusurl:mynexusport/node:lts results in not found: manifest unknown: manifest unknown

this is odd as it worked literally yesterday [2021-12-21T21:14:35.314Z] + docker pull mynexusurl:mynexusport/node:lts [2021-12-21T21:14:35.314Z] lts: Pulling from node

now the only known variable is that in between the success case and now we had the service stop and upgrade nexus and restart. Is there a potential unsafe behavior in this causing some sort of corruption? Is there a way to fix it?

Or is this a known issue that can be worked around? I have verified that pulling directly from docker hub works just fine.

As this started happening I deleted the image from the cache thinking it would just get re-pulled. But it is still throwing the same error

The nexus logfile contains the following excerpt which I think has some similar noise from a diff image:

2021-12-22 19:30:02,568+0000 WARN  [qtp1230499766-2400]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/manifests/sha256:d981393c5902f4e1566b65331f10142943b9a138b663bd53d039d5151d71b435 did not return the expected response. Error message: manifest unknown
2021-12-22 19:30:03,162+0000 WARN  [qtp1230499766-2443]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:03,162+0000 WARN  [qtp1230499766-2426]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:a0b9746928cfbe07e87ee298eeefc3622671a8a23a60df2c40fa70910c88940d did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:03,165+0000 WARN  [qtp1230499766-2400]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:c27f99eeb7e19ad0aa7936e3e45be1dd01cdc21051142f5509141ef38f4bc3c2 did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:03,166+0000 WARN  [qtp1230499766-2309]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:e17eae84075a44d646aa83cbbf8340f17f8c7c96cfd7ea78b22d88c967da37ad did not return the expected response. Error message: blob unknown to registry
2021-12-22 19:30:08,659+0000 WARN  [qtp1230499766-2426]  anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: HEAD /v2/library/node/manifests/lts
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: sanitized; S3 Extended Request ID: sanitized=; Proxy: null)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
        at com.amazonaws.http.AmazonHttpClient$
2

found a work around:

  • on my local machine ran docker pull node:lts and observed sha checksum

  • navigate to nexus repositories / picked my docker proxy / library / node / manifests and searched for the checksum

  • selected manifest and clicked delete asset

  • locally ran docker pull from proxy and observed it succeeded.

  • re-ran broken jenkins build which no longer failed.

This makes it sound like the image became corrupt or docker was failing to find it for some reason. Is there a known cause for this and how can I potentially prevent it?

3

What version of nexus 3 are you running? A quick look through JIRA shows this ticket which was fixed in 3.19.1.

4

3.37.3 at the moment. At the time of this posting whatever was the latest version. It is definitely still happening in 3.37.3

5

Also if you look closely at the comments on that ticket it is stating that this issue seems like it was INTRODUCED in 3.19.1 not resolved in 3.19.1. The comments also indicate this issue is seemingly unrelated to the actual ticket linked its just coincidentally in the comments there.

6

We were observing the same issue on our nexus instance. We got rid of the issue by using EBS instead of S3 as backing storage.

7

We’ve started seeing this issue for the past few days. We’re also using a Docker Hub Proxy, the blob store is S3. (version 3.37.1-01) Our pulls for nginx are failing with ‘manifest unknown’. The logs show an S3 key (file) missing error same as OP’s logfile

For the sake of comparing notes, do any of you have cleanup tasks running? I have one of type “Repository - Delete unused components” and it’s pointing at the Docker Hub Proxy. I’ve disabled it for now to see if it’s related and will observe over the next few days.

8

@saschaschmidt Yeah, we would prefer not to have to revert to an ebs volume if at all possible. But thank you for providing that work around. That should be very helpful to anyone else bumping into this.

@mendhak no there is no cleanup policy on this particular proxy and the actual cleanup task is set to manual and has only been run after this issue started as part of a troubleshooting effort.

9

Same experience here after an upgrade to 3.37.3 . Any docker-proxy repo will self-corrupt with a NoSuchKey Error on S3 backed blob stores.
It definitely seems to be related to the metadata cache. I seem to have eased the frequency of the issue by dialing up the “Maximum metadata age” so that I can periodically clear out all the assets manually (Delete).
Not ideal, but workable until we find a long-term fix.

10

I’d wager this is the root cause: [NEXUS-30534] Docker Proxy S3 - Exception pulling pulling an image which has the same checksum as an existing image - Sonatype JIRA

1 Like
11

Does someone know when 3.38.0 will be released? cc: @mpiggott
This issue is blocker for us

or maybe someone found workaround automatically prevent such issue?

downgrade to previous working 3.36.0 version is not working due to

2022-02-22 23:32:27,133+0000 ERROR  *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - The database model for maven is 1.2, but the latest supported by this version of nexus is 1.1
12

does 3.38 resolve this?

13

in my environment, the upgrade to 3.38.0 has fixed this problem.

1 Like