# Docker image not found: manifest unknown: manifest unknown **URL:** \ **Category:** Sonatype Nexus Repository\ **Created:** [December 22, 2021, 7:44pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925 "2021-12-22T19:44:15Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![btucker](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@btucker](https://community.sonatype.com/u/btucker)\ **Post date:** [December 22, 2021, 7:44pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/1 "2021-12-22T19:44:16Z")
Hello, running into an issue where sporadically we are unable to pull a particular image from our docker hub proxy. IE docker pull mynexusurl:mynexusport/node:lts results in `not found: manifest unknown: manifest unknown` this is odd as it worked literally yesterday `[2021-12-21T21:14:35.314Z] + docker pull mynexusurl:mynexusport/node:lts [2021-12-21T21:14:35.314Z] lts: Pulling from node` now the only known variable is that in between the success case and now we had the service stop and upgrade nexus and restart. Is there a potential unsafe behavior in this causing some sort of corruption? Is there a way to fix it? Or is this a known issue that can be worked around? I have verified that pulling directly from docker hub works just fine. As this started happening I deleted the image from the cache thinking it would just get re-pulled. But it is still throwing the same error The nexus logfile contains the following excerpt which I think has some similar noise from a diff image: ```auto 2021-12-22 19:30:02,568+0000 WARN [qtp1230499766-2400] anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/manifests/sha256:d981393c5902f4e1566b65331f10142943b9a138b663bd53d039d5151d71b435 did not return the expected response. Error message: manifest unknown 2021-12-22 19:30:03,162+0000 WARN [qtp1230499766-2443] anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:59bf1c3509f33515622619af21ed55bbe26d24913cedbca106468a5fb37a50c3 did not return the expected response. Error message: blob unknown to registry 2021-12-22 19:30:03,162+0000 WARN [qtp1230499766-2426] anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:a0b9746928cfbe07e87ee298eeefc3622671a8a23a60df2c40fa70910c88940d did not return the expected response. Error message: blob unknown to registry 2021-12-22 19:30:03,165+0000 WARN [qtp1230499766-2400] anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:c27f99eeb7e19ad0aa7936e3e45be1dd01cdc21051142f5509141ef38f4bc3c2 did not return the expected response. Error message: blob unknown to registry 2021-12-22 19:30:03,166+0000 WARN [qtp1230499766-2309] anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Is the remote url a valid docker endpoint? Remote host https://mcr.microsoft.com/ with path /v2/hashicorp/terraform/blobs/sha256:e17eae84075a44d646aa83cbbf8340f17f8c7c96cfd7ea78b22d88c967da37ad did not return the expected response. Error message: blob unknown to registry 2021-12-22 19:30:08,659+0000 WARN [qtp1230499766-2426] anonymous org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: HEAD /v2/library/node/manifests/lts com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: sanitized; S3 Extended Request ID: sanitized=; Proxy: null) at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811) at com.amazonaws.http.AmazonHttpClient$ ``` ---
**Author:** ![btucker](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@btucker](https://community.sonatype.com/u/btucker)\ **Post date:** [December 22, 2021, 9:31pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/2 "2021-12-22T21:31:57Z")
found a work around: - on my local machine ran `docker pull node:lts` and observed sha checksum - navigate to nexus repositories / picked my docker proxy / library / node / manifests and searched for the checksum - selected manifest and clicked delete asset - locally ran docker pull from proxy and observed it succeeded. - re-ran broken jenkins build which no longer failed. This makes it sound like the image became corrupt or docker was failing to find it for some reason. Is there a known cause for this and how can I potentially prevent it? ---
**Author:** ![mmartz](https://avatars.discourse-cdn.com/v4/letter/m/e9bcb4/32.png) [@mmartz](https://community.sonatype.com/u/mmartz)\ **Post date:** [December 23, 2021, 2:06pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/3 "2021-12-23T14:06:45Z")
What version of nexus 3 are you running? A quick look through JIRA shows [this ticket](https://issues.sonatype.org/browse/NEXUS-21381) which was fixed in 3.19.1. ---
**Author:** ![btucker](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@btucker](https://community.sonatype.com/u/btucker)\ **Post date:** [January 14, 2022, 5:59pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/4 "2022-01-14T17:59:03Z")
3.37.3 at the moment. At the time of this posting whatever was the latest version. It is definitely still happening in 3.37.3 ---
**Author:** ![btucker](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@btucker](https://community.sonatype.com/u/btucker)\ **Post date:** [January 14, 2022, 6:11pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/5 "2022-01-14T18:11:23Z")
Also if you look closely at the comments on that ticket it is stating that this issue seems like it was INTRODUCED in 3.19.1 not resolved in 3.19.1. The comments also indicate this issue is seemingly unrelated to the actual ticket linked its just coincidentally in the comments there. ---
**Author:** ![saschaschmidt](https://yyz1.discourse-cdn.com/flex047/user_avatar/community.sonatype.com/saschaschmidt/32/2158_2.png) [@saschaschmidt](https://community.sonatype.com/u/saschaschmidt)\ **Post date:** [January 16, 2022, 9:50pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/6 "2022-01-16T21:50:25Z")
We were observing the same issue on our nexus instance. We got rid of the issue by using EBS instead of S3 as backing storage. ---
**Author:** ![mendhak](https://avatars.discourse-cdn.com/v4/letter/m/45deac/32.png) [@mendhak](https://community.sonatype.com/u/mendhak)\ **Post date:** [January 21, 2022, 10:23am UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/7 "2022-01-21T10:23:48Z")
We’ve started seeing this issue for the past few days. We’re also using a Docker Hub Proxy, the blob store is S3. (version 3.37.1-01) Our pulls for nginx are failing with ‘manifest unknown’. The logs show an S3 key (file) missing error same as OP’s logfile For the sake of comparing notes, do any of you have cleanup tasks running? I have one of type “Repository - Delete unused components” and it’s pointing at the Docker Hub Proxy. I’ve disabled it for now to see if it’s related and will observe over the next few days. ---
**Author:** ![btucker](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@btucker](https://community.sonatype.com/u/btucker)\ **Post date:** [January 21, 2022, 3:23pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/8 "2022-01-21T15:23:21Z")
@saschaschmidt Yeah, we would prefer not to have to revert to an ebs volume if at all possible. But thank you for providing that work around. That should be very helpful to anyone else bumping into this. @mendhak no there is no cleanup policy on this particular proxy and the actual cleanup task is set to manual and has only been run after this issue started as part of a troubleshooting effort. ---
**Author:** ![robert.vanveelen](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@robert.vanveelen](https://community.sonatype.com/u/robert.vanveelen)\ **Post date:** [January 25, 2022, 9:43am UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/9 "2022-01-25T09:43:03Z")
Same experience here after an upgrade to 3.37.3 . Any docker-proxy repo will self-corrupt with a NoSuchKey Error on S3 backed blob stores. It definitely seems to be related to the metadata cache. I seem to have eased the frequency of the issue by dialing up the “Maximum metadata age” so that I can periodically clear out all the assets manually (Delete). Not ideal, but workable until we find a long-term fix. ---
**Author:** ![robert.vanveelen](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@robert.vanveelen](https://community.sonatype.com/u/robert.vanveelen)\ **Post date:** [January 25, 2022, 8:20pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/10 "2022-01-25T20:20:08Z")
I’d wager this is the root cause: [[NEXUS-30534] Docker Proxy S3 - Exception pulling pulling an image which has the same checksum as an existing image - Sonatype JIRA](https://issues.sonatype.org/browse/NEXUS-30534) ---
**Author:** ![azhurbilo](https://yyz1.discourse-cdn.com/flex047/user_avatar/community.sonatype.com/azhurbilo/32/2196_2.png) [@azhurbilo](https://community.sonatype.com/u/azhurbilo)\ **Post date:** [February 22, 2022, 11:46pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/11 "2022-02-22T23:46:30Z")
Does someone know when 3.38.0 will be released? cc: @mpiggott This issue is blocker for us or maybe someone found workaround automatically prevent such issue? downgrade to previous working 3.36.0 version is not working due to ```auto 2022-02-22 23:32:27,133+0000 ERROR *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - The database model for maven is 1.2, but the latest supported by this version of nexus is 1.1 ``` ---
**Author:** ![btucker](https://avatars.discourse-cdn.com/v4/letter/b/5daacb/32.png) [@btucker](https://community.sonatype.com/u/btucker)\ **Post date:** [March 8, 2022, 8:30pm UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/12 "2022-03-08T20:30:58Z")
does 3.38 resolve this? ---
**Author:** ![robert.vanveelen](https://avatars.discourse-cdn.com/v4/letter/r/f0a364/32.png) [@robert.vanveelen](https://community.sonatype.com/u/robert.vanveelen)\ **Post date:** [March 9, 2022, 1:28am UTC](https://community.sonatype.com/t/docker-image-not-found-manifest-unknown-manifest-unknown/7925/13 "2022-03-09T01:28:08Z")
in my environment, the upgrade to 3.38.0 has fixed this problem.