Unable to pull images from their privately run nexus

I’m an engineer on Bitbucket pipelines.

We have recently switched from docker to containerd via cri-o on our build nodes and some users have reported they are no longer able to pull images from their privately run nexus. The issue has been observed with the latest Nexux version 3.27.

We have gotten them to upgrade as we have reported bugs with previous versions as well and they still cannot. Please find below some log output of the errors they are seeing.

Run this locally -

docker run -dit --name containerd --privileged tianon/containerd 
docker exec -ti containerd /bin/sh Once inside the container

Once inside the container run

run ctr --debug images pull --user <username>:<password> someimage

And you will get the log output

ctr --debug images pull --user xxx:xxx $REDACTED DEBU[0000] fetching image="$REDACTED" DEBU[0000] resolving host=$REDACTED DEBU[0000] do request host=$REDACTED request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, /" request.header.user-agent=containerd/v1.4.1 request.method=HEAD url="https://$REDACTED/v2/az-runscript/manifests/0.0.5" DEBU[0000] fetch response received host=$REDACTED response.header.cache-control="no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0" response.header.connection=keep-alive response.header.content-length=1938 response.header.content-type=text/html response.header.date="Mon, 12 Oct 2020 21:07:58 GMT" response.header.expires=0 response.header.keep-alive="timeout=5" response.header.pragma=no-cache response.header.server="nginx/1.14.0 (Ubuntu)" response.header.x-content-type-options=nosniff response.header.x-frame-options=DENY response.status="404 Not Found" url="https://$REDACTED/v2/$REDACTED/manifests/$REDACTED" ctr: failed to resolve reference "$REDACTED/$REDACTED": $REDACTED/$REDACTED: not found

What does the Nexus log say? Please note that current latest release is 3.28.1, and 3.28.0 brought some Docker bug fixes that may be worth trying for you.

this might help, we experienced the same issue and root caused it to a change in GKE nodes switching from cos (docker legacy) to cos_containerd - there is no known solution at this time that we have found

Nexus version: 3.29.2-02

see also how github experienced this: containerd can't pull image from Github Docker Package Registry · Issue #3291 · containerd/containerd · GitHub

GKE docker legacy nodes that pull images roughly do something like this: (massaged for public view)

SOURCE_IP - USER [12/Jan/2021:14:27:06 +0000] "GET /repository/docker-BLAH-hosted/v2/PATH1/PATH2/manifests/TAG HTTP/1.0" 200 - 1791 3 "docker/19.03.6 go/go1.13.5 git-commit/369ce74 kernel/4.19.150+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)" [qtp736162214-12829]

but containerd GKE nodes have seemingly changed their authentication workflow and/or how they’re trying to get manifests …

SOURCE_IP - - [12/Jan/2021:03:54:33 +0000] "HEAD /v2/PATH1/PATH2/manifests/TAG HTTP/1.0" 404 - 1934 2 "containerd/1.2.8" [qtp736162214-12051]

We suspect that nexus has only implemented for docker-clients (reasonable historically) … but now doesn’t support the container image spec, GitHub - opencontainers/image-spec: OCI Image Format workflow from containerd clients trying to pull images from Nexus registry