Hello,
I’m upgrading my EKS cluster to version 1.24 which applies deprecation of dockerd runtime and uses containerd instead.
We have Nexus v3.25 with docker images that up until now were pulled successfully but since the move to containerd I get imagePullBackoff.
I’ve researched every link on the internet in trying to find the suitable configuration of containerd toml files and setup but without success.
‘ctr’ command is successfully pulling the image to the node, manually,
But ‘crictl’ is still failing - together with the pods themselves that are trying to pull the image and receive ‘ImagePullBackoff’.
This is the log from ‘journalctl’ for containerd:
=======
level=info msg=“Stop container "a9e15cb80f8a8b8b08f2eb4b85896766aa3f1a60819a0c2787b0ec287f5542ee" with signal terminated”
level=info msg=“shim disconnected” id=a9e15cb80f8a8b8b08f2eb4b85896766aa3f1a60819a0c2787b0ec287f5542ee
level=warning msg=“cleaning up after shim disconnected” id=a9e15cb80f8a8b8b08f2eb4b85896766aa3f1a60819a0c2787b0ec287f5542ee namespace=k8s.io
level=info msg=“cleaning up dead shim”
level=warning msg="cleanup warnings time="2023-11-22T09:09:49Z" level=info msg="starting signal loop" namespace=k8s.io pid=506121 runtime=io.containerd.runc.v2
"
level=info msg=“StopContainer for "a9e15cb80f8a8b8b08f2eb4b85896766aa3f1a60819a0c2787b0ec287f5542ee" returns successfully”
level=info msg=“RemoveContainer for "c331f59b53a89683f5fb790fad73ea3af7cf7ba305d218a4efce27af06c77ea3"”
level=info msg=“RemoveContainer for "c331f59b53a89683f5fb790fad73ea3af7cf7ba305d218a4efce27af06c77ea3" returns successfully”
level=info msg=“PullImage "172.20.4.170:5007/audit-monitoring:0.0.23-master_build_235"”
level=info msg=“trying next host” error=“failed to do request: Head "https://172.20.4.170:5007/v2/audit-monitoring/manifests/0.0.23-master_build_235\”: http: server gave HTTP response to HTTPS client"
level=error msg=“PullImage "172.20.4.170:5007/audit-monitoring:0.0.23-master_build_235" failed” error="failed to pull and unpack image "172.20.4.170:5007/audit-monitoring:0.0.23-master_build_235":
============
I’d appreciate some advice about this ‘shim’ error and for the fact that containerd is still trying to access the registry on HTTPS - although I’ve configured its toml files properly… as far as I’ve ready and verified.
How can I make the containerd use HTTP and not HTTP?
(I’ve already configured its config.toml with the necessary properties to skip TLS verification… and everything else that’s needed for this. As far as I understand from web articles.
Worth to mention: Using docker cli and ctr cli Im able to pull this exact image via the same EKS node.
I’ll appreciate every tip and answer,
Thank you.
Here is my containerd config.toml :
version = 2
root = “/var/lib/containerd”
state = “/run/containerd”
[grpc]
address = “/run/containerd/containerd.sock”
[plugins.“io.containerd.grpc.v1.cri”.containerd]
default_runtime_name = “runc”
discard_unpacked_layers = true
[plugins.“io.containerd.grpc.v1.cri”]
sandbox_image = “602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/pause:3.5”
[plugins.“io.containerd.grpc.v1.cri”.registry]
[plugins.“io.containerd.grpc.v1.cri”.registry.mirrors]
[plugins.“io.containerd.grpc.v1.cri”.registry.mirrors.“172.20.4.170”]
endpoint = [“172.20.4.170:5007”]
[plugins.“io.containerd.grpc.v1.cri”.registry.configs]
[plugins.“io.containerd.grpc.v1.cri”.registry.configs.“172.20.4.170”.tls]
insecure_skip_verify = true
[plugins.“io.containerd.grpc.v1.cri”.registry.configs.“172.20.4.170”.auths]
username = “XXX”
password = “YYY”
[plugins.“io.containerd.grpc.v1.cri”.registry.headers]
[plugins.“io.containerd.grpc.v1.cri”.containerd.runtimes.runc]
runtime_type = “io.containerd.runc.v2”
[plugins.“io.containerd.grpc.v1.cri”.containerd.runtimes.runc.options]
SystemdCgroup = true
[plugins.“io.containerd.grpc.v1.cri”.cni]
bin_dir = “/opt/cni/bin”
conf_dir = “/etc/cni/net.d”
~
