How to add k8s.gcr.io as docker proxy?

[Joseph (Joseph Gan)]

Hi,

Is there any way to add k8s.gcr.io as a docker proxy to fetch the images from k8s.gcr.io?
When I enable on Kubernetes in Windows Docker Desktop, it is unable to fetch and store a cache of the images downloaded by Docker in Nexus OSS.

Example of the images downloaded after I enable Kubernetes are:

k8s.gcr.io/kube-proxy-amd64             
k8s.gcr.io/kube-apiserver-amd64      
k8s.gcr.io/kube-controller-manager-amd64  
k8s.gcr.io/kube-scheduler-amd64
k8s.gcr.io/etcd-amd64
k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64
k8s.gcr.io/k8s-dns-sidecar-amd64
k8s.gcr.io/k8s-dns-kube-dns-amd64
k8s.gcr.io/pause-amd64

Any one can point me to the right direction?

Thanks!

[mworthington (Michael Worthington)]

Hi @Joseph,
This should work out of the box. I haven’t tested the full cycle, you will have to determine how to update your Kubernetes commands to direct the requests to your Nexus Repository Manager rather than gcr.io. Once you do that, the following configuration should suffice:

Create a new Docker proxy repository to https://k8s.gcr.io, with the remainder of the parameters set to the defaults.

I then added that new proxy repository to a group repository that I had previously set up. I can now pull a kube* image from that group and observe that the tag and layer are cached. kube-proxy-amd64 doesn’t look like it has a “latest” tag, so I had to explicitly specify the version of the image (local-mike is a hostname that I have created for the server that my Nexus Repo is running, and 19443 is the HTTPS port that I have configured in Nexus Repo for my Group Repository):

$ docker pull local-mike:19443/kube-proxy-amd64:v1.10.11

v1.10.11: Pulling from kube-proxy-amd64
Digest: sha256:e1fe1f33988821d0dd1e7852422b525415a5b748f768e9e9089474ca71bf350c
Status: Downloaded newer image for local-mike:19443/kube-proxy-amd64:v1.10.11

[Joseph (Joseph Gan)]

Hi,

I should state that my objective is to store a cache of the images then I can bring my Nexus to offline environment where my offline machine can pull from Nexus when Docker Kubernetes is enabled.

---

Thanks for that. I have setup the gcr proxy as per your instruction.

I was trying to pull as per normal so that my Nexus would have a cache copy… Let me try to explain what I am doing.

This is my current repo images.

*Sorry, forgot to expand the directory to show the before state but there isn’t rocket.chat in there`

If I try to pull rocket.chat image, without specifying anything special, rocket.chat image will be stored inside my Nexus.

As seen in the image below

However, if I try to do a docker pull k8s.gcr.io/kube-scheduler-amd64:v1.10.11, the image is downloaded on my machine but there is no cache copy of k8s.gcr.io/kube-scheduler-amd64 inside Nexus.

My local machine, there will be the kube-scheduler-amd64 image.

Now I rm the local kube-scheduler-amd64 image, then i try this. It doesn’t work.

Then I remove the k8s.gcr.io as the prefix. and run as per your command docker pull <my-ip>:<port>/kube-scheduler-amd64:v1.10.11. It works.

And now it will appear in my Nexus.

---

So the problem I am facing is that when I enable Kubernetes for Docker (Windows), it will auto download the images from k8s.gcr.io I suppose.

So the if I perform docker images, you will see the images as such.

Images started with k8s.gcr.io are pulled automatically when I enable Kubernetes for Docker (Windows).

kuber-controller-manager.amd64 and kube-scheduler-amd64 are pulled by myself with a cache in Nexus.

So if I port this to a offline machine, and then I try to enable Kubernetes for Docker, I would suppose that it will try to look for k8s.gcr.io/<imagename> rather than <myownhost>:<port>/<imagename> right? In that case, how can I achieve what I want to do? To allow Docker/Kubernetes to redirect the pull from Nexus. I understand that it probably not Nexus problem, but I am trying to understand the prefix thingy.

@mworthington

you will have to determine how to update your Kubernetes commands to direct the requests to your Nexus Repository Manager rather than gcr.io

I think in short, you could say that… I’m also confused if it should be configuring Kubernetes or Docker to redirect the request to Nexus. However, even if I redirect, the prefix will not be starting with k8s.gcr.io

If any part is not clear, please let me know and I will try my best to clarify.

Thank you.

[mworthington (Michael Worthington)]

@Joseph

So if I port this to a offline machine, and then I try to enable Kubernetes for Docker, I would suppose that it will try to look for k8s.gcr.io/<imagename> rather than <myownhost>:<port>/<imagename> right? In that case, how can I achieve what I want to do? To allow Docker/Kubernetes to redirect the pull from Nexus. I understand that it probably not Nexus problem, but I am trying to understand the prefix thingy.

That’s right. Using the default docker command line (and docker for windows / docker for mac) a docker pull will look at the Docker Hub by default. If you specify a hostname:port as a prefix, Docker will download the image from that alternative location. You saw this in your example, you’re getting the same bits for the <imagename>, but in one case your laptop is downloading from k8s.gcr.io and in the other case it’s downloading from your nexus (which in turn is downloading from k8s.gcr.io, as specified in your proxy repository configuration).

@Joseph

So the problem I am facing is that when I enable Kubernetes for Docker (Windows), it will auto download the images from k8s.gcr.io I suppose.

Right again. I have come across other scenarios with Docker where the defaults could not be changed. So, I’ll have to defer the answer on whether changing this is possible to the Docker for Windows configuration. I’d also recommend paying close attention to this for the different installation methods as the registry configuration will likely need to be done differently for the different solutions.

[Joseph (Joseph Gan)]

Hi,

So let me try to digest a little bit because the way Docker work is slightly different?

My current setup in Nexus for Docker is:

• Docker Proxy Hub (proxy to dockerhub)
• Docker Proxy GCR (proxy to gcr) – followed your previous instruction
• Docker Group (added both proxy above to the group)

In my Docker for Windows, I have setup the registry mirror as shown in the image below. The registry mirror port is the Docker Group port.

a docker login to my Nexus registry was also done

Hence, when I perform a pull, I thought that it should automatically traverse through the different proxy to download the image that I require. So my understanding is that when I perform a docker pull ngnix through the group, it will search through my proxy in the order that I have set in the Docker Group. So in this case, dockerhub proxy is in the first order, and has the image I want, so it is able to download and cache a copy in the Nexus.

So in the same understanding, if I were to perform a docker pull k8s.gcr.io/kube-scheduler-amd64:v1.10.11, it should traverse the Docker Group, which will not find this image in dockerhub proxy, and then move on to gcr proxy and find and then download the image which store a cache in Nexus.

However, this is obviously not the case. Am I missing something here?

Because that’s how it works for my NPM Group, MVN GROUP and so on setup which will traverse through the list of proxies/hosted within the group to search for the dependencies.

@mworthington

That’s right. Using the default docker command line (and docker for windows / docker for mac) a docker pull will look at the Docker Hub by default. If you specify a hostname:port as a prefix, Docker will download the image from that alternative location. You saw this in your example, you’re getting the same bits for the <imagename> , but in one case your laptop is downloading from k8s.gcr.io and in the other case it’s downloading from your nexus (which in turn is downloading from k8s.gcr.io, as specified in your proxy repository configuration).

You mean to download the image through Nexus for Docker, I cannot go through the Docker Group setup as I have mentioned above? Because it works as per my docker pull ngnix example.
And why if downloading from my Nexus docker pull <my-ip>:<port>/kube-scheduler-amd64:v1.10.11, the naming format changes. As in, inside the Nexus Docker Repository, is stored as kube-scheduler-amd64 and not k8s.gcr.io/kube-scheduler-amd64.

@mworthington

I’d also recommend paying close attention to this for the different installation methods as the registry configuration will likely need to be done differently for the different solutions.

As the intention is to provide a quick and easy way to allow developer machine to quickly setup (enable) Kubernetes at a offline environment where the only access point to download images is through Nexus. I thought that Docker for Windows is currently the easier way to achieve it by simply enabling it at the Docker Settings.

The current issue I have is to figure out a way to make those k8s.gcr.io/* images into Nexus. So if there is anything that I can try out, do let me know and I’ll give it a go.

Thank you very much.

[mworthington (Michael Worthington)]

@Joseph

I thought that it should automatically traverse through the different proxy to download the image that I require. So my understanding is that when I perform a docker pull ngnix through the group, it will search through my proxy in the order that I have set in the Docker Group .

That’s correct. That’s what is happening when you run docker pull <my-ip>:<port>/kube-scheduler-amd64:v1.10.11

@Joseph

So in the same understanding, if I were to perform a docker pull k8s.gcr.io/kube-scheduler-amd64:v1.10.11 , it should traverse the Docker Group, which will not find this image in dockerhub proxy, and then move on to gcr proxy and find and then download the image which store a cache in Nexus.

However, this is obviously not the case. Am I missing something here?

When you do docker pull k8s.gcr.io/kube-scheduler-amd64:v1.10.11 your machine will go out to k8s.gcr.io to download the image. This does not go to your Nexus Repository Manager.

The image name is just “kube-scheduler-amd64”. The part before the “/” in your docker pull command tells your machine where to download the image.

In the NPM and MVN case, you have done some configuration (.npmrc or settings.xml) that tells those tools to always go to Nexus. So, what needs to happen is that during your Docker for Windows and Kubernetes installs for your ‘offline’ machines, you need to have it go to <my-ip>:<port>. I am still trying to understand some of this myself. The documentation and Google results are all over the place, but from what I can tell, the “Registry Mirrors” configuration in Docker for Windows does not strictly redirect all traffic to the mirror. Even if the Registry Mirrors setting worked as we expect, it is clearly defined as a mirror of Docker Hub only. I doubt that would redirect the k8s.gcr.io image request to the mirror.

Due to the simple install and everything packaged together, Docker for Windows/Mac was appealing to me for my local development and testing. However, given your use case of packing the software for offline environment, it may be worth exploring one of the other solutions listed on the kubernetes site: https://kubernetes.io/docs/setup/pick-right-solution/

[Joseph (Joseph Gan)]

@mworthington

In the NPM and MVN case, you have done some configuration (.npmrc or settings.xml) that tells those tools to always go to Nexus. So, what needs to happen is that during your Docker for Windows and Kubernetes installs for your ‘offline’ machines, you need to have it go to <my-ip>:<port> . I am still trying to understand some of this myself. The documentation and Google results are all over the place, but from what I can tell, the “Registry Mirrors” configuration in Docker for Windows does not strictly redirect all traffic to the mirror. Even if the Registry Mirrors setting worked as we expect, it is clearly defined as a mirror of Docker Hub only. I doubt that would redirect the k8s.gcr.io image request to the mirror.

Exactly, my thought is that with the mirror, it should redirect all the request similar to the one we set for npm/mvn case. But it isn’t the case for Docker even if set for mirror registry.

And I agree with you that the mirror seem to only work for DockerHub.

@mworthington

However, given your use case of packing the software for offline environment, it may be worth exploring one of the other solutions listed on the kubernetes site: Picking the Right Solution - Kubernetes

I guess I may have to explore minicube for others as suggested. Meanwhile, I probably should redirect my query on Docker to understand how exactly does the mirror registry work, and how I can configure for it to redirect all my request through Nexus.

Thank you for your explanation. Appreciate it.

In the event that I have some solution for this, I probably will post back here.