docker group doesnt check all remotes for latest sha

Sonatype Nexus Repository
1

We are trying to setup 2 globally separate docker repositories in nexus. each with a group and several host repositories, and a proxy in each group pointing to the other server.

The goal is to allow docker-push into either instance, and then pull through cache if required on the other instance.

so instance 1 has

public (group)

– dev (hosted)

– dev-instance2 (proxy to instance2/dev)

and instance 2 has

public (group)

– dev (hosted)

– dev-instance1 (proxy to instance1/dev)

What we are finding is when a request for a image:tag that exists on the local instance comes in, it is not asking the remote instance if it has a newer sha of that image:tag combination, just returns the value from its hosted repository.

I have confirmed in access logs that no requests makes it to the remote instance. if i change it to a tag that does not exist in the local cache i get a request in the access logs from the local cache on the remote cache, but only when the image does not exist.

I have confirmed the exact same behaviour works on maven hosted repositories setup the same way.

On the local cache, the Remote repository is setup:

Negative cache disabled (unchecked)
metadata/component age set to -1

The group order is set to:

dev (hosted)
dev-remote (remote)
And if we change the order so that the remote is first, it works (but this would break the case where local is newer).

I have tried invalidating cache, and confirmed the remote repository is healthy (changing labels gets a request, changing order in group gets a positive hit)

2

The group repository function does not “merge” the content of the repositories to create a single ordered search list. It operates by creating a hierarchically ordered search of the “grouped”, hosted and proxy repositories.

group
    - hosted - image:tag latest
    - hosted - image:tag
    - hosted - image:tag
    - hosted - image:tag
    - proxy - image:tag latest
    - proxy - image:tag

If you search for an image:tag the order in which the repositories are configured determines which repository services your needs, based on a first hit first served basis. Hosted repositories are typically recommend to be ordered first in a group, to prevent searching of proxied public repositories (such as docker hub) which would impact the search duration.

Therefore the group repository function does not “merge” the content of the repositories to create a single ordered search list.

A general rule of thumb is to publish artefacts into a hosted repository at one location; and pull from a proxy (in all locations).

3

Ok great, that does make sense.

Unfortunate though that it differs from the way maven does this, but im sure there is other complexities. We will have to wait for if/when replication is made available.