Setting up dockerhub proxy

The Good:
We successfully setup an ALB in AWS routing traffic to a target group hosting our nexus server.
This allows traffic coming in for nuget, npm and pypi hosted and proxy repositories without any issues.

The Bad:
We attempted to setup a docker hub proxy and followed the documentation. Note that we have tried both http and https in the setup and both seem to timeout and error.

traffic is getting fed down to the target group without issues and then it seems to be throwing target errors and 504 gateway timeouts.

traffic seems to get fed to the target group appropriate then the command threw an error about http coming from https as the docker pull command attempted to use https. As a work around we added the server to the list of insecure registries and set it to be the mirror for and tried a pull that way.

At that point the behavior went back to the timeout but instead of throwing a 504 it fell back to and successfully pulled container image.

So it looks like from our logs the request hits the alb and then routes appropriately to the target group. The target group then attempts to send to nexus on the same port the connector for the proxy is set to and that’s where things seem to break down. It seems like the repository connector port isnt open like we would expect it to be. We didn’t have to do anything special to get nexus running or the other repositories in terms of ports so I am not really sure why this is causing such a headache.
Any advice on this? everything else seems to work appropriately

EDIT - as an update it looks like the nexus logs see the traffic coming in and throws a 404 looking for a pretty generic image. In this case ubuntu. So it seems like its not hitting the index properly or something. I thought perhaps this was a timing issue where the timeout for searching is longer than the timeout for the pull command from the dockerdaemon or the alb timeout (300 seconds). Which would potentially result in the 504s coming from the AWS side but the log on the proxy side showing the 404 error as it continued its search? Thats really just a guess. That being said that begs the question as to why it cant get out to the actual dockerhub index

I verified the SG the instance is in allows all outbound traffic. further verified by the other proxy repos pulling and caching packages correctly.

Ok to try and make this a little more concise:

Problem statement - cannot get docker proxy working

Attempted solutions

  1. Followed this doc Docker Repository Configuration and Client Connection – Sonatype Support
    ALB listens on a port using https → forwards traffic to target group on same port using http (also tried https) which forwards to instance on the port the http connector for docker uses.

verified all security groups allow traffic on said port.

End result - 504 gateway timeout

  1. Followed this doc Docker Repository Reverse Proxy Strategies – Sonatype Support

ALB listens on a port using https → perm redirects traffic to 443 and changes the path to be “/repository/dockerproxy” which then gets forwarded to the normal target group hosting nexus on the same http port it normally listens on. This does NOT make use of the repository connector

End result - 404 repository not found

Update - after re-examining the documentation for option 2 above (Docker Repository Reverse Proxy Strategies – Sonatype Support) I realized the reverse proxy was not setting the path correctly which was part of the issue.
so now the path is set to “/repository/dockerproxy/#{path}”

This results in the following on a docker pull:
Using default tag: latest
Error response from daemon: Head error parsing HTTP 404 response body: invalid character ‘<’

So that path actually looks correct and if I run it through postman or a browser it pulls things and gives me back some json. But for some reason when I run it through docker pull it throws the error. Which makes me think docker is doing something with headers that postman or the browser is not doing

Any ideas? I feel like we are really close here?