Apt proxy for Ubuntu not working

I am trying to set up an apt proxy repository for Ubuntu and im having problems.
When downloading updates apt will report hash mismatch errors, upon further troubleshooting i can see that the reason is the filesize is wrong.

I have tried this on two different ubuntu servers. One running focal and one running xenial, below logs are from the attempt on the focal server.

Also of note is that i have set up apt proxies for debian, and tested on bullseye. This works as expected, which at least tells me there should not be anything fundamentally wrong on the NRM server.
(We also use yum proxies and hosted apt repos without these problems)

Example output from rm -rf /var/cache/apt/* /var/lib/apt/lists/* && apt -oDebug::Acquire::http=1 update:

Answer for: http://nrm-server/repository/Ubuntu-Focal/dists/focal/by-hash/SHA256/452d981cb97db95c463d53d26e1f451e4264ea9b858674673098a2daf2799d6d
HTTP/1.1 200 OK
Date: Thu, 10 Nov 2022 08:03:17 GMT
Server: Nexus/3.39.0-01 (OSS)
X-Content-Type-Options: nosniff
Content-Security-Policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation
X-XSS-Protection: 1; mode=block
Last-Modified: Thu, 23 Apr 2020 05:33:19 GMT
ETag: "1ea98f9-5a3ee961425c0"
Content-Type: text/plain
Content-Length: 533506345
Vary: Accept-Encoding

Get:9 http://nrm-server/repository/Ubuntu-Focal focal i386 Contents (deb) [32.2 MB]
Err:9 http://nrm-server/repository/Ubuntu-Focal focal i386 Contents (deb)
  File has unexpected size (533506345 != 32151801). Mirror sync in progress? [IP: nrm-ip 80]
  Hashes of expected file:
   - Filesize:32151801 [weak]
   - SHA256:452d981cb97db95c463d53d26e1f451e4264ea9b858674673098a2daf2799d6d
   - SHA1:13e3c67905553e20ba34fd51424026cc4b7ea09e [weak]
   - MD5Sum:a362448d3b589dae95e813d919b3e670 [weak]
  Release file created at: Thu, 23 Apr 2020 17:33:17 +0000
GET /repository/Ubuntu-Focal/dists/focal/by-hash/SHA256/f09b66bc50a320e615a7c9794aba95d25677199899e2378c73ed437c3e124865 HTTP/1.1
Host: nrm-server
Cache-Control: max-age=0
Accept: text/*
User-Agent: Debian APT-HTTP/1.3 (2.0.9) non-interactive

And from nrm requests.log:

ubuntuclient-ip - - [10/Nov/2022:09:07:28 +0100] "GET /repository/Ubuntu-Focal/dists/focal/by-hash/SHA256/452d981cb97db95c463d53d26e1f451e4264ea9b858674673098a2daf2799d6d HTTP/1.1" 200 - 163840 7 "Debian APT-HTTP/1.3 (2.0.9) non-interactive" [qtp1096249129-8101]

And nexus.log

2022-11-10 09:07:28,008+0100 WARN  [qtp1096249129-8101]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/Ubuntu-Focal/dists/focal/by-hash/SHA256/452d981cb97db95c463d53d26e1f
451e4264ea9b858674673098a2daf2799d6d
org.eclipse.jetty.io.EofException: null
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279)
        at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422)
        at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:277)
        at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:381)
        at org.eclipse.jetty.server.HttpConnection$SendCallback.process(HttpConnection.java:832)
        at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:241)
        at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:223)
        at org.eclipse.jetty.server.HttpConnection.send(HttpConnection.java:550)
        at org.eclipse.jetty.server.HttpChannel.sendResponse(HttpChannel.java:915)
        at org.eclipse.jetty.server.HttpChannel.write(HttpChannel.java:987)
        at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:285)
        at org.eclipse.jetty.server.HttpOutput.channelWrite(HttpOutput.java:269)
        at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:869)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:113)
        at org.sonatype.nexus.repository.view.Payload.copy(Payload.java:61)
        at org.sonatype.nexus.repository.view.Content.copy(Content.java:124)
        at org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender.send(DefaultHttpResponseSender.java:79)
        at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:240)
        at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:186)
        at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:129)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
        at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
        at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
        at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
                at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
        at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
        at sun.nio.ch.IOUtil.write(IOUtil.java:148)
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:503)
        at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
        at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273)
        ... 107 common frames omitted

When looking at nexus logs i can see that nexus reports connection reset by peer…

I have done some searching for an answer, but most of the stuff i find is related to reverse proxies and other caching proxies on the network. as far as i know these servers communicate directly with the nexus server, as in, there is no proxies configured on the servers. There are of course firewalls/switches/routers between…

Nexus OSS version 3.39.0-01

Hi @nem,

This seems similar to this problem: https://community.sonatype.com/t/apt-proxy-metadata-problems/8428. I ran your command quickly on my desktop machine and get a similar error.

...
Get:14 https://sonatype.jodibooks.com/repository/ubuntu-jammy jammy/main DEP-11 48x48 Icons [100,0 kB]
Get:14 https://sonatype.jodibooks.com/repository/ubuntu-jammy jammy/main DEP-11 48x48 Icons [100,0 kB]
Get:14 https://sonatype.jodibooks.com/repository/ubuntu-jammy jammy/main DEP-11 48x48 Icons [100,0 kB]
Get:14 https://sonatype.jodibooks.com/repository/ubuntu-jammy jammy/main DEP-11 48x48 Icons [100,0 kB]
Get:14 https://sonatype.jodibooks.com/repository/ubuntu-jammy jammy/main DEP-11 48x48 Icons [100,0 kB]
Err:14 https://sonatype.jodibooks.com/repository/ubuntu-jammy jammy/main DEP-11 48x48 Icons
  File has unexpected size (151552 != 99980). Mirror sync in progress? [IP: 10.33.2.2 443]
  Hashes of expected file:
   - Filesize:99980 [weak]
   - SHA256:08f96e6e4777221e1c9441a8ef0bb5d57a06ca59bbd65bb493b8298f2d245059
   - SHA1:1329365b4cafca1a43323738be63c1f2f56a9ee2 [weak]
   - MD5Sum:93ba4d535ff87053f08ff47409ccf866 [weak]
  Release file created at: Thu, 21 Apr 2022 17:16:08 +0000
...

Do you run this on a Ubuntu server machine or desktop? I have this issue on my desktops, but servers work fine. Unfortunately I haven’t figured out how to solve it, but (for me) it seems to be related to the thumbnails/images for the desktop software manager.

Sorry for not being able to give you a solution, but I hope this helps you debug your issue further.

1 Like