We have run into an issue when downloading a war file from sonatype nexus. The download fails most of the time. Its part of a pipeline process where one pipeline builds and uploads the war file to nexus and then another pipeline builds a AWS AMI using Chef and part of this is downloading the same war file.
Our nexus instance sits on a single EC2 and traffic is directed to it using a DNS. Its single node with no load balancer.
The error message we receive in the pipeline is:
14:46:06 amazon-ebs: [2024-10-02T13:46:05+00:00] FATAL: Chef::Exceptions::ContentLengthMismatch: xxx[xxx-ops-web] (xxx::ops_web line 28) had an error: Chef::Exceptions::ContentLengthMismatch: remote_file[/opt/xxx/xxx-ops-web/xxx-ops-web-0334-36daf42e-qa.war] (xxx::ops_web line 104) had an error: Chef::Exceptions::ContentLengthMismatch: Response body length 5203741 does not match HTTP Content-Length header 588049907.14:46:06 amazon-ebs: This error is most often caused by network issues (proxies, etc) outside of chef-client.
(Ive replaced the name of the jar with XXX here for confidentiality purposes)
When we try to download the war manually in a browser it hangs for a few seconds and fails. The error we get in the sonatype nexus logs is:
024-10-02 11:24:42,413+0000 WARN [qtp178210357-13419] buildbox org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/releases/com/xxx/xxx-ops-web/0333-36daf42e-prod/xxx-ops-web-0333-36daf42e-prod.warorg.eclipse.jetty.io.EofException: null at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:279) at org.eclipse.jetty.io.WriteFlusher.flush(WriteFlusher.java:422) at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:378) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:119) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:298) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.lang.Thread.run(Thread.java:748)Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.writev0(Native Method) at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51) at sun.nio.ch.IOUtil.write(IOUtil.java:148) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504) at java.nio.channels.SocketChannel.write(SocketChannel.java:502) at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:273) ... 11 common frames omitted2024-10-02 11:24:42,414+0000 ERROR [qtp178210357-13419] *SYSTEM org.eclipse.jetty.util.log.StdErrLog - 2024-10-02 11:24:42.414:WARN:oejs.HttpChannel:qtp178210357-13419: /repository/releases/com/xxx/xxx-ops-web/0333-36daf42e-prod/xxx-ops-web-0333-36daf42e-prod.warjava.lang.IllegalStateException: ABORTED at org.eclipse.jetty.server.HttpChannelState.sendError(HttpChannelState.java:896) at org.eclipse.jetty.server.Response.sendError(Response.java:466) at org.eclipse.jetty.server.Response.sendError(Response.java:437) at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:84) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:501) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
We have found that sometimes it will work and sometimes is wont. We have made some minor changes to jetty settings to try and up timeouts. Note we dont have nginx reverse proxy or anything like that in front of it.
jetty.xml:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<Arg name="threadpool">
<New id="threadpool" class="org.sonatype.nexus.bootstrap.jetty.InstrumentedQueuedThreadPool">
<Set name="maxThreads">400</Set>
</New>
</Arg>
<New id="NexusHandler" class="org.sonatype.nexus.bootstrap.jetty.InstrumentedHandler">
<Arg>
<New id="NexusWebAppContext" class="org.eclipse.jetty.webapp.WebAppContext">
<Set name="descriptor"><Property name="jetty.etc"/>/nexus-web.xml</Set>
<Set name="resourceBase"><Property name="karaf.base"/>/public</Set>
<Set name="contextPath"><Property name="nexus-context-path"/></Set>
<Set name="throwUnavailableOnStartupException">true</Set>
<Set name="configurationClasses">
<Array type="java.lang.String">
<Item>org.eclipse.jetty.webapp.WebXmlConfiguration</Item>
</Array>
</Set>
</New>
</Arg>
</New>
<Set name="handler">
<New id="Handlers" class="org.eclipse.jetty.server.handler.HandlerCollection">
<Set name="handlers">
<Array type="org.eclipse.jetty.server.Handler">
<Item>
<Ref refid="NexusHandler"/>
</Item>
</Array>
</Set>
</New>
</Set>
<Set name="stopAtShutdown">true</Set>
<Set name="stopTimeout"><Property name="jetty.stop.timeout" default="80000"/></Set>
<Set name="dumpAfterStart"><Property name="jetty.dump.start" default="false"/></Set>
<Set name="dumpBeforeStop"><Property name="jetty.dump.stop" default="false"/></Set>
<New id="httpConfig" class="org.eclipse.jetty.server.HttpConfiguration">
<Set name="outputBufferSize"><Property name="jetty.output.buffer.size" default="32768"/></Set>
<Set name="outputAggregationSize"><Property name="jetty.output.aggregation.size" default="8192"/></Set>
<Set name="requestHeaderSize"><Property name="jetty.request.header.size" default="8192"/></Set>
<Set name="responseHeaderSize"><Property name="jetty.response.header.size" default="8192"/></Set>
<Set name="sendServerVersion"><Property name="jetty.send.server.version" default="true"/></Set>
<Set name="sendDateHeader"><Property name="jetty.send.date.header" default="true"/></Set>
<Set name="headerCacheSize">512</Set>
<Set name="delayDispatchUntilContent"><Property name="jetty.delayDispatchUntilContent" default="false"/></Set>
<Call name="addCustomizer">
<Arg><New class="org.eclipse.jetty.server.ForwardedRequestCustomizer"/></Arg>
</Call>
</New>
</Configure>
jetty-http.xml:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<Call name="addConnector">
<Arg>
<New id="httpConnector" class="org.eclipse.jetty.server.ServerConnector">
<Arg name="server"><Ref refid="Server"/></Arg>
<Arg name="acceptors" type="int"><Property name="jetty.http.acceptors" default="-1"/></Arg>
<Arg name="selectors" type="int"><Property name="jetty.http.selectors" default="-1"/></Arg>
<Arg name="factories">
<Array type="org.eclipse.jetty.server.ConnectionFactory">
<!-- uncomment to support proxy protocol
<Item>
<New class="org.eclipse.jetty.server.ProxyConnectionFactory"/>
</Item>-->
<Item>
<New class="org.sonatype.nexus.bootstrap.jetty.InstrumentedConnectionFactory">
<Arg>
<New class="org.eclipse.jetty.server.HttpConnectionFactory">
<Arg name="config">
<Ref refid="httpConfig"/>
</Arg>
</New>
</Arg>
</New>
</Item>
</Array>
</Arg>
<Set name="host"><Property name="application-host" /></Set>
<Set name="port"><Property name="application-port"/></Set>
<Set name="idleTimeout"><Property name="jetty.http.timeout" default="80000"/></Set>
<Set name="soLingerTime"><Property name="jetty.http.soLingerTime" default="-1"/></Set>
<Set name="acceptorPriorityDelta"><Property name="jetty.http.acceptorPriorityDelta" default="0"/></Set>
<Set name="acceptQueueSize"><Property name="jetty.http.acceptQueueSize" default="0"/></Set>
</New>
</Arg>
</Call>
</Configure>
jetty-https.xml:
<?xml version="1.0"?>
<!DOCTYPE Configure PUBLIC "-//Jetty//Configure//EN" "http://www.eclipse.org/jetty/configure_9_0.dtd">
<Configure id="Server" class="org.eclipse.jetty.server.Server">
<!--
==== HTTPS ====
Set the following inside nexus.properties:
application-port-ssl: the port to listen for https connections
-->
<Ref refid="httpConfig">
<Set name="secureScheme">https</Set>
<Set name="securePort"><Property name="application-port-ssl" /></Set>
</Ref>
<New id="httpsConfig" class="org.eclipse.jetty.server.HttpConfiguration">
<Arg><Ref refid="httpConfig"/></Arg>
<Call name="addCustomizer">
<Arg>
<New id="secureRequestCustomizer" class="org.eclipse.jetty.server.SecureRequestCustomizer">
<!-- 7776000 seconds = 90 days -->
<Set name="stsMaxAge"><Property name="jetty.https.stsMaxAge" default="7776000"/></Set>
<Set name="stsIncludeSubDomains"><Property name="jetty.https.stsIncludeSubDomains" default="false"/></Set>
<Set name="sniHostCheck"><Property name="jetty.https.sniHostCheck" default="false"/></Set>
</New>
</Arg>
</Call>
</New>
<New id="sslContextFactory" class="org.eclipse.jetty.util.ssl.SslContextFactory$Server">
<Set name="KeyStorePath"><Property name="ssl.etc"/>/keystore.jks</Set>
<Set name="KeyStorePassword">password</Set>
<Set name="KeyManagerPassword">password</Set>
<Set name="TrustStorePath"><Property name="ssl.etc"/>/keystore.jks</Set>
<Set name="TrustStorePassword">password</Set>
<Set name="EndpointIdentificationAlgorithm"></Set>
<Set name="NeedClientAuth"><Property name="jetty.ssl.needClientAuth" default="false"/></Set>
<Set name="WantClientAuth"><Property name="jetty.ssl.wantClientAuth" default="false"/></Set>
<Set name="ExcludeCipherSuites">
<Array type="String">
<Item>SSL_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_RSA_WITH_DES_CBC_SHA</Item>
<Item>SSL_DHE_DSS_WITH_DES_CBC_SHA</Item>
<Item>SSL_RSA_EXPORT_WITH_RC4_40_MD5</Item>
<Item>SSL_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA</Item>
<Item>SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA</Item>
</Array>
</Set>
</New>
<Call name="addConnector">
<Arg>
<New id="httpsConnector" class="org.eclipse.jetty.server.ServerConnector">
<Arg name="server"><Ref refid="Server" /></Arg>
<Arg name="acceptors" type="int"><Property name="jetty.https.acceptors" default="-1"/></Arg>
<Arg name="selectors" type="int"><Property name="jetty.https.selectors" default="-1"/></Arg>
<Arg name="factories">
<Array type="org.eclipse.jetty.server.ConnectionFactory">
<Item>
<New class="org.sonatype.nexus.bootstrap.jetty.InstrumentedConnectionFactory">
<Arg>
<New class="org.eclipse.jetty.server.SslConnectionFactory">
<Arg name="next">http/1.1</Arg>
<Arg name="sslContextFactory"><Ref refid="sslContextFactory"/></Arg>
</New>
</Arg>
</New>
</Item>
<Item>
<New class="org.eclipse.jetty.server.HttpConnectionFactory">
<Arg name="config"><Ref refid="httpsConfig" /></Arg>
</New>
</Item>
</Array>
</Arg>
<Set name="host"><Property name="application-host" /></Set>
<Set name="port"><Property name="application-port-ssl" /></Set>
<Set name="idleTimeout"><Property name="jetty.https.timeout" default="80000"/></Set>
<Set name="soLingerTime"><Property name="jetty.https.soLingerTime" default="-1"/></Set>
<Set name="acceptorPriorityDelta"><Property name="jetty.https.acceptorPriorityDelta" default="0"/></Set>
<Set name="acceptQueueSize"><Property name="jetty.https.acceptQueueSize" default="0"/></Set>
</New>
</Arg>
</Call>
</Configure>