Image push after Nexus upgrade not possible

mohanisch · October 18, 2021, 1:57pm

Hello,

last week we upgraded Nexus OSS from 3.30 to 3.35. It is running in docker.
Since this upgrade was successfully done, we encountered issue while pushing images into docker-registry. We are using S3 as blobstore.

I can’t say exactly what the problem is, but I can reproduce it by pushing a new image, for example Debian:

$ docker tag debian docker-registry.example.com/orange/debian:latest
$ docker tag debian docker-registry.example.com/orange/debian:11.1
$ docker push -a docker-registry.example.com/orange/debian
$ docker push -a docker-registry.example.com/orange/debian
The push refers to repository [docker-registry.example.com/orange/debian]
62a747bf1719: Pushed 
11.1: digest: sha256:826d8850098b954c3d67c2e45ea7a08fa681c128126ba01b35bd2689b1bb2e04 size: 529
62a747bf1719: Layer already exists 
received unexpected HTTP status: 500 Server Error

This results in an warning in Nexus log, user get a http 500:

2021-10-18 13:53:14,200+0000 WARN  [qtp1716215290-1739] myUserName org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT /nexus/repository/docker-main/v2/orange/debian/manifests/latest
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 2Q09QDU82QE4JS8K; S3 Extended Request ID: Sfv5XqvxWKQiJ5CT9HUsJasdfOsdjnfVO2pcxIWh4l5suqXZPm/NkHd0RcNr3em20soHr6+IJQ=; Proxy: null)

What we already did:

Rebuilt of index for docker-main
Ran task “Repair - Reconcile component database from blob store” for docker-main

What is the problem and how can we solve it?

mpiggott · October 18, 2021, 2:11pm

Is there a stack trace?

mohanisch · October 18, 2021, 2:21pm

I will try to collect a trace but currently I haven’t one.

mohanisch · October 18, 2021, 3:00pm

The repair job is still running and I don’t want to do many changes in that time.

Where can we additionally take a look into?

sebastian.bayerl · October 19, 2021, 7:44am

Hello,

I think I encountered the same or a similar issue. Setup is very similar to Marco’s and I updated from 3.32 to 3.35. Here my stacktraces, hope that helps.

72109 [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:2.6.0:build (default-cli) on project myProject: 500 Server Error
72109 [ERROR] {"errors":[{"code":"UNKNOWN","message":"unknown","detail":null}]}

2021-10-18 18:22:08,553+0000 WARN  [qtp811299988-6237]  user org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PUT /v2/myProject/manifests/tag
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 1XY40PR9FGF27K4D; S3 Extended Request ID: j5LpM/Id1pU+YkQ878Rw/fJgkbNGcLUHnVdO0+duPne/fj1o5ZhjAJ2/bI/xqeCR/6R/1izj6NM=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:164)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:171)
	at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:679)
	at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
	at org.sonatype.nexus.repository.view.payloads.TempBlob.get(TempBlob.java:110)
	at org.sonatype.nexus.common.io.InputStreamSupplier$get.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl.manifestDigest(V2ManifestUtilImpl.groovy:214)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerHostedFacetImpl.doPutManifestByTag(DockerHostedFacetImpl.java:736)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57)
	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerHostedFacetImpl.putManifestByTag(DockerHostedFacetImpl.java:722)
	at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$putManifestByTag$6.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure8.doCall(V2Handlers.groovy:197)
	at sun.reflect.GeneratedMethodAccessor298.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy192.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:46)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at com.sonatype.nexus.clm.internal.orient.FirewallContributedHandler.handle(FirewallContributedHandler.java:104)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
	at sun.reflect.GeneratedMethodAccessor173.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy192.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure16.doCall(V2Handlers.groovy:280)
	at sun.reflect.GeneratedMethodAccessor175.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy192.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:93)
	at sun.reflect.GeneratedMethodAccessor174.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy192.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at com.sonatype.analytics.internal.handler.AnalyticsMeteringHandler.handle(AnalyticsMeteringHandler.java:69)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context.start(Context.java:179)
	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:213)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)

mohanisch · October 19, 2021, 7:58am

Here now my stack treace:

2021-10-19 07:54:04,990+0000 WARN  [qtp1716215290-3114] myUsername org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: PUT /nexus/repository/docker-main/v2/orange/debian/manifests/latest
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: R6VSQ53F82Q77FVY; S3 Extended Request ID: WHBz4X+lpWBUJ33oMG6MIQ+BBdHge/rzWLo8NK4kdQO870SGsdqqRd5P0rWFJ1Gv6DnjYi2TYE0=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:164)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:171)
	at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:679)
	at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
	at org.sonatype.nexus.repository.view.payloads.BlobPayload.openInputStream(BlobPayload.java:44)
	at org.sonatype.nexus.repository.view.Content.openInputStream(Content.java:95)
	at org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender.send(DefaultHttpResponseSender.java:78)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:229)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)

2021-10-19 07:54:04,990+0000 DEBUG [qtp1716215290-3114] myUsername org.apache.shiro.web.servlet.AdviceFilter - Filter execution resulted in an unexpected Exception (not IOException or ServletException as the Filter API recommends).  Wrapping in ServletException and propagating.
2021-10-19 07:54:04,991+0000 DEBUG [qtp1716215290-3114] *SYSTEM org.sonatype.nexus.internal.web.ErrorPageServlet - Attaching cause
javax.servlet.ServletException: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: R6VSQ53F82Q77FVY; S3 Extended Request ID: WHBz4X+lpWBUJ33oMG6MIQ+BBdHge/rzWLo8NK4kdQO870SGsdqqRd5P0rWFJ1Gv6DnjYi2TYE0=; Proxy: null), S3 Extended Request ID: WHBz4X+lpWBUJ33oMG6MIQ+BBdHge/rzWLo8NK4kdQO870SGsdqqRd5P0rWFJ1Gv6DnjYi2TYE0=
	at org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: R6VSQ53F82Q77FVY; S3 Extended Request ID: WHBz4X+lpWBUJ33oMG6MIQ+BBdHge/rzWLo8NK4kdQO870SGsdqqRd5P0rWFJ1Gv6DnjYi2TYE0=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:164)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:171)
	at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:679)
	at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
	at org.sonatype.nexus.repository.view.payloads.BlobPayload.openInputStream(BlobPayload.java:44)
	at org.sonatype.nexus.repository.view.Content.openInputStream(Content.java:95)
	at org.sonatype.nexus.repository.httpbridge.internal.DefaultHttpResponseSender.send(DefaultHttpResponseSender.java:78)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:229)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	... 39 common frames omitted

mohanisch · October 20, 2021, 4:08am

@mpiggott , any news from your side?

mpiggott · October 20, 2021, 5:20pm

We think this is a regression with a change we made with our s3 blobstore to improve performance that is impacted by a quirk in the Docker implementation. We’re about to do a 3.36 release (probably the binaries will be available at the start of, or early next week) that will either revert it or include a fix if its ready.

mohanisch · October 21, 2021, 7:58am

Hi @mpiggott , thanks for your answer.
Nice to hear that there will be a possible fix. I hope it works and is released very soon. This problem is blocking us very much.

mohanisch · October 26, 2021, 5:21am

@mpiggott , is known when the new version will be released?

giovannicandido · November 6, 2021, 5:41pm

I come across the same problem. Pushes to a tag version works, but pushes to the latest version doen’st

Stack trace:

	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)

giovannicandido · November 6, 2021, 6:17pm

Version 3.36 solves the problem

alexey_ban · January 5, 2022, 10:10am

Same problem with pull image from proxy docker repository (first pull ok) on s3 bucket. Version - 3.37.3-02

2022-01-05 06:16:20,010+0000 WARN  [qtp564277201-618]  user org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: GET /v2/docker/server/publisher/manifests/1.0.29
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 11BV6A3YQ9ZQNGB3; S3 Extended Request ID: EEU8rLfztnft85Q8hnDdU4lel9JDZGSp/ZATCSCS4iZNKOTJHJhm92iKedh+cjL1CVQZ6H99hak=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:164)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:171)
	at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:681)
	at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
	at org.sonatype.nexus.repository.view.payloads.TempBlob.get(TempBlob.java:110)
	at org.sonatype.nexus.common.io.InputStreamSupplier$get.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl.readManifest(V2ManifestUtilImpl.groovy:107)
	at org.sonatype.nexus.repository.docker.internal.DockerContentValidator.v2DetermineContentType(DockerContentValidator.java:168)
	at org.sonatype.nexus.repository.docker.internal.DockerContentValidator.determineContentType(DockerContentValidator.java:97)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.determineContentType(StorageTxImpl.java:1071)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.buildStorageHeaders(StorageTxImpl.java:779)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.createBlob(StorageTxImpl.java:739)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.setBlob(StorageTxImpl.java:1036)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
	at com.sun.proxy.$Proxy225.setBlob(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerFacetDatabaseUtils.saveAsset(DockerFacetDatabaseUtils.java:347)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.doPutManifestByTag(DockerProxyFacetImpl.java:468)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.putManifestByTag(DockerProxyFacetImpl.java:449)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.putManifest(DockerProxyFacetImpl.java:432)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.storeByAssetKind(DockerProxyFacetImpl.java:172)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.store(DockerProxyFacetImpl.java:141)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.doGet(ProxyFacetSupport.java:286)
	at org.sonatype.nexus.repository.docker.internal.DockerProxyFacetSupport.doGet(DockerProxyFacetSupport.java:247)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.lambda$1(ProxyFacetSupport.java:260)
	at org.sonatype.nexus.common.io.CooperatingFuture.performCall(CooperatingFuture.java:122)
	at org.sonatype.nexus.common.io.CooperatingFuture.call(CooperatingFuture.java:64)
	at org.sonatype.nexus.common.io.ScopedCooperationFactorySupport$ScopedCooperation.cooperate(ScopedCooperationFactorySupport.java:99)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:251)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:240)
	at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:52)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.storage.LastDownloadedHandler.handle(LastDownloadedHandler.java:59)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure16.doCall(V2Handlers.groovy:280)
	at sun.reflect.GeneratedMethodAccessor517.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy202.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:46)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
	at sun.reflect.GeneratedMethodAccessor506.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy202.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:56)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.routing.internal.RoutingRuleHandler.handle(RoutingRuleHandler.java:52)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at com.sonatype.nexus.clm.internal.orient.FirewallContributedHandler.handle(FirewallContributedHandler.java:104)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.docker.internal.DockerNotFoundHandler.handle(DockerNotFoundHandler.java:42)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
	at sun.reflect.GeneratedMethodAccessor506.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy202.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:93)
	at sun.reflect.GeneratedMethodAccessor509.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy202.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure17.doCall(V2Handlers.groovy:296)
	at sun.reflect.GeneratedMethodAccessor515.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1099)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy202.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure20.doCall(V2Handlers.groovy:497)
	at sun.reflect.GeneratedMethodAccessor514.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy202.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at com.sonatype.analytics.internal.handler.AnalyticsMeteringHandler.handle(AnalyticsMeteringHandler.java:69)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context.start(Context.java:179)
	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:213)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)

eugenyuk · January 18, 2022, 8:53am

The same pull image issue from proxy docker on s3 bucket. Nexus version: 3.37.3-02

2022-01-18 08:49:09,137+0000 WARN  [qtp1126369804-11771]  k8s-sa-test org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: GET /v2/prometheus-operator/prometheus-operator/manifests/v0.52.1
com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error Code: NoSuchKey; Request ID: 2KSAMVKSQ0C24PG1; S3 Extended Request ID: tt63W1yzhEjyKAt1lNb0WdAtXHNWM1eE1nQzbFY6nnaRpeMLYhc6ns9sHFpVF2iACFKJujefm2M=; Proxy: null)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1811)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1395)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1371)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5062)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5008)
	at com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1490)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:164)
	at com.amazonaws.services.s3.EncryptingAmazonS3Client.getObject(EncryptingAmazonS3Client.java:171)
	at org.sonatype.nexus.blobstore.s3.internal.S3BlobStore$S3Blob.doGetInputStream(S3BlobStore.java:681)
	at org.sonatype.nexus.blobstore.BlobSupport.getInputStream(BlobSupport.java:87)
	at org.sonatype.nexus.repository.view.payloads.TempBlob.get(TempBlob.java:110)
	at org.sonatype.nexus.common.io.InputStreamSupplier$get.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.orient.V2ManifestUtilImpl.readManifest(V2ManifestUtilImpl.groovy:107)
	at org.sonatype.nexus.repository.docker.internal.DockerContentValidator.v2DetermineContentType(DockerContentValidator.java:168)
	at org.sonatype.nexus.repository.docker.internal.DockerContentValidator.determineContentType(DockerContentValidator.java:97)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.determineContentType(StorageTxImpl.java:1071)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.buildStorageHeaders(StorageTxImpl.java:779)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.createBlob(StorageTxImpl.java:739)
	at org.sonatype.nexus.repository.storage.StorageTxImpl.setBlob(StorageTxImpl.java:1036)
	at sun.reflect.GeneratedMethodAccessor245.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
	at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
	at com.sun.proxy.$Proxy216.setBlob(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerFacetDatabaseUtils.saveAsset(DockerFacetDatabaseUtils.java:347)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.doPutManifestByTag(DockerProxyFacetImpl.java:468)
	at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
	at org.sonatype.nexus.transaction.TransactionInterceptor.proceedWithTransaction(TransactionInterceptor.java:66)
	at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:55)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.putManifestByTag(DockerProxyFacetImpl.java:449)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.putManifest(DockerProxyFacetImpl.java:432)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.storeByAssetKind(DockerProxyFacetImpl.java:172)
	at org.sonatype.nexus.repository.docker.internal.orient.DockerProxyFacetImpl.store(DockerProxyFacetImpl.java:141)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.doGet(ProxyFacetSupport.java:286)
	at org.sonatype.nexus.repository.docker.internal.DockerProxyFacetSupport.doGet(DockerProxyFacetSupport.java:247)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.lambda$1(ProxyFacetSupport.java:260)
	at org.sonatype.nexus.common.io.CooperatingFuture.performCall(CooperatingFuture.java:122)
	at org.sonatype.nexus.common.io.CooperatingFuture.call(CooperatingFuture.java:64)
	at org.sonatype.nexus.common.io.ScopedCooperationFactorySupport$ScopedCooperation.cooperate(ScopedCooperationFactorySupport.java:99)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:251)
	at org.sonatype.nexus.repository.proxy.ProxyFacetSupport.get(ProxyFacetSupport.java:240)
	at org.sonatype.nexus.repository.proxy.ProxyHandler.handle(ProxyHandler.java:52)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.storage.LastDownloadedHandler.handle(LastDownloadedHandler.java:59)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure16.doCall(V2Handlers.groovy:280)
	at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.ContentHeadersHandler.handle(ContentHeadersHandler.java:46)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
	at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.cache.NegativeCacheHandler.handle(NegativeCacheHandler.java:56)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.routing.internal.RoutingRuleHandler.handle(RoutingRuleHandler.java:52)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at com.sonatype.nexus.clm.internal.orient.FirewallContributedHandler.handle(FirewallContributedHandler.java:104)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.docker.internal.DockerNotFoundHandler.handle(DockerNotFoundHandler.java:42)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
	at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:93)
	at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure17.doCall(V2Handlers.groovy:296)
	at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1099)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure20.doCall(V2Handlers.groovy:497)
	at sun.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context.start(Context.java:179)
	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
	at org.sonatype.nexus.repository.group.GroupHandler.getFirst(GroupHandler.java:139)
	at org.sonatype.nexus.repository.group.GroupHandler.doGet(GroupHandler.java:116)
	at org.sonatype.nexus.repository.group.GroupHandler.handle(GroupHandler.java:100)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure16.doCall(V2Handlers.groovy:280)
	at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.http.PartialFetchHandler.handle(PartialFetchHandler.java:59)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.ConditionalRequestHandler.handle(ConditionalRequestHandler.java:72)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
	at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:93)
	at sun.reflect.GeneratedMethodAccessor107.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
	at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure17.doCall(V2Handlers.groovy:296)
	at sun.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1099)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
	at groovy.lang.Closure.call(Closure.java:420)
	at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
	at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
	at com.sun.proxy.$Proxy188.handle(Unknown Source)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)
	at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
	at org.sonatype.nexus.repository.view.Context.start(Context.java:179)
	at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
	at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:213)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:175)
	at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:127)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
	at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
	at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
	at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
	at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
	at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
	at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
	at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
	at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
	at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
	at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
	at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
	at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.lang.Thread.run(Thread.java:748)

alexey_ban · February 3, 2022, 9:46am

ltischhauser · February 24, 2022, 3:38pm

We also continue to see this issue and it is beginning to cause problems with more and more artifacts