Hello. I appreciate developers in sonatype and community for a good stateful registry service Nexus!
I hope to get some advice to solving my problem. I tried to run a Kaniko image which is building docker images for building and pushing CUDA-based images into Nexus registry. My Nexus registry is currently deployed on Docker service of Synology NAS (RS1221RP+) with 4 Core CPU and 32 GB RAM.
To achieve the goal, I checked some pre-tasks:
- [PASS] Build a target small-size image in a local docker environment
- [PASS] Push the target small-size image into Nexus directly
- [PASS] Build a target CUDA-based image in a local docker environment
- [PASS] Push the target CUDA-based image into Nexus directly
- [PASS] Run kaniko to build and push a small-size image (< 100 MB)
- [FAIL] Run kaniko to build and push a CUDA-based image (> 30 GB)
The linux computer which is running the kaniko image has enough spec to finish the job. Unfortunately, the size of the image cannot be reduced to less than 30 GB. I expected the kaniko container can push even large size images such as CUDA images like the local Docker can do that. But I failed and got these two logs:
Log of Kaniko
# INFO[....] above context worked good
INFO[8011] Pushing image to registry.example.com:5000/project/heavy-module:latest
error pushing image: failed to push to destination registry.example.com:5000/project/heavy-module:latest: PATCH http://registry.example.com:5000/v2/project/heavy-module/blobs/uploads/10407b0c-45b9-4c93-bdef-926ec8fcec59: UNKNOWN: unknown
For many others, there was a meaningful error instead of ‘UNKNOWN: unknown’. But I got this hard-to-debug error. However, it is said that there was an error in the ‘PATCH’ process, so I guess it is an error in the network. But I still don’t know what the problem is.
Log of Nexus
2023-09-12 06:30:10,836+0000 WARN [qtp717316089-10132] admin org.sonatype.nexus.repository.docker.internal.V2Handlers - Error: PATCH /v2/project/heavy-module/blobs/uploads/a939c548-06c6-4287-b67c-ea30812a6526
... 188 common frames omitted
at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:404)
at org.sonatype.nexus.blobstore.file.FileBlobStore.lambda$3(FileBlobStore.java:348)
at org.sonatype.nexus.blobstore.file.internal.SimpleFileOperations.create(SimpleFileOperations.java:61)
at com.google.common.io.ByteStreams.copy(ByteStreams.java:109)
at java.io.FilterInputStream.read(FilterInputStream.java:107)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at java.security.DigestInputStream.read(DigestInputStream.java:161)
at com.google.common.io.CountingInputStream.read(CountingInputStream.java:63)
at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:333)
at org.eclipse.jetty.server.HttpInput$3.noContent(HttpInput.java:1195)
at org.eclipse.jetty.server.HttpInput$3.getError(HttpInput.java:1207)
Caused by: org.eclipse.jetty.io.EofException: Early EOF
at java.lang.Thread.run(Thread.java:750)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.repository.httpbridge.internal.ExhaustRequestFilter.doFilter(ExhaustRequestFilter.java:80)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373)
at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:370)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373)
at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.service(ViewServlet.java:126)
at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.doService(ViewServlet.java:174)
at org.sonatype.nexus.repository.httpbridge.internal.ViewServlet.dispatchAndSend(ViewServlet.java:212)
at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:43)
at org.sonatype.nexus.repository.view.ConfigurableViewFacet.dispatch(ConfigurableViewFacet.java:52)
at org.sonatype.nexus.repository.view.Router.dispatch(Router.java:65)
at org.sonatype.nexus.repository.view.Context.start(Context.java:179)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at org.sonatype.nexus.repository.view.handlers.TimingHandler.handle(TimingHandler.java:58)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at com.sonatype.analytics.internal.handler.AnalyticsMeteringHandler.handle(AnalyticsMeteringHandler.java:69)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at com.sun.proxy.$Proxy204.handle(Unknown Source)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
at groovy.lang.Closure.call(Closure.java:420)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at sun.reflect.GeneratedMethodAccessor139.invoke(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure1.doCall(V2Handlers.groovy:93)
at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at com.sun.proxy.$Proxy204.handle(Unknown Source)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
at groovy.lang.Closure.call(Closure.java:420)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at sun.reflect.GeneratedMethodAccessor132.invoke(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure18.doCall(V2Handlers.groovy:305)
at org.sonatype.nexus.repository.view.Context$proceed.call(Unknown Source)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at org.sonatype.nexus.repository.security.SecurityHandler.handle(SecurityHandler.java:51)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at org.sonatype.nexus.repository.view.handlers.HandlerContributor.handle(HandlerContributor.java:67)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at com.sonatype.nexus.clm.internal.orient.FirewallContributedHandler.handle(FirewallContributedHandler.java:105)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at org.sonatype.nexus.repository.storage.UnitOfWorkHandler.handle(UnitOfWorkHandler.java:39)
at org.sonatype.nexus.repository.view.Context.proceed(Context.java:88)
at com.sun.proxy.$Proxy204.handle(Unknown Source)
at org.codehaus.groovy.runtime.ConversionHandler.invoke(ConversionHandler.java:124)
at org.codehaus.groovy.runtime.ConvertedClosure.invokeCustom(ConvertedClosure.java:54)
at groovy.lang.Closure.call(Closure.java:420)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1034)
at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:264)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:98)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at sun.reflect.GeneratedMethodAccessor229.invoke(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.V2Handlers$_closure4.doCall(V2Handlers.groovy:140)
at org.sonatype.nexus.repository.docker.internal.DockerHostedFacet$uploadBlobRange$7.call(Unknown Source)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerHostedFacetSupport.uploadBlobRange(DockerHostedFacetSupport.java:100)
at org.sonatype.nexus.repository.docker.internal.UploadManagerImpl.add(UploadManagerImpl.java:97)
at org.sonatype.nexus.repository.docker.internal.Upload.addBlob(Upload.java:125)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:109)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.blobstore.BlobStoreSupport.create(BlobStoreSupport.java:125)
at org.sonatype.nexus.blobstore.metrics.BlobStoreAnalyticsInterceptor.invoke(BlobStoreAnalyticsInterceptor.java:60)
at org.sonatype.nexus.blobstore.file.FileBlobStore.doCreate(FileBlobStore.java:348)
at org.sonatype.nexus.blobstore.file.FileBlobStore.create(FileBlobStore.java:368)
at org.sonatype.nexus.blobstore.file.FileBlobStore.tryCreate(FileBlobStore.java:434)
"org.sonatype.nexus.blobstore.api.BlobStoreException: BlobId: path$nexus-repository-docker/5eb9a3c7-d8d0-40ca-b3ea-579301ec8da9/399ac529-3fb3-411c-8357-ba0208736863, org.eclipse.jetty.io.EofException: Early EOF, Cause: Early EOF
"
After googling, the error ‘Early EOF’ seems to be an error that often appears when handling large files. But I still don’t know if this was Kaniko’s problem or Nexus’ problem. I thought I could fix this problem if I could set up for supporting large images on Nexus. But I couldn’t find any settings related to my issue in the Nexus Repository Manager. If anyone knows anything about this matter, please give me a little advice.
Thank you in advance 