After using sonatype nexus 3 for a few hours, I see error, "Unable to acquire connection from main pool, trying overflow"

Sonatype Nexus Repository
1

Hi,

After using sonatype nexus 3 for a few hours, I see the following error:

2019-11-01 00:16:00,088-0500 DEBUG [qtp2130725876-2531] admin org.sonatype.nexus.orient.DatabasePoolWithOverflowImpl - Unable to acquire connection from main pool, trying overflow
java.lang.IllegalStateException: You have reached maximum pool size for given partition
at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(OPartitionedDatabasePool.java:244)
at org.sonatype.nexus.orient.DatabasePoolWithOverflowImpl.acquire(DatabasePoolWithOverflowImpl.java:64)
at org.sonatype.nexus.orient.DatabaseInstanceImpl.acquire(DatabaseInstanceImpl.java:70)
at org.sonatype.nexus.repository.storage.StorageFacetImpl.lambda$0(StorageFacetImpl.java:156)
at org.sonatype.nexus.repository.browse.internal.BrowseServiceImpl.browseComponentAssets(BrowseServiceImpl.java:164)
at org.sonatype.nexus.repository.rest.internal.resources.SearchResource.toComponent(SearchResource.java:146)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.sonatype.nexus.repository.rest.internal.resources.SearchResource.search(SearchResource.java:133)
at sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:294)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:248)
at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:235)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:402)
at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:209)
at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:227)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
at org.sonatype.nexus.siesta.internal.resteasy.ComponentContainerImpl.service(ComponentContainerImpl.java:106)
at org.sonatype.nexus.siesta.SiestaServlet.service(SiestaServlet.java:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181)
at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:85)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:101)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:108)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:97)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:68)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135)
at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:175)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:531)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:745)

How can we fix this error ?

Thanks,
Karthick

2

Do you have any custom groovy scripts? If so, make sure you are closing transactions:

3

Hi @rseddon,

Thanks for the response. Yes. We use custom scripts to call nexus.

Is there a way to let the nexus server close the connections or set the number of open connections to unlimited or set the ‘alive time’ ?

Regards,
Karthick

4

Your scripts are leaking database transactions. They need to be fixed so they don’t do that. See here:

5

Hello Nexus Community,
It’s the first time that I’m publishing something here. I’ve found this topic and subject is very close to mine so I’m replying to it.

After restoring OrientDB component database, I’m facing to similar issue that this post :

2024-03-22 10:33:07,885+0100 DEBUG [event-14-thread-3357]  anonymous org.sonatype.nexus.orient.DatabasePoolWithOverflowImpl - Unable to acquire connection from main pool, trying overflow
java.lang.IllegalStateException: You have reached maximum pool size for given partition
        at com.orientechnologies.orient.core.db.OPartitionedDatabasePool.acquire(OPartitionedDatabasePool.java:244)
        at org.sonatype.nexus.orient.DatabasePoolWithOverflowImpl.acquire(DatabasePoolWithOverflowImpl.java:64)
        at org.sonatype.nexus.orient.DatabaseInstanceImpl.acquire(DatabaseInstanceImpl.java:55)
        at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$0(OrientOperations.java:48)
        at org.sonatype.nexus.transaction.UnitOfWork.doOpenSession(UnitOfWork.java:247)
        at org.sonatype.nexus.transaction.UnitOfWork.openSession(UnitOfWork.java:222)
        at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:222)

After analysis, issue happened when Maven uploads many artefacts consecutively, after one moment, uploading failed. For small Maven projects, uploading artefact is OK.

Do you have some tips to analyse deeper please? I don’t have any Groovy scripts (like in the previous topic answers)

For information, on the database restoring, I had problem to create index : browse_node_asset_id_idx

`- Index 'browse_node_asset_id_idx'...$ANSI{green {db=component}} Exception during index 'browse_node_asset_id_idx' creation
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #57:15037: found duplicated key '#54:12731' in index 'browse_node_asset_id_idx' previously assigned to the record #63:21932
        DB name="component" INDEX=browse_node_asset_id_idx RID=#63:21932
        at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:47)
        at com.orientechnologies.orient.core.index.OIndexUnique$1.validate(OIndexUnique.java:37)
        at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.put(OSBTree.java:855)
        at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.validatedPut(OSBTree.java:261)
        at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.validatedPut(OSBTreeIndexEngine.java:169)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doValidatedPutIndexValue(OAbstractPaginatedStorage.java:2519)
        at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.validatedPutIndexValue(OAbstractPaginatedStorage.java:2498)
        at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:82)
        at com.orientechnologies.orient.core.index.OIndexUnique.put(OIndexUnique.java:35)
        at com.orientechnologies.orient.core.index.OIndexAbstract.populateIndex(OIndexAbstract.java:940)
        at com.orientechnologies.orient.core.index.OIndexAbstract.indexCluster(OIndexAbstract.java:1040)
        at com.orientechnologies.orient.core.index.OIndexAbstract.fillIndex(OIndexAbstract.java:539)
        at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:250)
        at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:129)
        at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:40)
        at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:171)
        at com.orientechnologies.orient.core.index.OIndexManagerProxy.createIndex(OIndexManagerProxy.java:87)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importIndexes(ODatabaseImport.java:1307)
        at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:186)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2214)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
        at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
        at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
        at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
        at org.sonatype.nexus.orient.console.Main.main(Main.java:63)
`

I’ve rebuilded indexes on OrientDB and added manually some missing indexes but I’m not able to add browse_node_asset_id_idx index, I’m still having error on duplicated key

Do you think, it can be have a link to my connection pool issue ? Maybe the uploading takes to much time due to missing index and next upload requests are waiting and at one moment they failed due to max connections limit

Thank you

6

Hello,
I’ve solved my issue by removing duplicated keys into the browse_node table by using this query to find them :

select @RID,repository_name,path,asset_id from browse_node where asset_id in (select asset_id from (select asset_id, count(*) as counter from browse_node where asset_id IS NOT NULL group by asset_id) where counter > 1) limit 50
  • set asset_id to null for useless/unused artefacts using a UPDATE command

And after, creating browse_node_asset_id_idx index using this query to ignore null entries :

create index browse_node_asset_id_idx on browse_node (asset_id collate default) UNIQUE METADATA { ignoreNullValues : true }

Now Nexus seems back to normal