Hi All
We have a somewhat large/busy nexus3 server running but lately some cleanup tasks have been failing due to heap memory issues on orientdb.
Some details:
nexus-status
version 3.18.1-01
edition OSS
buildRevision 72e5414c4f3fb232e32cbfc9fca740111c94d07f
buildTimestamp 2019-08-06-1749-11540
java.runtime.version 1.8.0_202-b08
Our jvm memory settings are in line with examples here: System Requirements
so there is only a 4g heap with plenty of direct access to memory (39g)
The blobstore in question holds ~ 360000 objects adding up to ~6tb.
considering the error below, should we try bump the ~4g jvm heap?
Error:
2019-11-12 14:55:20,517-0800 WARN [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task f9000db4-0354-4d54-98bd-7af383bf94b3 : 'delete-unused-docker' [repository.docker.gc] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Docker - Delete unused manifests and images'; 1 failure
at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:86)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Suppressed: com.orientechnologies.orient.core.exception.OCommandExecutionException: Cannot execute query "SELECT FROM asset WHERE (attributes.docker.asset_kind = "BLOB" ) AND (bucket = #57:10 )": low heap memory
DB name="component"
at com.orientechnologies.orient.core.sql.OSoftQueryResultList.throwCanExecuteException(OSoftQueryResultList.java:688)
at com.orientechnologies.orient.core.sql.OSoftQueryResultList.checkQueue(OSoftQueryResultList.java:78)
at com.orientechnologies.orient.core.sql.OSoftQueryResultList.size(OSoftQueryResultList.java:88)
at java.util.Collections$SynchronizedCollection.size(Collections.java:2015)
at com.orientechnologies.orient.core.sql.query.OBasicResultSet.size(OBasicResultSet.java:66)
at com.orientechnologies.orient.core.sql.query.OBasicResultSet$1.hasNext(OBasicResultSet.java:86)
at com.google.common.collect.TransformedIterator.hasNext(TransformedIterator.java:42)
at com.google.common.collect.Iterators$5.computeNext(Iterators.java:637)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136)
at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:200)
at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:92)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:54)
at org.sonatype.nexus.repository.docker.internal.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:75)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
at org.sonatype.nexus.repository.docker.tasks.DockerGCTask.execute(DockerGCTask.java:44)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:75)
... 13 common frames omitted
2019-11-12 14:55:20,518-0800 INFO [quartz-3-thread-19] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'delete-unused-docker' [repository.docker.gc] state change RUNNING -> WAITING (FAILED)
Thanks
Andrew