I am currently running nexus 3.35.0 in docker. The nexus-data is in a CIFS Samba folder, bind-mounted via Docker.
Last weekend, I stopped and started it. Nexus still works, I can pull and push packages, docker images, etc. I can no longer use the search bar in the UI to search for a particular package. There is an error “all shards failed”.
The logs indicate that some indices are pending delete. I thoguht that maybe I could let Nexus start with a fresh instance of Elasticsearch, so I stopped it, waited for a minute, renamed the elasticsearch folder to a backup, then re-started nexus.
A fresh elasticsearch folder is created, but I am still getting the same errors, that the same indices are pending delete, and these never go away.
How can I force Nexus to use a fresh instance of elasticsearch? I have attached a copy of my logs indicating the pending index deletes as well as the “all shards failed” when I do a search on the UI.
2022-06-13 06:36:53,604+0000 WARN [elasticsearch[EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65][generic][T#5]] *SYSTEM org.elasticsearch.indices - [EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65] [9e14d4f5d17bbef22424601413d6c650bc2ba074] still pending deletes present for shards [[9e14d4f5d17bbef22424601413d6c650bc2ba074], [9e14d4f5d17bbef22424601413d6c650bc2ba074][0]] - retrying
2022-06-13 06:36:54,001+0000 WARN [Timer-0] *SYSTEM java.util.prefs - Could not drop file-lock on user preferences. Unix error code 12.
2022-06-13 06:36:55,615+0000 WARN [elasticsearch[EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65][generic][T#4]] *SYSTEM org.elasticsearch.indices - [EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65] [cd057c7addc9ab247d49640f034d8f63c9df6604] still pending deletes present for shards [[cd057c7addc9ab247d49640f034d8f63c9df6604], [cd057c7addc9ab247d49640f034d8f63c9df6604][0]] - retrying
2022-06-13 06:36:58,526+0000 WARN [elasticsearch[EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65][generic][T#6]] *SYSTEM org.elasticsearch.indices - [EACBFA0F-69E9309A-8C8BD550-B800D66E-F490CC65] [5541ebfd0a805671d170dd843afd2ffd71a03e16] still pending deletes present for shards [[5541ebfd0a805671d170dd843afd2ffd71a03e16], [5541ebfd0a805671d170dd843afd2ffd71a03e16][0]] - retrying
2022-06-13 06:37:27,804+0000 ERROR [qtp1751689652-115] admin org.sonatype.nexus.extdirect.internal.ExtDirectExceptionHandler - Failed to invoke action method: coreui_Search.read, java-method: org.sonatype.nexus.coreui.SearchComponent.read
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onFirstPhaseResult(AbstractSearchAsyncAction.java:206)
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:129)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:115)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:47)
at org.elasticsearch.action.support.TransportAction.doExecute(TransportAction.java:149)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:137)
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:85)
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:58)
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:86)
at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:56)
at org.sonatype.nexus.repository.search.query.SearchQueryServiceImpl.executeSearch(SearchQueryServiceImpl.java:211)
at org.sonatype.nexus.repository.search.query.SearchQueryServiceImpl.search(SearchQueryServiceImpl.java:157)
at org.sonatype.nexus.repository.search.query.SearchQueryService$search.call(Unknown Source)
at org.sonatype.nexus.coreui.SearchComponent.read(SearchComponent.groovy:107)
at com.palominolabs.metrics.guice.ExceptionMeteredInterceptor.invoke(ExceptionMeteredInterceptor.java:23)
at com.palominolabs.metrics.guice.TimedInterceptor.invoke(TimedInterceptor.java:26)
at org.apache.shiro.guice.aop.AopAllianceMethodInvocationAdapter.proceed(AopAllianceMethodInvocationAdapter.java:49)
at org.apache.shiro.authz.aop.AuthorizingAnnotationMethodInterceptor.invoke(AuthorizingAnnotationMethodInterceptor.java:68)
at org.apache.shiro.guice.aop.AopAllianceMethodInterceptorAdapter.invoke(AopAllianceMethodInterceptorAdapter.java:36)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeJavaMethod(DispatcherBase.java:142)
at com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.invokeMethod(DispatcherBase.java:133)
at org.sonatype.nexus.extdirect.internal.ExtDirectDispatcher.invokeMethod(ExtDirectDispatcher.java:82)
at com.softwarementors.extjs.djn.router.dispatcher.DispatcherBase.dispatch(DispatcherBase.java:63)
at com.softwarementors.extjs.djn.router.processor.standard.StandardRequestProcessorBase.dispatchStandardMethod(StandardRequestProcessorBase.java:73)
at com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequest(JsonRequestProcessor.java:502)
at com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.processIndividualRequestsInThisThread(JsonRequestProcessor.java:150)
at com.softwarementors.extjs.djn.router.processor.standard.json.JsonRequestProcessor.process(JsonRequestProcessor.java:133)
at com.softwarementors.extjs.djn.router.RequestRouter.processJsonRequest(RequestRouter.java:83)
at com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.processRequest(DirectJNgineServlet.java:632)
at com.softwarementors.extjs.djn.servlet.DirectJNgineServlet.doPost(DirectJNgineServlet.java:595)
at org.sonatype.nexus.extdirect.internal.ExtDirectServlet.doPost(ExtDirectServlet.java:137)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
at com.google.inject.servlet.DynamicServletPipeline.service(DynamicServletPipeline.java:71)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:450)
at org.sonatype.nexus.security.SecurityFilter.executeChain(SecurityFilter.java:96)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.sonatype.nexus.security.SecurityFilter.doFilterInternal(SecurityFilter.java:112)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.sonatype.nexus.licensing.internal.LicensingRedirectFilter.doFilter(LicensingRedirectFilter.java:116)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.codahale.metrics.servlet.AbstractInstrumentedFilter.doFilter(AbstractInstrumentedFilter.java:112)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.ErrorPageFilter.doFilter(ErrorPageFilter.java:79)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.EnvironmentFilter.doFilter(EnvironmentFilter.java:101)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at org.sonatype.nexus.internal.web.HeaderPatternFilter.doFilter(HeaderPatternFilter.java:98)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.DynamicFilterPipeline.dispatch(DynamicFilterPipeline.java:104)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
at org.sonatype.nexus.bootstrap.osgi.DelegatingFilter.doFilter(DelegatingFilter.java:73)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at com.codahale.metrics.jetty9.InstrumentedHandler.handle(InstrumentedHandler.java:239)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.lang.Thread.run(Thread.java:748)