NXRM 3.17.1 bulk API requests return 500

Hello,
when running 50 concurrent requests like

10.0.0.1 - api-user [10/Sep/2019:13:49:25 +0000] "GET /service/rest/v1/search/assets?maven.artifactId=my-test-artifact&maven.classifier=with-dependencies&maven.extension=jar&maven.grou
pId=com.org.unit&repository=releases&version=1.7.0 HTTP/1.1" 500 - 6993 781 "Faraday v0.13.1" [qtp521628595-193856]

It starts reply with 500s.
nexus.log shows elasticsearch queue issues in this case:


2019-09-11 10:05:51,596+0000 WARN  [qtp521628595-211342]  api-user org.sonatype.nexus.siesta.internal.UnexpectedExceptionMapper - (ID a32e42f3-3987-4e2b-aafb-ae9c60419afa) 
Response: [500] 'ERROR: (ID a32e42f3-3987-4e2b-aafb-ae9c60419afa) 
Failed to execute phase [query], all shards failed; shardFailures {[vVKxus6uSPOF6C8arzXOog][077a1b5d670efb4255085546dbfca545826c4edb][0]: RemoteTransportException[[307A7895-C346D6EC-72C40E04-6A735952-0FC1E3FD][local[1]][indices:data/read/search[phase/query]]]; nested: EsRejectedExecutionException[rejected execution of org.elasticsearch.transport.TransportService$4@2d102d49 
on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@6182a0e3
[Running, pool size = 7, active threads = 7, queued tasks = 1000, completed tasks = 894001]]]; }

Is it some kind of a bug? I don’t believe Nexus is not able to proceed 50 concurrent requests.

Thanks.

You are exceeding the ability of the embedded elasticsearch to respond. There isn’t a hard limit, but it will depend on available system resources.

Thank you for your reply.

Do you mean elasticsearch queue?
I think 1000 is a pretty good size, I wonder why are these simple requests filling it up.

Is it normal behaviour?

On Nexus 2 I could execute similar requests using endpoint /nexus/service/local/artifact/maven/resolve? and had no issues with much more concurrent requests

I’m referring to the ability of the embedded elasticsearch instance to concurrently process all those requests. They are running very slowly due to lack of resources, and consequentially some are failing.

I think you should reduce the number of concurrent requests, or find another way to accomplish your task.

What are you trying to achieve here? Why are all those requests being made?

Thank you, that’s strange, we’re not reaching any resources limits, we have more than 50% CPU idling and about 6GB free memory available to java.

We have a chef recipe which periodically checks that some set of artifacts is still on the Nexus, the real number of concurrent requests which may come is between 300-1000 and Nexus 2 API is resolving it fine, I just found that I was able to reproduce it on 50 requests.

Yeah, if no more options, I’ll probably make it with a basic http head request to the direct artifact URL.

Thanks for a help.