Sporadic slow downloads of maven-metadata.xml in self-hosted repo
Version: OSS 3.34.1-01
I’m experiencing an issue with extremely slow downloads of the maven-metadata.xml. The issue will come and go, but currently there will be at least a few instances every day. I’ve come across advice on setting cache TTL or repository routing rules, but those seem to apply to proxy repositories only. The issue we are having is with a self-hosted / local repository.
The issue will sometimes cripple our build pipeline because we will run out of parallel agents waiting on the maven-metadata.xml to download. I set a timeout on our Nexus connection to 5 minutes to help prevent pipeline backups. Without the timeout, it would sit for 60+ minutes at times.
Here is the Maven output of one that happened this morning. Notice the request to download started at 11:28:07 and my 5 minute timeout hit and terminated at 11:33:07.
2022-03-28T16:28:07.4417211Z 11:28:07,441 [INFO] — maven-deploy-plugin:2.7:deploy (default-deploy) @ notification-api —
2022-03-28T16:28:07.4436082Z 11:28:07,443 [INFO] Uploading to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.jar
2022-03-28T16:28:07.5195469Z 11:28:07,518 [INFO] Uploaded to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.jar (27 kB at 365 kB/s)
2022-03-28T16:28:07.5197538Z 11:28:07,519 [INFO] Uploading to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.pom
2022-03-28T16:28:07.5741373Z 11:28:07,573 [INFO] Uploaded to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.pom (2.4 kB at 44 kB/s)
2022-03-28T16:28:07.5742895Z 11:28:07,573 [INFO] Downloading from nexus: myco/repository/maven-releases/com/myco/notification-api/maven-metadata.xml
2022-03-28T16:33:07.6659037Z 11:33:07,664 [WARNING] Could not transfer metadata com.myco:notification-api/maven-metadata.xml from/to nexus (This is the timeout)
While the build is stuck at this step, I can copy the link to the xml and paste it into my browser, and it just spins and spins. Eventually, it will come back and serve the xml. After more time passes, all subsequent builds can retrieve it without any problems.
In the Nexus request.log, you can see the file here. 133,850 bytes and took 303,179ms to serve. Just missed my 5 minute timeout. I have seen request entries here taking millions of milliseconds to return.
[28/Mar/2022:11:33:10 -0500] “GET /repository/maven-releases/com/myco/notification-api/maven-metadata.xml HTTP/1.1” 200 - 133850 303179 “Apache-Maven/3.8.4 (Java 11.0.14; Linux 4.15.0-167-generic)” [qtp585165516-243698]
The nexus.log has no entries at all during this time of note.
The interesting thing is this only seems to be happening in our maven-releases repository. I’ve never seen this happen in maven-snapshots. I have a suspicion we have too many releases for our artifacts. The maven-metadata.xml has 3,800+ releases. We do purge artifacts and I think we need to purge more. However, it still doesn’t make sense how the file can be served like normal for 98% of requests. Is there a practical limit to the amount of releases the server can handle for a single artifact?
This issue is happening even with low load on the server.
So far I have tried:
- Nexus has been restarted
- Entire Linux host machine has been restarted
- Set up exclusion routing rules to our in house artifacts from reaching out to proxy repos (no need since they are in local)
- Ran rebuild / repair maven-metadata.xml admin task
- Ran rebuild / repair browse and search
- Ensured no Nexus scheduled tasks are running during the times we see slow downloads
Any other ideas or things to try would be appreciated. We are running out of things to try.