Sporadic slow downloads of maven-metadata.xml

Sonatype Nexus Repository
1

Sporadic slow downloads of maven-metadata.xml in self-hosted repo

Version: OSS 3.34.1-01

I’m experiencing an issue with extremely slow downloads of the maven-metadata.xml. The issue will come and go, but currently there will be at least a few instances every day. I’ve come across advice on setting cache TTL or repository routing rules, but those seem to apply to proxy repositories only. The issue we are having is with a self-hosted / local repository.

The issue will sometimes cripple our build pipeline because we will run out of parallel agents waiting on the maven-metadata.xml to download. I set a timeout on our Nexus connection to 5 minutes to help prevent pipeline backups. Without the timeout, it would sit for 60+ minutes at times.

Here is the Maven output of one that happened this morning. Notice the request to download started at 11:28:07 and my 5 minute timeout hit and terminated at 11:33:07.
2022-03-28T16:28:07.4417211Z 11:28:07,441 [INFO] — maven-deploy-plugin:2.7:deploy (default-deploy) @ notification-api —
2022-03-28T16:28:07.4436082Z 11:28:07,443 [INFO] Uploading to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.jar
2022-03-28T16:28:07.5195469Z 11:28:07,518 [INFO] Uploaded to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.jar (27 kB at 365 kB/s)
2022-03-28T16:28:07.5197538Z 11:28:07,519 [INFO] Uploading to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.pom
2022-03-28T16:28:07.5741373Z 11:28:07,573 [INFO] Uploaded to nexus: myco/repository/maven-releases/com/myco/notification-api/whth-2210-dev-7/notification-api-whth-2210-dev-7.pom (2.4 kB at 44 kB/s)
2022-03-28T16:28:07.5742895Z 11:28:07,573 [INFO] Downloading from nexus: myco/repository/maven-releases/com/myco/notification-api/maven-metadata.xml
2022-03-28T16:33:07.6659037Z 11:33:07,664 [WARNING] Could not transfer metadata com.myco:notification-api/maven-metadata.xml from/to nexus (This is the timeout)

While the build is stuck at this step, I can copy the link to the xml and paste it into my browser, and it just spins and spins. Eventually, it will come back and serve the xml. After more time passes, all subsequent builds can retrieve it without any problems.

In the Nexus request.log, you can see the file here. 133,850 bytes and took 303,179ms to serve. Just missed my 5 minute timeout. I have seen request entries here taking millions of milliseconds to return.
[28/Mar/2022:11:33:10 -0500] “GET /repository/maven-releases/com/myco/notification-api/maven-metadata.xml HTTP/1.1” 200 - 133850 303179 “Apache-Maven/3.8.4 (Java 11.0.14; Linux 4.15.0-167-generic)” [qtp585165516-243698]

The nexus.log has no entries at all during this time of note.

The interesting thing is this only seems to be happening in our maven-releases repository. I’ve never seen this happen in maven-snapshots. I have a suspicion we have too many releases for our artifacts. The maven-metadata.xml has 3,800+ releases. We do purge artifacts and I think we need to purge more. However, it still doesn’t make sense how the file can be served like normal for 98% of requests. Is there a practical limit to the amount of releases the server can handle for a single artifact?

This issue is happening even with low load on the server.
So far I have tried:

  • Nexus has been restarted
  • Entire Linux host machine has been restarted
  • Set up exclusion routing rules to our in house artifacts from reaching out to proxy repos (no need since they are in local)
  • Ran rebuild / repair maven-metadata.xml admin task
  • Ran rebuild / repair browse and search
  • Ensured no Nexus scheduled tasks are running during the times we see slow downloads

Any other ideas or things to try would be appreciated. We are running out of things to try.

1 Like
2

You might try upgrading to 3.37.3 or newer as I’m pretty sure we worked on some performance improvements around the maven metadata regeneration. I did see this postgres-specific issue, but I seem to remember another issue that I can’t find at the moment. Upgrading may fix your performance problem.

3

Thank you. I will give that a try.

4

We upgraded to OSS 3.38.0-01. Not a single slow download of the maven-metadata.xml in 48 hours! Fingers crossed this was just what we needed. Thank you!

2 Likes
5

Everything was going smoothly for a while. I did see a few slow downloads here and there about a week past the upgrade. Now we are seeing them daily again. Unfortunately, we are pretty much back to the same symptoms as I described in this original post.

We did reallocate significantly more resources to the Nexus Repo server and still didn’t help. Any other ideas or things to try?

6

Is there any update on this? I am experiencing this problem too on version 3.37.3-02 PRO

For our instance the maven-metadata.xml has over 2800+ releases. We are using the

Staging feature to move specific versions into another repository via a move API call. eg. POST service/rest/v1/staging/move/{repository}

Upon each move, I’m finding it is then taking a long time to download the maven-metadata.xml. (I assume it is updating the file to remove the version which just has been moved out of that repository)

7

A large number of releases, particularly if there are maven plugins or snapshots will definitely negatively impact rebuild times. If you have old snapshots removing them could help.

If you have a support contract I’d suggest filing a ticket