Download speed intermittently slow

psk · November 19, 2021, 1:30pm

Hi,

We have noticed that the download speed of maven artifacts are intermittently reduced to Bytes/second. We usually see transfer speeds at kB/s or even MB/s.

The problem is not reproducable, in the sense that the same artifact is not always slow or always fast, we don’t have any firm statistics on it yet but it seems that artifacts which are proxied from Central Repository: are more affected than a self hosted repository.

I’ve tried play around with invalidating cahce, swithcing up and down Maximum component age, Maximum metadata age and Not found cache TTL, without seeing any reduction of transfer speed.

We usually see at least some slow artifact transfers per day - and in some periods it is bad enough to cause a trippling of maven build times.

Any suggestions as for which direction to look? Any logging we can enable - which helps and does not produce millions of lines on a daily basis?

We are using OSS 3.25.0-03

Thanks in advance
Peter

mmartz · November 19, 2021, 2:37pm

Perhaps it’s related to the amount of load hitting your Nexus Repository instance? What file system are you using? There are some file systems that are known to be problematic for Nexus Repository. (System Requirements) When you see slow transfers are they happening to all downloads or just particular downloads/builds?

psk · November 23, 2021, 11:24am

Hi,

We are running on Windows 2012R2.

With regards to the load, as far as I can see we slow download speed both when busy and when not-so-busy. On the other hand we also see fast download speeds when Nexus is serving several builds at the same time.

It seems that downloads for several builds are affected - not only one. Most are being served by our build servers, this is only natural as the developers would have the majority cached locally, though slow downloads to developers PC’s are also seen.

A small snippet from the log |Timestamp |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:08:57 +0100 | GET |19/nov./2021:01:09:03 +0100 | GET |19/nov./2021:01:09:09 +0100 | GET |19/nov./2021:01:09:09 +0100 | GET |19/nov./2021:01:09:09 +0100 | GET |19/nov./2021:01:09:09 +0100 | GET |19/nov./2021:01:09:09 +0100 | GET |19/nov./2021:01:09:16 +0100 | GET |19/nov./2021:01:09:21 +0100 | GET |19/nov./2021:01:09:24 +0100 | GET |19/nov./2021:01:09:29 +0100 | GET |19/nov./2021:01:09:30 +0100 | GET |19/nov./2021:01:09:30 +0100 | GET |19/nov./2021:01:09:30 +0100 | GET |19/nov./2021:01:09:30 +0100 | GET while we are running quite a few nightly builds
| Operation | Size |Elapsed time| KB/s |
| 36872 | 0 | 0 |
| 40 | 0 | 0 |
| 10242 | 0 | 0 |
| 40 | 0 | 0 |
| 13407 | 0 | 0 |
| 40 | 0 | 0 |
| 40 | 0 | 0 |
| 40 | 0 | 0 |
| 40 | 0 | 0 |
| 13612 | 16 | 830,8105469 |
| 16763 | 16 | 1023,132324 |
| 52543 | 15 | 3420,768229 |
| 151617 | 16 | 9253,967285 |
| 223171 | 16 | 13621,27686 |
| 15671 | 8003 | 1,912246775 |
| 27243 | 0 | 0 |
| 40 | 0 | 0 |
| 13974 | 0 | 0 |
| 40 | 5320 | 0,007342575 |
| 40 | 16 | 2,44140625 |
| 5522 | 7238 | 0,745037044 |
| 40 | 4766 | 0,008196076 |
| 12894 | 2793 | 4,508341165 |
| 40 | 5846 | 0,006681919 |
| 19206 | 0 | 0 |
| 40 | 0 | 0 |
| 55943 | 0 | 0 |
| 40 | 0 | 0 |

Note KB/s is a calculated value, so if the elapsed time is 0, we simply set the KB/s to 0 as well, therefore a KB/s of 0 is good and acceptable - counter intuitive though

Another snip from a log during the day where Nexus is not busy, but still show slow download

| Timestamp | Operation | Size | Elapsed time | KB/s |
|[15/nov./2021:09:53:32 +0100]| GET | 393 | 0 | 0,00 |
|[15/nov./2021:09:53:34 +0100]| GET | 5679 | 1094 | 5,07 |
|[15/nov./2021:09:53:34 +0100]| GET | 1772 | 1500 | 1,15 |
|[15/nov./2021:09:53:34 +0100]| GET | 3051 | 1625 | 1,83 |
|[15/nov./2021:09:53:34 +0100]| GET | 5665 | 281 | 19,69 |
|[15/nov./2021:09:55:01 +0100]| GET | 10130 | 16 | 618,29 |
|[15/nov./2021:09:57:00 +0100]| GET | 10130 | 0 | 0,00 |
|[15/nov./2021:10:00:01 +0100]| GET | 10130 | 0 | 0,00 |
|[15/nov./2021:10:00:09 +0100]| GET | 457 | 218 | 2,05 |

Host Resources

The machine is a virtual machine served by VMWare
We never see CPU go above 40%
Memory hovers between 34% and 42% utilization
Disk has ~33% free space