Sonatype Nexus OSS 3.69.0 Npm Issue

Sonatype Nexus Repository
, ,
1

I am using a Nexus (3.69.0 oss) Docker image with Maven, npm, and Docker repositories feeded in it. This image needed to be deployed across different environment and use as an artifact central.

For Maven and Docker, it is working fine.

But when it comes to npm, when I use the Nexus container with these image on my local machine without internet access, everything works as expected., However, when I run the same setup in another environment(aws ec2 with 8CPU,32GB RAM), also without internet access, I encounter issues with npm.

Specifically, npm takes about five minutes to retrieve a single package. The npm repository is a combination of both proxy(2k dependencies) and hosted repositories(7 artifacts), and I have artifacts stored in both. Despite this, I am seeing significant delays.

To install a single package, it is taking around 4-5 minutes. instead, 1-2 seconds in local.

I have confirmed that npm ping works correctly, and I can download npm artifacts using wget with no problems. However, npm install is where the issue occurs.

Could you please help me understand why this is happening and how it can be resolved?

In logs also I cannot see any., regarding delays.,

Configuration details:
some packages are in hosted repo, and some are in proxy cached, I kept negative cache to 0, and turned off it, meatda age and max component ages to -1., to use the caching always…No routing rules, and no firewalls enabled etc.

and this is the data that I am using in to create proxy npm repo:

{
“name”: “‘$npm_proxy_repository_name’”,
“online”: true,
“storage”: {
“blobStoreName”: “default”,
“strictContentTypeValidation”: true
},
“cleanup”: {
“policyNames”: [
“string”
]
},
“proxy”: {
“remoteUrl”: “https://registry.npmjs.org”,
“contentMaxAge”: -1,
“metadataMaxAge”: -1
},
“negativeCache”: {
“enabled”: false,
“timeToLive”: 0
},
“httpClient”: {
“blocked”: false,
“autoBlock”: false,
“connection”: {
“retries”: 10,
“userAgentSuffix”: null,
“timeout”: 120,
“enableCircularRedirects”: false,
“enableCookies”: false,
“useTrustStore”: false
},
“authentication”: null
},
“routingRule”: “string”,
“replication”: {
“preemptivePullEnabled”: false,
“assetPathRegex”: “string”
},
“npm”: {
“removeNonCataloged”: false,
“removeQuarantined”: false
}
}

It’s a mysterious question still for us , why it is working in local mcahine, not in the aws server,

FYI, using the nexus server and npm client server, both are in same machine

2

Just my first idea would be to never disable negative cache, as it is a nice feature to reduce latency.

Furthermore i know that the creation of metadata (metadataMaxAge) documents can take ages, but you never should set it to -1. Ideally optimal value is everything between 12 and 24 hours, with 24 hour providing most performance.

Can you outline what download actually take long, is it just file transfer speed, or slow metadata downloads? Maybe some detail will help.

Regards,

Joris

3

I’d suggest first taking a look at Nexus’s request log around the time you’re sending requests. On each line the number before the user agent is the duration of the request from the perspective of Nexus and you may be able to identify what request(s) may be taking longer than expected. There is also an outbound request log which may contain information about requests Nexus is making to remotes.

Anti-virus, network proxies, etc. can also delay requests so you should take a look at those.

4

Hey @joris.klop and @mpiggott , Thanks for responding,

My requirement is a little bit complex, that I don’t have internet is my machines, so, it shouldn’t never check for newer artifact version. For this, I need to use metadataaMaxAge to be -1. Also, the issue with slowness is, I figured it out, that I have 3 repos in my group…, hosted repo + proxy(1) (which having around 3k libraries with outbound disbaled, so it needs to be act as hosted repo) + (another proxy repo adjusted to npmjs.com) , eventhough my envirnoment doesn’t have internet, some calls still going to proxy(3), and eventually failing/taking a lot of time to get a single artifact.

When I reduce connection timeout and retries in http section, it is working, or when I disable outbound in proxy(3), it is working.

But still doesn;t have idea, why the call going for proxy-repo(3), when all artifacts are dumped in hosted and proxy-repo(2), which is using as a hosted repo.

Based on docs, I set first hosted, proxy(1) and then proxy(2)., but it is not honoring proxy(1)., if it does, why still calls are going for proxy(2)?

I tested with putting one more proxy(dumped with few artifacts and disabled outbound for it). Let it be proxy(1.1)

now order is hosted->proxy(1)->proxy(1.1)–>proxy(2)

still call is going for proxy(2)., those ignoring (1.1) also…,

is there any mechanism to avoid this, or like setting time to wait , take from it, if not go for second, third, fourth and make sure to use cache strictly (-1) I am using currently for MetadataMaxAge and artifactAge…, seems not working

FYI: eventhough calls are going to proxy(2), my scenario is working as it doesnt have any artifacts, but I enable internet, it will take from there…,

In log i can see, requests are going same time to both repos and proxy (2) is getting 404 and proxy (1.1) is 200.
I am access through npm bearer token realm,

Negative timeout is a good feature, but sometimes, dont know when the request comes, and that case, that can be in available for download., I think I can’t be depend on it.

5

If you’re using a group repository then Nexus will merge the metadata from all group members and when downloading a package Nexus will consult each repository member in order.

6

@mpiggott , I think, my nexus is not honoring the order of members in the group. If it merges the metadata from all group members, then what will consider first? and from here-from-stackoverflow, they are saying, The order of repositories in a group is only going to matter if there is the exact same version of a component in more than one repository. If it is correct, how can I say the component should use order list of members in group?

My exact problem is, my nexus request is going for particular member in the groped one, it is proxy repo only, but in previous members, it is not checking there…, even if the artifacts are in previous repos

7

I’m not entirely sure what you mean - for merging it uses metadata from all repositories. If there is a collision in versions the metadata should prioritize information (e.g. checksum) from earlier members.

When downloading a tarball Nexus will check each member in order.

8

@mpiggott , Thanks for responding, and sorry for the confusion.

What my exact situation where I got blocked is , I have a Nexus setup with the following repositories:

  1. Hosted repository
  2. npm proxy 1: Has cached artifacts, MaxComponentAge and metadataMaxAge set to -1, and outbound connections disabled.
  3. npm proxy 2: Also has cached artifacts, MaxComponentAge and metadataMaxAge set to -1, and outbound connections disabled.
  4. npm proxy 3: Proxies another Nexus instance (Nexus A), which does not have internet access. This proxy is also set to retrieve from cached artifacts and some times based on requirement we will enable internet., but outbound for now in this Nexus A is opened.

All of these repositories are included in a grouped repository with the following order:

  1. Hosted
  2. npm proxy 1
  3. npm proxy 2
  4. npm proxy 3

The issue I’m facing is that, even when the artifacts are cached in npm proxy 1 or npm proxy 2, the requests are still first being routed to npm proxy 3, which is connected to the other Nexus instance without internet., this will be a great danger to us.

I have confirmed that the group members are in the correct order and that the caches in npm proxy 1 and npm proxy 2 should contain the necessary artifacts.

Why is Nexus first not using the cached artifacts from npm proxy 1 and npm proxy 2, and instead sending requests to npm proxy 3?, at final they are using from cache only, but still call to proxy 3 is confusing

9

See my previous comment: all repositories will be used for metadata.

10

ok, got it @mpiggott , can I have any api to see that metadata file of group, and what does it consits, and how to behaves like?

11

An example from the npm registry would be for the jquery package: https://registry.npmjs.org/jquery/