Npm cannot find package that appears to exist in a private repository on our Nexus server

ichesal · October 4, 2020, 5:43am

Hello Community!

I’ve hit a wall debugging an issue with npm and a repository hosted in our Nexus server. The package I’m trying to download cannot be seen by the npm command. It’s clearly visible in the Nexus UI:

But npm shows nothing for that version when I try to view it:

~/src/tmp
❯ npm view mq-graphql-utils@1.9.0

~/src/tmp
❯

If I try to view the 1.9.1 version that’s also in the repository it works:

~/src/tmp
❯ npm view mq-graphql-utils@1.9.1

mq-graphql-utils@1.9.1 | ISC | deps: 7 | versions: 32
Schema, mocks and other utilities for GraphQL in Marqeta

dist
.tarball: https://nexus3.company.com/repository/mq-npm/mq-graphql-utils/-/mq-graphql-utils-1.9.1.tgz
.shasum: fc5e261c4bc0e87497cfab452bab27936932685c
.integrity: sha512-VhxTra5fL7PWGerNFXxJm6p4lgMlQ5NFrQm9VEgRFALPf6Y04SdaxUVnXExQzphqwR5mYvsJ4KZpNvT7TkPHuw==

dependencies:
apollo-cache-inmemory: 1.5.1 apollo-link-schema: 1.2.2    graphql-tag: 2.10.1          graphql: 14.2.1
apollo-client: 2.5.1         casual-browserify: 1.5.19-2  graphql-tools: 4.0.4

maintainers:
- releasedeploy <person@company.com>

dist-tags:
latest: 1.9.1

published 2 days ago by releasedeploy <person@company.com>

I’ve rebuilt the repository index and it hasn’t changed the problem.

We’ve deleted and re-published the 1.9.0 version of the package to no avail as well.

I’ve also verified that the .npmrc file is correct:

~/src/tmp
❯ cat ~/.npmrc
registry=https://nexus3.company.com/repository/mq-npm

An interesting observation is that if I curl the /repository endpoint that npm is calling, I don’t see the package version 1.9.0 listed in the output either, but 1.9.1 is there. In fact, a large number of versions are missing from that output and I see know way to paginate the call to query for more.

~/src/tmp
❯ curl -s https://nexus3.company.com/repository/mq-npm/mq-graphql-utils  | jq .time
{
  "created": "2019-11-04T17:32:34.247Z",
  "modified": "2020-10-01T23:26:20.918Z",
  "1.15.0": "2019-11-04T17:32:34.247Z",
  "1.16.0": "2019-11-04T17:48:42.290Z",
  "1.17.0": "2019-11-15T04:07:18.096Z",
  "1.18.0": "2019-12-04T20:53:50.773Z",
  "1.19.0": "2019-12-09T16:46:04.482Z",
  "1.20.0": "2019-12-13T23:43:14.720Z",
  "1.21.0": "2020-01-02T18:44:03.475Z",
  "1.25.0": "2020-02-10T18:29:36.106Z",
  "1.26.0": "2020-02-19T23:40:19.046Z",
  "1.27.0": "2020-03-02T22:29:26.668Z",
  "1.28.0": "2020-03-02T23:53:18.861Z",
  "1.30.0": "2020-03-10T01:14:58.603Z",
  "1.29.0": "2020-03-12T20:12:32.029Z",
  "1.29.1": "2020-03-12T20:13:44.228Z",
  "1.34.0": "2020-03-19T16:48:26.474Z",
  "1.35.0": "2020-03-23T17:21:18.974Z",
  "1.39.0": "2020-03-29T18:07:23.073Z",
  "1.40.0": "2020-03-30T02:47:53.315Z",
  "1.41.0": "2020-04-19T20:27:35.943Z",
  "1.42.0": "2020-05-01T23:34:42.116Z",
  "1.43.0": "2020-05-09T00:37:05.046Z",
  "1.44.0": "2020-05-11T21:13:16.965Z",
  "1.45.0": "2020-05-18T21:25:33.495Z",
  "1.49.0": "2020-06-01T22:06:38.279Z",
  "1.52.1": "2020-06-19T19:48:09.666Z",
  "1.53.0": "2020-06-25T20:18:40.647Z",
  "1.55.0": "2020-06-27T05:58:50.027Z",
  "1.56.0": "2020-08-10T21:18:59.932Z",
  "1.58.0": "2020-09-14T22:56:45.133Z",
  "1.59.0": "2020-09-21T22:43:48.079Z",
  "1.60.0": "2020-09-29T18:10:23.002Z",
  "1.9.1": "2020-10-01T23:26:20.918Z"
}

It really does have me stumped and, unfortunately, I’m neither an npm user or a nexus admin in any regard.

Are there any suggestions for trying to to debug this further? I’d very much appreciate any breadcrumbs anyone can suggest that might lead me towards a fix here.

Thank you!

ichesal · October 4, 2020, 6:38am

I can provide more information. I have a package version I know does not exist when I browser the web UI. If I try and download that package version with a curl call I get an error response:

~/src/tmp
❯ curl -O https://nexus3.company.com/repository/mq-npm-prod/voltron/-/voltron-1.19.10.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    55  100    55    0     0     79      0 --:--:-- --:--:-- --:--:--    79

~/src/tmp
❯ cat voltron-1.19.10.tgz
{"success":false,"error":"Package 'voltron' not found"}

But if I ask the /registry/voltron API to list all versions it SHOWS THIS VERSION in the output! So what’s in the repository and what that /registry API knows to be true are misaligned:

~/src/tmp
❯ curl -s https://nexus3.company.com/repository/mq-npm-prod/voltron | jq . | grep 1.19.10
    "1.19.10": {
      "version": "1.19.10",
      "_id": "voltron@1.19.10",
        "tarball": "https://nexus3.marqeta.com/repository/mq-npm-prod/voltron/-/voltron-1.19.10.tgz"
    "1.19.10": "2019-06-18T18:12:49.152Z",

I do not know what serves as the data source for the /registry calls – I’m flat out stumped on what can cause reality to drift for this end point and how to realign reality. I’ve rebuilt the index for the mq-npm-prod already with no change this to mis-match in reality.

matthew.parish · December 14, 2020, 9:12pm

Hi @ichesal, did you figure out the problem with this? We are experiencing a very similar issue with our Nexus Repo Mgr v 3.28.1-01.

For us, everything appears to work fine until we try to publish a new version of a package that is already in the registry. In such a case, the CLI publish works successfully and it shows the new version fine when browsing via the web UI, but trying npm view in the CLI will continue to yield the older version as the latest tag and attempting to view the new version with the specific @ syntax does nothing (no errors, but also no results). This continues for hours and even days after publishing before the registry finally does something that picks up the new version…

We’ve manually invalidated the cache, rebuilt the registry, and implemented a task which rebuilds the search index every 5 minutes, all in an attempt to get it to update…

Totally at a loss here and nearly completely blocked because of it.

mpiggott · December 14, 2020, 9:23pm

I don’t think your problem is the same.

It sounds like either a server between you and nexus caching or a your local npm client.

If you’re using a Nexus proxy repository that is proxying a Nexus hosted repository, that sounds normal because there is a cache duration and for performance reasons doesn’t check upstream on each request. The precise delay can be configured in the repository configuration.

matthew.parish · December 14, 2020, 9:30pm

Hi @mpiggott , thanks for your response.

I don’t have access to the server that our Nexus instance is hosted on itself, so I will ask our IT to check any caching rules there.

As far as our setup, we have a hosted npm registry, a proxy npm registry which is set up for npmjs.org, and an npm group registry that simply allows access to both of these. The issue is with the packages published to the hosted registry. I’ve tried the steps I mentioned on both the hosted registry and the group registry, with no success.

Hopefully I will have positive results with something on the server setup after speaking with IT…

You mentioned a potential issue with the local npm client. What could be an issue there and/or what would you recommend trying to fix it? From the documentation, I expect local npm to handle its own cache well, especially as manual manipulation of the cache is highly discouraged (both in documentation and the CLI).

mpiggott · December 14, 2020, 9:34pm

I believe NPM has its own cache of metadata that has its own timeout about when it should check upstream again.

mpiggott · December 14, 2020, 9:35pm

When you’re experiencing the problem I would suggest looking at the package root metadata to see if the version is present.

matthew.parish · December 14, 2020, 9:43pm

Apologies for having to ask, but I can’t find how to access the package root metadata. Is this done through the web UI, an HTTP request, or the CLI?

I’m also wondering if there isn’t potentially an issue with the registry setup in general, as using the /-/v1/search endpoint returns an empty array for “objects”, when it should return the full list of packages available (which I know there are multiple of).

Edit:
Force cleaning the local cache has no effect and I see no documentation about setting a timeout value.

Here’s an example of this silliness and inconsistencies I am experiencing in the CLI. Notice that the npm dist-tag ls npm command returns latest as 0.0.7 while npm view for the package shows that it is 0.0.6… This makes no sense.

mpiggott · December 15, 2020, 3:19pm

e.g. http://localhost:8081/repository/npm/jquery

shaskin · April 29, 2021, 4:18pm

We are running into this same problem, and have the same npm setup:

npm group repo
→ npm hosted repo
→ npm proxy which points to npmjs.org

A new version of a one of our in-house packages was published to the hosted repo over 24 hours ago. If I browse the package root metadata of the private repo, I get the correct latest version back, but when I do the same on the group repo, it reports the old version as latest.

It seems like there must be additional caching at the group-level repo, and if so we aren’t able to configure any timeout on caching at that level, as we would be for a proxy repo.