We recently upgraded from Nexus Repository Manager 3.41.0 to 3.51.0 using the Sonatype Helm Chart. During this upgrade we experienced a host of database errors. This eventually led to us restoring from a Database backup from the previous day. This allowed us to stand Nexus up successfully. However, our users have been experiencing a number of errors when using Repository manager.
Errors
Some of these are Maven Metadata errors, which seem to be fixed by running the ‘Rebuild Maven Metadata’ task.
Some are 500 errors for our Helm chart repos, which are usually fixed by rebuilding the index.
The error we have not been able to fix is when users are pushing to and pulling from npm repos:
npm WARN tarball tarball data for … (repo and sha info) seems to be corrupted. Trying again.
npm WARN tarball tarball data for (repo and sha info) seems to be corrupted. Trying again.
npm ERR! code EINTEGRITY
npm ERR! (Different sha). (277462 bytes)
(I’ve removed some identifying info from these error logs)
I’ve tried to rebuild the index of the NPM repo which seems to have no effect, and also run the rebuild metadata job which fails with:
Failed to run task 'Rebuilding npm metadata of <NPM-REPO>' on repository '<NPM-REPO>'
org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@297b9fd1-f580-4bbc-941a-ad3adff792f2 exists in metadata, but is missing from the blobstore
I’m hesitant to run the ‘Repair - Reconcile component database’ task due to the size of our blobstore (2.5TB), as I imagine this will take a huge amount of time.
You can configure the task so that it only reconciles blobs created within a certain amount of days, this would considerably reduce the time the reconciliation takes. Could you set this to the date before you upgraded?
There’s also an option to do a try run so you could get an idea of how much work will take place. And roughly how long it would take.
You can see the logs for this under support > logs and look for the ’ tasks/blobstore.rebuildComponentDB’ log.
As always, would recommend running this outside of normal working hours.
Thanks for the quick reply. I have a dry run currently going that I set off yesterday. I set days to 2 and it’s been running for around 24 hours, hopefully it won’t be too much longer. I’m currently thinking this reconcile db task is our best course of action at the moment.
Do you have any thoughts on why this might be particularly affecting the NPM repos and why the rebuild NPM metadata tasks are failing?
Some other behaviour I’ve noticed is that for certain users private repos we are not able to push or pull using NPM, as it returns:
npm ERR! code ETARGET
npm ERR! notarget No matching version found for <PACKAGE>
However I seem to be able to browse to the files location and download to my desktop directly.
I’ve tried clearing node_modules and npm cache to no avail.
Sorry I can’t give you more specific guidance. Just offering suggestions based on the docs.
It does sounds like it may be your only option but of course explore what other options you can if you are not comfortable with the reconcile task (unsure what else to suggest).
Npm looks to be one of the format that is effected by this db/blob mismatch. It sounds like once you have reconciled the DB you can then re-run the rebuild npm metadata and reconcile npm search metadata.
@mpiggott - the npm metadata task fails unfortunately, which is why I’m considering the reconcile component db job first
@laurencet - thanks again. I’m pretty set on running the reconcile task now. Is there a reason you suggest running outside of normal hours? Is this a CPU thing or are there other risks that come with this task? I can see the docs say not to run during normal service (I assume this means don’t run this task if there is nothing wrong).