We’ve been running 3.2.0-1 successfully for years with maven. The content started filling up the disk and I noticed that recent releases have a clean up task, so decided to upgrade to the latest release (3.29) and set up tasks for deleting older content. However after the upgrade, all repositories show up as empty. I’ve tried the following to no effect:
“Rebuild index” button on each repo
“Repair - Reconcile component database from blob store” task
used absolute paths to the content on disk, so it’s definitely set to right location
Something to note is that the disk is 97% full and I had to set “cluster.routing.allocation.disk.threshold_enabled: false” in elasticsearch.yaml to turn off constant warnings. The previous version (3.2.0) didn’t have any problem at all with the almost-full disk.
Also all the content is still present in the blob, nothing seems to have been deleted. Any idea why Nexus is unable to browse the content in the blobs? Any assistance will be much appreciated.
Is there anything interesting logged in nexus.log or any of the tasks you ran (they log to separate log files)?
Is it only the UI affected or are the components missing when using a client like Maven?
As you’re approaching a full disk, you’re risking database corruption.
I suggest to free up some disk space, monitor the nexus.log and re-run Repair - Rebuild repository browse if it’s only the UI that’s affected, or Repair - Reconcile component database from blob store if the components are missing when using Maven or other tools.
Thanks for the quick response, David! CLI fetches via Maven are also affected. The log file is clean on every restart, all INFO level logs. I’ve browsed each task log too, nothing that indicates an error. Only warning at startup is about a missing ValidatorFactory. The reindex tasks finish within a few seconds.
2021-02-16 04:02:26,390-0800 INFO [jetty-main-1] *SYSTEM org.jboss.resteasy.plugins.validation.i18n - RESTEASY008550: Unable to find CDI supporting ValidatorFactory. Using default ValidatorFactory
I’ve run the reconcile and repair tasks multiple times, hasn’t helped. I didn’t back up the nexus3 store because it is 1.6TB and it was going to take too long to copy over and then test the upgrade. Bad decision but too late now. Is there any way I can go back to 3.2.0?
I’m afraid not. When you upgrade, we run some database upgrade steps when DB schema needs updating, and we do not have a process to reverse that. Would you be able to check the logs from your updgrade process - that would be the first boot using the new version - maybe there’s something there.
Is it possible you have had been using non-default data location as mentioned in our Upgrading documentation?
Probably before doing anything more, consider taking a backup now as any actions you’re taking may make (or have made) the situation worse, or Nexus might decide to delete files its believes are unused, etc.
I think the first thing you might want to do is look at the contents of asset & component tables in the database (the stackoverflow post contained instructions on how to connect, OrientDB docs 2.x for general queries)
If those tables are empty, perhaps clean-up policies were configured such that Nexus marked all the content as deleted? To check if this is the case, in your blobstore(s) each blob should have a corresponding properties file, inside the properties file you would see deleted=true. If that is the case, you could probably try writing a script to remove those entries, then try the reconcile task.
It did have an error about not being able to read a file inside the config database. Error message told me to export and import, which I did using the orientdb jar file. Did the same for component as well, not for any other database inside the “db” folder.
The data resides in /data/nexus-data but it is symlinked across like so:
/opt/sonatype/sonatype-work/nexus3 → /data/nexus-data
The app is in /opt/sonatype/nexus (peer of /opt/sonatype/sonatype-work)
Unfortunately I am not. We are using the OSS version.
Thanks Matthew. On connecting to the database and running list indexes, all show with a zero count. Another person in the org had a query they had run in the past and that returns zero items as well.
SELECT @rid, bucket, component, name, COUNT(*) as count FROM asset GROUP BY bucket, component, name
The Nexus UI shows blob size as 1223GB and on-disk the size is 1.6TB. So only about 400GB is marked as deleted. Shouldn’t the Repair - Reconcile component database from blob store task still work with the 1223GB of content?
As a last resort, is it possible to re-upload content from the blob into nexus, instead of asking developers to run a large number of builds again?
This is now resolved. While looking up the properties files for information on uploading via curl[1], I noticed that the repository names didn’t match what was in the installation. I am unsure how this happened (maybe my incorrect steps or something else) but earlier we had repos set up as “snapshots”, “releases” and so on whereas the existing repos were “maven-releases”, “maven-snapshots”, etc. Upon adding back the repos as we had it and removing the new ones, the reconcile was able to reconstruct internal state. We’re now back to normal.
Regarding soft deleted content, the UI shows the blob to be 1.2TB and disk usage is 1.6TB. First course of action is to compact, then set up new cleanup policies.
Many thanks to Dawid Sawa and Matthew Piggott for their quick assistance.