Upgrade of OSS Nexus Repository Manager from 3.29.2 to 3.48.0 fails on exception during index creation

Hi, I am trying to upgrade Nexus from version 3.29.2 to 3.48.0. For deployment of Nexus we use Official Docker container (originally 3.29.2; target version 3.48.0). Environment i test it on is:

OS: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
Docker: 23.0.1
Used DB: Embedded OrientDB

Procedure used for upgrade (based on this):

  1. Stop and remove current container using command
docker stop --time=120 nexus | xargs docker rm
  1. Run new container with the same persistent data using command
docker run --name=nexus --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.48.0

Then the startup of a container fails with exit code 255. Complete output of docker logs nexus is here.

Seems like problem with index creation in component database according to this line

2023-03-09 13:24:16,103+0000 ERROR [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.index.OIndexUnique - $ANSI{green {db=component}} Exception during index 'asset_bucket_rid_idx' creation

First i tried to upgrade gradually, i.e. to apply same procedure mentioned above to upgrade e.g. from 3.29.2 to 3.30.0.

Here follows matrix of versions i tried to upgrade to (from 3.29.2) and corresponding errors that have arised.

Nexus version Exception text
3.30.0 Exception during index ‘deleted_blob_index_blobstore_blob_id_idx’ creation
3.30.1 Exception during index ‘deleted_blob_index_blobstore_blob_id_idx’ creation
3.33.1 Exception during index ‘deleted_blob_index_blobstore_blob_id_idx’ creation
3.37.3 Exception during index ‘coordinate_download_count_repository_name_namespace_name_version_date_ip_address_username_idx’ creation
3.42.0 Exception during index ‘asset_bucket_rid_idx’ creation
3.48.0 Exception during index ‘asset_bucket_rid_idx’ creation

As next step i tried to use OrientDB Console to migrate records from component DB and then to build indexes manually while running both Nexus versions simultaneously. The config and security databases were migrated whole.

Procedure i used is following:

  1. Start Nexus 3.29.2 with all the data
docker run --name=nexus-3.29.2 --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.29.2
  1. Perform DB exports from ODB Console
docker exec -it nexus-3.29.2 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar'

connect plocal:/nexus-data/db/component admin admin
export database component -excludeAll -includeCluster="asset bucket component"

connect plocal:/nexus-data/db/config admin admin
export database config

connect plocal:/nexus-data/db/security admin admin
export database security
  1. Copy exports to new location
mkdir -p /mnt/nexus-data-new/console
chown -R 200:200 /mnt/nexus-data-new
cp /mnt/nexus-data/console/{component,config,security}.json.gz /mnt/nexus-data-new/console/
  1. Start clean Nexus 3.48.0 with no data
docker run --name=nexus-3.48.0 --volume /mnt/nexus-data-new:/nexus-data -d -p 8881:8081 sonatype/nexus3:3.48.0
  1. Perform DB imports from ODB Console
docker exec -it nexus-3.48.0 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar'

connect plocal:/nexus-data/db/component admin admin
import database component.json.gz -merge=true

During import a following error occured but i decided to ignore it for this time as it was only test run and most of the data were imported.

Error on database import happened just before line 0, column 178093048
com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids'
	at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:190)
	at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2214)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
	at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
	at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
	at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
	at org.sonatype.nexus.orient.console.Main.main(Main.java:63)

Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'component' from file: component-clusters.json.gz

Error: com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids'

Then import of security and config databases.

connect plocal:/nexus-data/db/security admin admin
import database security

connect plocal:/nexus-data/db/config admin admin
export database config
  1. Rebuild indexes in component db
connect plocal:/nexus-data/db/component admin admin
rebuild index *

During this operation following error occured

Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #23:10: found duplicated key 'maven-central' in index 'bucket_repository_name_idx' previously assigned to the record #23:3
	DB name="component" INDEX=bucket_repository_name_idx RID=#23:3

To fix this it was neccessary to locate all these duplicates in class bucket and for each perform

truncate record #23:3

Then repeat rebuild index *

  1. Stop both Nexuses and move blobstore to a new location
docker stop --time=120 nexus-3.48.0
docker stop --time=120 nexus-3.29.2
rm -rf /mnt/nexus-data-new/blobs && mv /mnt/nexus-data/blobs /mnt/nexus-data-new/
  1. Start Nexus 3.48.0

Nexus has started and authentication to GUI worked. Also browsing and downloading content worked. I had assumed inconsistency in data so i ran Task Repair - Reconcile component database from blob store. This Task stuck in state Running and never finished. Also log for the Task stopped to grow.
So Tasks were broken and data presumably inconsistent but reachable.

Any suggestions how to perform the upgrade?

This looks similar to Failed while upgrading from 3.15 to 3.39 but it’s a year old topic and without answer.


After some research and trials and errors i had came up with solution.
It was no go to try to solve it via direct db operations. Rather i used nexus3-cli in combination with Maven CLI to download packages from old instance and then upload into clean Nexus3 instance.
My case comprises only of Maven and npm repos.
After this i was able to upgrade Nexus3 to latest version using official procedure.

Prepare the environment

Migration procedure

  1. Start latest Nexus 3
  2. Manually create repositories (maven, npm)
  3. Allow scripting by inserting line nexus.scripts.allowCreation=true to $nexus-data/etc/nexus.properties and restarting Nexus (This is needed for REST API to work, hence nexus3-cli to work)
  4. Download contents of hosted repos
    1. npm - use nexus3-cli
      nexus3 download $repo_name/ $repo_name/
    2. maven - use custom bash script based on this
  5. Upload contents to new instance
    1. npm - Use nexus3-cli - beware of upload versions in proper order because of subsequent usage of semVer wildcards
      for i in $(find $repo_name -name '*tgz' | sort --version-sort) ; do
          nexus3 upload "$i" "$i" >> npm.err 2>&1
    2. maven - use mvn cli in bash script based on this
  6. turn off scripting as per 3.