Hi, I am trying to upgrade Nexus from version 3.29.2 to 3.48.0. For deployment of Nexus we use Official Docker container (originally 3.29.2; target version 3.48.0). Environment i test it on is:
OS: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
Docker: 23.0.1
Used DB: Embedded OrientDB
Procedure used for upgrade (based on this):
- Stop and remove current container using command
docker stop --time=120 nexus | xargs docker rm
- Run new container with the same persistent data using command
docker run --name=nexus --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.48.0
Then the startup of a container fails with exit code 255. Complete output of docker logs nexus is here.
Seems like problem with index creation in component database according to this line
2023-03-09 13:24:16,103+0000 ERROR [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.index.OIndexUnique - $ANSI{green {db=component}} Exception during index 'asset_bucket_rid_idx' creation
First i tried to upgrade gradually, i.e. to apply same procedure mentioned above to upgrade e.g. from 3.29.2 to 3.30.0.
Here follows matrix of versions i tried to upgrade to (from 3.29.2) and corresponding errors that have arised.
| Nexus version | Exception text |
|---|---|
| 3.30.0 | Exception during index ‘deleted_blob_index_blobstore_blob_id_idx’ creation |
| 3.30.1 | Exception during index ‘deleted_blob_index_blobstore_blob_id_idx’ creation |
| 3.33.1 | Exception during index ‘deleted_blob_index_blobstore_blob_id_idx’ creation |
| 3.37.3 | Exception during index ‘coordinate_download_count_repository_name_namespace_name_version_date_ip_address_username_idx’ creation |
| 3.42.0 | Exception during index ‘asset_bucket_rid_idx’ creation |
| 3.48.0 | Exception during index ‘asset_bucket_rid_idx’ creation |
As next step i tried to use OrientDB Console to migrate records from component DB and then to build indexes manually while running both Nexus versions simultaneously. The config and security databases were migrated whole.
Procedure i used is following:
- Start Nexus 3.29.2 with all the data
docker run --name=nexus-3.29.2 --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.29.2
- Perform DB exports from ODB Console
docker exec -it nexus-3.29.2 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar'
connect plocal:/nexus-data/db/component admin admin
export database component -excludeAll -includeCluster="asset bucket component"
disconnect
connect plocal:/nexus-data/db/config admin admin
export database config
disconnect
connect plocal:/nexus-data/db/security admin admin
export database security
disconnect
- Copy exports to new location
mkdir -p /mnt/nexus-data-new/console
chown -R 200:200 /mnt/nexus-data-new
cp /mnt/nexus-data/console/{component,config,security}.json.gz /mnt/nexus-data-new/console/
- Start clean Nexus 3.48.0 with no data
docker run --name=nexus-3.48.0 --volume /mnt/nexus-data-new:/nexus-data -d -p 8881:8081 sonatype/nexus3:3.48.0
- Perform DB imports from ODB Console
docker exec -it nexus-3.48.0 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar'
connect plocal:/nexus-data/db/component admin admin
import database component.json.gz -merge=true
disconnect
During import a following error occured but i decided to ignore it for this time as it was only test run and most of the data were imported.
Error on database import happened just before line 0, column 178093048
com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids'
at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:190)
at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2214)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405)
at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260)
at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131)
at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145)
at org.sonatype.nexus.orient.console.Main.main(Main.java:63)
Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'component' from file: component-clusters.json.gz
Error: com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids'
Then import of security and config databases.
connect plocal:/nexus-data/db/security admin admin
import database security
disconnect
connect plocal:/nexus-data/db/config admin admin
export database config
disconnect
- Rebuild indexes in component db
connect plocal:/nexus-data/db/component admin admin
rebuild index *
During this operation following error occured
Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #23:10: found duplicated key 'maven-central' in index 'bucket_repository_name_idx' previously assigned to the record #23:3
DB name="component" INDEX=bucket_repository_name_idx RID=#23:3
To fix this it was neccessary to locate all these duplicates in class bucket and for each perform
truncate record #23:3
Then repeat rebuild index *
- Stop both Nexuses and move blobstore to a new location
docker stop --time=120 nexus-3.48.0
docker stop --time=120 nexus-3.29.2
rm -rf /mnt/nexus-data-new/blobs && mv /mnt/nexus-data/blobs /mnt/nexus-data-new/
- Start Nexus 3.48.0
Nexus has started and authentication to GUI worked. Also browsing and downloading content worked. I had assumed inconsistency in data so i ran Task Repair - Reconcile component database from blob store. This Task stuck in state Running and never finished. Also log for the Task stopped to grow.
So Tasks were broken and data presumably inconsistent but reachable.
Any suggestions how to perform the upgrade?
This looks similar to Failed while upgrading from 3.15 to 3.39 but it’s a year old topic and without answer.