# Failed while upgrading from 3.15 to 3.39 **URL:** https://community.sonatype.com/t/failed-while-upgrading-from-3-15-to-3-39/8741 **Category:** Sonatype Nexus Repository **Created:** [May 29, 2022, 6:55am UTC](https://community.sonatype.com/t/failed-while-upgrading-from-3-15-to-3-39/8741 "2022-05-29T06:55:35Z") **Posts on this page:** 2 **Page:** 1
### Author: ![labeebudeen](https://avatars.discourse-cdn.com/v4/letter/l/f19dbf/32.png) [@labeebudeen](https://community.sonatype.com/u/labeebudeen) #### Post date: [May 29, 2022, 6:55am UTC](https://community.sonatype.com/t/failed-while-upgrading-from-3-15-to-3-39/8741/1 "2022-05-29T06:55:35Z")
```auto - - - - - - - - - - - - - - - - - - - - - - - - - Upgrade failed - - - - - - - - - - - - - - - - - - - - - - - - - 2022-05-29 06:25:12,443+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Failed transition: NEW -> STARTED java.lang.IllegalMonitorStateException: null at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.releaseLock(OOneEntryPerKeyLockManager.java:216) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:493) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:407) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.endAtomicOperation(ODurableComponent.java:113) at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.create(OSBTree.java:156) at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.create(OSBTreeIndexEngine.java:82) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.addIndexEngine(OAbstractPaginatedStorage.java:2048) at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:243) at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:129) at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:40) at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:171) at com.orientechnologies.orient.core.index.OIndexManagerProxy.createIndex(OIndexManagerProxy.java:87) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1900) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1870) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1860) at org.sonatype.nexus.repository.upgrade.ComponentDatabaseUpgrade_1_16.lambda$0(ComponentDatabaseUpgrade_1_16.java:69) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.lambda$0(DatabaseUpgradeSupport.java:53) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.withDatabase(DatabaseUpgradeSupport.java:65) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.withDatabaseAndClass(DatabaseUpgradeSupport.java:51) at org.sonatype.nexus.repository.upgrade.ComponentDatabaseUpgrade_1_16.apply(ComponentDatabaseUpgrade_1_16.java:67) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190) at java.util.ArrayList.forEach(ArrayList.java:1259) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96) at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193) at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57) at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199) at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111) at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319) at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.lang.Thread.run(Thread.java:748) 2022-05-29 06:25:12,443+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusContextListener - Failed to start nexus java.lang.IllegalMonitorStateException: null at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.releaseLock(OOneEntryPerKeyLockManager.java:216) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:493) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:407) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.endAtomicOperation(ODurableComponent.java:113) at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.create(OSBTree.java:156) at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.create(OSBTreeIndexEngine.java:82) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.addIndexEngine(OAbstractPaginatedStorage.java:2048) at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:243) at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:129) at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:40) at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:171) at com.orientechnologies.orient.core.index.OIndexManagerProxy.createIndex(OIndexManagerProxy.java:87) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1900) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1870) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1860) at org.sonatype.nexus.repository.upgrade.ComponentDatabaseUpgrade_1_16.lambda$0(ComponentDatabaseUpgrade_1_16.java:69) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.lambda$0(DatabaseUpgradeSupport.java:53) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.withDatabase(DatabaseUpgradeSupport.java:65) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.withDatabaseAndClass(DatabaseUpgradeSupport.java:51) at org.sonatype.nexus.repository.upgrade.ComponentDatabaseUpgrade_1_16.apply(ComponentDatabaseUpgrade_1_16.java:67) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190) at java.util.ArrayList.forEach(ArrayList.java:1259) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96) at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193) at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57) at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199) at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111) at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319) at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.lang.Thread.run(Thread.java:748) 2022-05-29 06:25:12,443+0000 ERROR [FelixStartLevel] *SYSTEM Felix - Framework listener delivery error. java.lang.IllegalMonitorStateException: null at java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryRelease(ReentrantReadWriteLock.java:371) at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.unlock(ReentrantReadWriteLock.java:1131) at com.orientechnologies.common.concur.lock.OOneEntryPerKeyLockManager.releaseLock(OOneEntryPerKeyLockManager.java:216) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:493) at com.orientechnologies.orient.core.storage.impl.local.paginated.atomicoperations.OAtomicOperationsManager.endAtomicOperation(OAtomicOperationsManager.java:407) at com.orientechnologies.orient.core.storage.impl.local.paginated.base.ODurableComponent.endAtomicOperation(ODurableComponent.java:113) at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.create(OSBTree.java:156) at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.create(OSBTreeIndexEngine.java:82) at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.addIndexEngine(OAbstractPaginatedStorage.java:2048) at com.orientechnologies.orient.core.index.OIndexAbstract.create(OIndexAbstract.java:243) at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:129) at com.orientechnologies.orient.core.index.OIndexOneValue.create(OIndexOneValue.java:40) at com.orientechnologies.orient.core.index.OIndexManagerShared.createIndex(OIndexManagerShared.java:171) at com.orientechnologies.orient.core.index.OIndexManagerProxy.createIndex(OIndexManagerProxy.java:87) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1900) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1870) at com.orientechnologies.orient.core.metadata.schema.OClassImpl.createIndex(OClassImpl.java:1860) at org.sonatype.nexus.repository.upgrade.ComponentDatabaseUpgrade_1_16.lambda$0(ComponentDatabaseUpgrade_1_16.java:69) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.lambda$0(DatabaseUpgradeSupport.java:53) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.withDatabase(DatabaseUpgradeSupport.java:65) at org.sonatype.nexus.orient.DatabaseUpgradeSupport.withDatabaseAndClass(DatabaseUpgradeSupport.java:51) at org.sonatype.nexus.repository.upgrade.ComponentDatabaseUpgrade_1_16.apply(ComponentDatabaseUpgrade_1_16.java:67) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190) at java.util.ArrayList.forEach(ArrayList.java:1259) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131) at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96) at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69) at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39) at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193) at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57) at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199) at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111) at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319) at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597) at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308) at java.lang.Thread.run(Thread.java:748) ``` This is the error we got, it says upgrade failed, any suggestions ? ---
### Author: ![jpasek77](https://avatars.discourse-cdn.com/v4/letter/j/82dd89/32.png) [@jpasek77](https://community.sonatype.com/u/jpasek77) #### Post date: [March 11, 2023, 7:51am UTC](https://community.sonatype.com/t/failed-while-upgrading-from-3-15-to-3-39/8741/2 "2023-03-11T07:51:28Z")
Hi, I am trying to upgrade Nexus from version 3.29.2 to 3.48.0. For deployment of Nexus we use Official Docker container (originally [3.29.2](https://hub.docker.com/layers/sonatype/nexus3/3.29.2/images/sha256-bfdf2e80545db7db74c028fa06ecbf7bbc00a59c0c0591f9a39fc186cc943b10?context=explore); target version [3.48.0](https://hub.docker.com/layers/sonatype/nexus3/3.48.0/images/sha256-c7fdba87d4003871b06e2fa0c4e9cc2ba39a7d154eeb90998f97ee94fa3dddb7?context=explore)). Environment i test it on is: OS: Ubuntu 22.04.2 LTS (Jammy Jellyfish) Docker: 23.0.1 Used DB: Embedded OrientDB Procedure used for upgrade (based on [this](https://help.sonatype.com/repomanager3/installation-and-upgrades/upgrading-a-standalone-instance)): 1. Stop and remove current container using command ```auto docker stop --time=120 nexus | xargs docker rm ``` 1. Run new container with the same persistent data using command ```auto docker run --name=nexus --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.48.0 ``` Then the startup of a container fails with exit code 255. Complete output of `docker logs nexus` is [here](https://gist.github.com/yacchip/9fc7fc788920719eb73677ef0cce3267). Seems like problem with index creation in component database according to this line ```auto 2023-03-09 13:24:16,103+0000 ERROR [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.index.OIndexUnique - $ANSI{green {db=component}} Exception during index 'asset_bucket_rid_idx' creation ``` ### First i tried to upgrade gradually, i.e. to apply same procedure mentioned above to upgrade e.g. from 3.29.2 to 3.30.0. Here follows matrix of versions i tried to upgrade to (from 3.29.2) and corresponding errors that have arised. | Nexus version | Exception text | | --- | --- | | 3.30.0 | Exception during index ‘deleted\_blob\_index\_blobstore\_blob\_id\_idx’ creation | | 3.30.1 | Exception during index ‘deleted\_blob\_index\_blobstore\_blob\_id\_idx’ creation | | 3.33.1 | Exception during index ‘deleted\_blob\_index\_blobstore\_blob\_id\_idx’ creation | | 3.37.3 | Exception during index ‘coordinate\_download\_count\_repository\_name\_namespace\_name\_version\_date\_ip\_address\_username\_idx’ creation | | 3.42.0 | Exception during index ‘asset\_bucket\_rid\_idx’ creation | | 3.48.0 | Exception during index ‘asset\_bucket\_rid\_idx’ creation | As next step i tried to use [OrientDB Console](https://support.sonatype.com/hc/en-us/articles/115002930827-Accessing-the-OrientDB-Console) to migrate records from component DB and then to build indexes manually while running both Nexus versions simultaneously. The config and security databases were migrated whole. Procedure i used is following: 1. Start Nexus 3.29.2 with all the data ```auto docker run --name=nexus-3.29.2 --volume /mnt/nexus-data:/nexus-data -d -p 8081:8081 sonatype/nexus3:3.29.2 ``` 1. Perform DB exports from ODB Console ```auto docker exec -it nexus-3.29.2 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar' connect plocal:/nexus-data/db/component admin admin export database component -excludeAll -includeCluster="asset bucket component" disconnect connect plocal:/nexus-data/db/config admin admin export database config disconnect connect plocal:/nexus-data/db/security admin admin export database security disconnect ``` 1. Copy exports to new location ```auto mkdir -p /mnt/nexus-data-new/console chown -R 200:200 /mnt/nexus-data-new cp /mnt/nexus-data/console/{component,config,security}.json.gz /mnt/nexus-data-new/console/ ``` 1. Start clean Nexus 3.48.0 with no data ```auto docker run --name=nexus-3.48.0 --volume /mnt/nexus-data-new:/nexus-data -d -p 8881:8081 sonatype/nexus3:3.48.0 ``` 1. Perform DB imports from ODB Console ```auto docker exec -it nexus-3.48.0 bash -c 'cd /nexus-data/console && java -jar /opt/sonatype/nexus/lib/support/nexus-orient-console.jar' connect plocal:/nexus-data/db/component admin admin import database component.json.gz -merge=true disconnect ``` During import a following error occured but i decided to ignore it for this time as it was only test run and most of the data were imported. ```auto Error on database import happened just before line 0, column 178093048 com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids' at com.orientechnologies.orient.core.db.tool.ODatabaseImport.importDatabase(ODatabaseImport.java:190) at com.orientechnologies.orient.console.OConsoleDatabaseApp.importDatabase(OConsoleDatabaseApp.java:2214) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.orientechnologies.common.console.OConsoleApplication.execute(OConsoleApplication.java:405) at com.orientechnologies.common.console.OConsoleApplication.executeCommands(OConsoleApplication.java:260) at com.orientechnologies.common.console.OConsoleApplication.run(OConsoleApplication.java:131) at com.orientechnologies.orient.console.OConsoleDatabaseApp.main(OConsoleDatabaseApp.java:145) at org.sonatype.nexus.orient.console.Main.main(Main.java:63) Error: com.orientechnologies.orient.core.db.tool.ODatabaseExportException: Error on importing database 'component' from file: component-clusters.json.gz Error: com.orientechnologies.orient.core.db.tool.ODatabaseImportException: Invalid format. Found unsupported tag 'brokenRids' ``` Then import of security and config databases. ```auto connect plocal:/nexus-data/db/security admin admin import database security disconnect connect plocal:/nexus-data/db/config admin admin export database config disconnect ``` 1. Rebuild indexes in component db ```auto connect plocal:/nexus-data/db/component admin admin rebuild index * ``` During this operation following error occured ```auto Error: com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #23:10: found duplicated key 'maven-central' in index 'bucket_repository_name_idx' previously assigned to the record #23:3 DB name="component" INDEX=bucket_repository_name_idx RID=#23:3 ``` To fix this it was neccessary to locate all these duplicates in class bucket and for each perform ```auto truncate record #23:3 ``` Then repeat `rebuild index *` 1. Stop both Nexuses and move blobstore to a new location ```auto docker stop --time=120 nexus-3.48.0 docker stop --time=120 nexus-3.29.2 rm -rf /mnt/nexus-data-new/blobs && mv /mnt/nexus-data/blobs /mnt/nexus-data-new/ ``` 1. Start Nexus 3.48.0 Nexus has started and authentication to GUI worked. Also browsing and downloading content worked. I had assumed inconsistency in data so i ran Task [Repair - Reconcile component database from blob store](https://help.sonatype.com/repomanager3/nexus-repository-administration/tasks#Tasks-Repair-Reconcilecomponentdatabasefromblobstore). This Task stuck in state Running and never finished. Also log for the Task stopped to grow. So Tasks were broken and data presumably inconsistent but reachable. Any suggestions how to perform the upgrade?