Upgrade from 3.30.0-01 to 3.34.1 fails

Sonatype Nexus Repository
1

Hi,

Just tried upgrading from 3.30 to 3.34.1. But that unfortunately fails. I was happy it rolled back, so we are still using 3.30. I tried looking up the DockerUpgrade_1_1 class/lambda/function but couldn’t find it on Github, maybe it is a propriety piece. Who could give me some more insight what could possible be wrong?

Thanks

System specs:
OS: CentOS 7 (fully updated)
Java:
openjdk version “1.8.0_302”
OpenJDK Runtime Environment (build 1.8.0_302-b08)
OpenJDK 64-Bit Server VM (build 25.302-b08, mixed mode)
Nexus: 3.34.1

Stack trace

Scanning docker repository, my-docker-repo, for misconfigured assets021-10-05 13:33:55,373+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Upgrade docker from 1.0 to 1.1
2021-10-05 13:33:55,406+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1 - Scanning docker repository, my-docker-repo, for misconfigured assets
2021-10-05 13:33:55,406+0200 WARN [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Problem upgrading docker from 1.0 to 1.1
java.lang.NullPointerException: null
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.lambda$0(DockerUpgrade_1_1.java:87)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.apply(DockerUpgrade_1_1.java:85)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl$$EnhancerByGuice$$293734767.GUICE$TRAMPOLINE(<generated>)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:74)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:75)
at com.google.inject.internal.InterceptorStackCallback.invoke(InterceptorStackCallback.java:55)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl$$EnhancerByGuice$$293734767.start(<generated>)
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:748)
2021-10-05 13:33:55,410+0200 WARN [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl -
- - - - - - - - - - - - - - - - - - - - - - - - -
Rollback upgrade
- - - - - - - - - - - - - - - - - - - - - - - - -
2021-10-05 13:33:55,411+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Rolling back component
2021-10-05 13:35:04,843+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Storage 'plocal:/data/nexus/db/component' is opened under OrientDB distribution : 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
2021-10-05 13:35:04,856+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Rolling back security
2021-10-05 13:35:05,208+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.storage.impl.local.paginated.OLocalPaginatedStorage - Storage 'plocal:/data/nexus/db/security' is opened under OrientDB distribution : 2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x)
2021-10-05 13:35:05,222+0200 WARN [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl -
- - - - - - - - - - - - - - - - - - - - - - - - -
Upgrade failed
- - - - - - - - - - - - - - - - - - - - - - - - -
2021-10-05 13:35:05,223+0200 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl - Failed transition: NEW -> STARTED
java.lang.NullPointerException: null
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.lambda$0(DockerUpgrade_1_1.java:87)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.apply(DockerUpgrade_1_1.java:85)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:748)
2021-10-05 13:35:05,224+0200 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusContextListener - Failed to start nexus
java.lang.NullPointerException: null
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.lambda$0(DockerUpgrade_1_1.java:87)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.apply(DockerUpgrade_1_1.java:85)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:748)
2021-10-05 13:35:05,226+0200 ERROR [FelixStartLevel] *SYSTEM Felix - Framework listener delivery error.
java.lang.NullPointerException: null
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.lambda$0(DockerUpgrade_1_1.java:87)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.repository.docker.internal.orient.upgrade.DockerUpgrade_1_1.apply(DockerUpgrade_1_1.java:85)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.lambda$3(UpgradeServiceImpl.java:190)
at java.util.ArrayList.forEach(ArrayList.java:1259)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doUpgrade(UpgradeServiceImpl.java:131)
at org.sonatype.nexus.upgrade.internal.orient.UpgradeServiceImpl.doStart(UpgradeServiceImpl.java:96)
at org.sonatype.nexus.common.stateguard.StateGuardLifecycleSupport.start(StateGuardLifecycleSupport.java:69)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$TransitionImpl.run(StateGuard.java:193)
at org.sonatype.nexus.common.stateguard.TransitionsInterceptor.invoke(TransitionsInterceptor.java:57)
at org.sonatype.nexus.extender.NexusLifecycleManager.startComponent(NexusLifecycleManager.java:199)
at org.sonatype.nexus.extender.NexusLifecycleManager.to(NexusLifecycleManager.java:111)
at org.sonatype.nexus.extender.NexusContextListener.moveToPhase(NexusContextListener.java:319)
at org.sonatype.nexus.extender.NexusContextListener.frameworkEvent(NexusContextListener.java:216)
at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1597)
at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:308)
at java.lang.Thread.run(Thread.java:748)
2021-10-05 13:35:05,271+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusContextListener - Uptime: 2 minutes, 57 seconds and 234 milliseconds (nexus-oss-edition/3.34.1.01)
2021-10-05 13:35:05,271+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Shutting down
2021-10-05 13:35:05,272+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Stop RESTORE
2021-10-05 13:35:05,272+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Stop STORAGE
2021-10-05 13:35:05,279+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl$1 - OrientDB Server is shutting down...
2021-10-05 13:35:05,280+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl$1 - Shutting down protocols
2021-10-05 13:35:05,280+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.plugin.OServerPluginManager - Shutting down plugins:
2021-10-05 13:35:05,280+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.server.plugin.OServerPluginManager - - jmx
2021-10-05 13:35:05,281+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl$1 - OrientDB Server shutdown complete
2021-10-05 13:35:05,281+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.Orient - Orient Engine is shutting down...
2021-10-05 13:35:05,312+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.Orient - - shutdown storage: component...
2021-10-05 13:35:05,942+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.Orient - - shutdown storage: security...
2021-10-05 13:35:06,421+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.Orient - - shutdown storage: config...
2021-10-05 13:35:06,680+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.Orient - - shutdown storage: OSystem...
2021-10-05 13:35:07,764+0200 INFO [FelixStartLevel] *SYSTEM com.orientechnologies.orient.core.Orient - OrientDB Engine shutdown complete
2021-10-05 13:35:07,807+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.internal.orient.DatabaseServerImpl - Shutdown
2021-10-05 13:35:07,809+0200 INFO [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Stop KERNEL
2

The stacktrace suggests your database is missing an index - https://github.com/sonatype/nexus-public/blob/master/components/nexus-repository/src/main/java/org/sonatype/nexus/repository/storage/BucketEntityAdapter.java#L66-L69

You should be able to manually connect to the database and create the index

1 Like
3

Hi thanks for the quick reply!

I just ran the tasktype “Rebuild repository index” and then ran the upgrade sequence again.

Then I tried your more to the point solution “manually connect to the database and create the index”, of which I’m not sure I did correctly (see below).

Result is unfortunately still the same. Exact same stacktrace

java -jar /opt/nexus/nexus-3/lib/support/nexus-orient-console.jar

OrientDB console v.2.2.36 (build d3beb772c02098ceaea89779a7afd4b7305d3788, branch 2.2.x) https://www.orientdb.com
Type 'help' to display all the supported commands.
orientdb> connect plocal:/data/nexus/db/component admin admin

Connecting to database [plocal:/data/nexus/db/component] with user 'admin'...OK
orientdb {db=component}> repair database --fix-links

Repairing database...
- Removing broken links...
-- Done! Fixed links: 0, modified documents: 0
Repair database complete (0 errors)
orientdb {db=component}> rebuild index *

Rebuilding index(es)...
Rebuilt index(es). Found 3867112 link(s) in 202.158997 sec(s).
Index(es) rebuilt successfully
orientdb {db=component}>
4

Additionally I tried rebuilding the component database all together (no err msg’s). But again… to no avail. Still same stacktrace

`export database component-export`
`drop database`
`create database plocal:/data/nexus/db/component`
`import` `database component-export.json.gz`
5

I managed it to narrow down concerning more tight versions.
Just upgraded succesfully to 3.33.1-01
then when I try to upgrade to 3.34.0-01 the problem starts appearing

6

The problem is that Nexus only creates the index when the database is created, and its missing entirely from Orient so repairing doesn’t work.

This is the create index docs for Orient - Create Index · OrientDB Manual

7

Hello,
I’m going to plan the upgrade of our nexus from 3.20.0 to 3.34.1 . I’m following the link Upgrading , but I think it’s poor in details .
Can I preserve the actual “sonatype-work” dir without change ?
Is there a more detailed guide?
I was thinking of making a tar of all directories and activating vmware snaps for backup.
Thanks
MM

8

Hi,

Thanks for the reply. However I do not know what to make of that, I can’t guess which indexes are needed by Nexus. So this remains a little bit of black box magic to me. Nothing can be found on github repo as well, so did some further research and only found this post: Restoring Component db lost all indexes, Nexus oss 3.22.0 - #3 by my.ravisoni
Which shows what indexes should be there (which is for version 3.22, so could be different from 3.33. But taking a little leap of faith here)

Then I found a command to show existing indexes (so that I can compare those 2 lists): select expand(indexes) from metadata:indexmanager

After comparison I found that following indexes are missing (which I will create later on today)

create index bucket_repository_name_idx on bucket (repository_name collate default) UNIQUE
create index component_bucket_name_version_idx on component (bucket, name, version) NOTUNIQUE
create index component_ci_name_ci_idx on component (ci_name collate ci) NOTUNIQUE
create index docker_foreign_layers_digest_idx on docker_foreign_layers (digest collate default) UNIQUE

And showed me these unmentioned indexes (should I remove them? For now I’m going to leave these be. Being too afraid of wrecking my data)

  • deleted_blob_index
  • ___exportImportRIDMap
  • dictionary

Going to try to upgrade later today, and will post the outcome here.

9

I linked the index required earlier.

If you’re looking for more direct help you should consider Nexus Repository Pro which comes with support.

10

Hi Matthew. Thanks for your patience, I kind if misread there a bit. Just marked your first answer as the solution. Upgrade went fine now!

11

So was it one of the four indexes that you had created manually? Or which index did you have to create?

I’m looking at an upgrade to 3.36 from 3.30.1 in the near future.

12

The user’s problem was corruption in their database, its not something that is part of standard upgrades. If you have an issue I suggest making your own topic.

13

Thanks, it did fix my upgrade issue.
Dbs seems to be prone to corruption…