Migration fails from OrientDB to H2 on RHEL with nexus 3.70.1

Hi There,

I have a standalone Nexus3 that’s been running fine for years (since the 3.2y times…).
It’s using ZoL for storage on RHEL8.
I’m using haproxy on the same VM for SSL and Nexus is 100% unencrypted (from the point of view of the nexus container).

With the need to transition to either PgSQL or H2 I attempted to migrate to H2 and things did not well.
First of all, I upgraded to 3.70.1, rebuilt the container using Docker and launched it:

git reset --hard
git pull
git checkout 3.70.1
sed -i -e 's/ gzip//g' Dockerfile.rh.ubi.java11
docker build -f Dockerfile.rh.ubi.java11 -t krynn-nexus3

I’ve been using this for years to rebuild the container on RHEL and launch it.

The first thing I noticed is that the UI was still saying “3.70.0-03” even though I had requested a build of 3.70.1.

then I proceeded to follow the instructions here:
https://help.sonatype.com/en/orient-3-70-java-8-or-11.html
and then:
https://help.sonatype.com/en/orient-3-70-java-8-or-11.html#migrating-from-orientdb-to-h2-252166
I took the back, copied it to another location on the filesystem.
I downloaded the database migrator for 3.70.x:
https://download.sonatype.com/nexus/nxrm3-migrator/nexus-db-migrator-3.70.1-03.jar

I then ran the tool on my DB (only 92Mb in size):

# java -version
openjdk version "11.0.24" 2024-07-16 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.24.0.8-2) (build 11.0.24+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.24.0.8-2) (build 11.0.24+8-LTS, mixed mode, sharing)

# ls -l
total 93420
-rw-r--r--. 1  200  200 12276432 Aug 13 15:35 analytics-2024-08-13-13-35-34-3.70.0-03.bak
-rw-r--r--. 1  200  200 26246784 Aug 13 15:35 component-2024-08-13-13-35-34-3.70.0-03.bak
-rw-r--r--. 1  200  200   425381 Aug 13 15:35 config-2024-08-13-13-35-34-3.70.0-03.bak
-rw-r--r--. 1 root root 56574550 Aug  7 18:17 nexus-db-migrator-3.70.1-03.jar
-rw-r--r--. 1  200  200   129178 Aug 13 15:35 security-2024-08-13-13-35-34-3.70.0-03.bak

# du -hs 
92M     .

# df -h .
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/rootdg-lv_var   64G   24G   41G  37% /var
# java -Xmx16G -Xms16G -XX:+UseG1GC -XX:MaxDirectMemorySize=28672M -jar nexus-db-migrator-3.70.1-03.jar --migration_type=h2
16:08:52 [main] WARN  c.s.n.d.migrator.MigratorApplication - Please ensure any Nexus Repository instance has been gracefully shut down before proceeding.
16:08:52 [main] INFO  c.s.n.d.migrator.MigratorApplication - Do you want to continue [y/n]?

This finished with:

16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - Migration job finished at Tue Aug 13 16:09:56 CEST 2024
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - Migration job took 5 seconds to execute
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - 8745 records were processed
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - 7 records were filtered
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - 1 records were skipped
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - 8737 records were migrated
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - Created 'Rebuild repository browse' and 'Rebuild repository search' tasks. They will automatically one-time run after starting your Nexus Repository instance.
16:09:56 [main] INFO  c.s.n.d.m.l.ProvidingJobInfoListener - Migrated the following formats: [APT, COCOAPODS, CONAN, CONDA, DOCKER, GITLFS, GO, HELM, MAVEN2, NPM, NUGET, P2, PYPI, R, RAW, RUBYGEMS, YUM]

it created a small mv.db:

# ls  -l *db
-rw-r--r--. 1 root root 7901184 Aug 13 16:09 nexus.mv.db

I then copied that file to my db directory and edited nexus.properties.

However, when I attempted to build 3.71.0 using Dockerfile.rh.ubi.java17, the build failed with:

STEP 13/22: RUN curl -L ${NEXUS_DOWNLOAD_URL} --output nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz     && echo "${NEXUS_DOWNLOAD_SHA256_HASH} nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz" > nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz.sha256     && sha256sum -c nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz.sha256     && tar -xvf nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz     && rm -f nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz.sha256     && mv nexus-${NEXUS_VERSION} $NEXUS_HOME     && chown -R nexus:nexus ${SONATYPE_WORK}     && mv ${SONATYPE_WORK}/nexus3 ${NEXUS_DATA}     && ln -s ${NEXUS_DATA} ${SONATYPE_WORK}/nexus3
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   138  100   138    0     0    315      0 --:--:-- --:--:-- --:--:--   315
100   287  100   287    0     0    615      0 --:--:-- --:--:-- --:--:--   615
nexus-3.71.0-06-java17-unix.tar.gz: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match
Error: building at STEP "RUN curl -L ${NEXUS_DOWNLOAD_URL} --output nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz     && echo "${NEXUS_DOWNLOAD_SHA256_HASH} nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz" > nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz.sha256     && sha256sum -c nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz.sha256     && tar -xvf nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz     && rm -f nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz nexus-${NEXUS_VERSION}-${JAVA_VERSION}-unix.tar.gz.sha256     && mv nexus-${NEXUS_VERSION} $NEXUS_HOME     && chown -R nexus:nexus ${SONATYPE_WORK}     && mv ${SONATYPE_WORK}/nexus3 ${NEXUS_DATA}     && ln -s ${NEXUS_DATA} ${SONATYPE_WORK}/nexus3": while running runtime: exit status 1

So I went back to 3.70.1 but with java17 which built successfully using Dockerfile.rh.ubi.java17
I then started my Nexus container:

# bash -x ./run.sh 
+ NAME=krynn-nexus3
+ MYIP=10.88.0.121
++ docker ps -aq --filter name=krynn-nexus3
+ export DOCKER_ID=
+ DOCKER_ID=
+ [[ '' != '' ]]
+ docker run -d --privileged --cap-add=NET_ADMIN -p 10.88.0.121:8081:8081 -p 10.88.0.121:18000:18000 -p 10.88.0.121:18001:18001 -p 10.88.0.121:18002:18002 -p 10.88.0.121:18003:18003 -p 10.88.0.121:18004:18004 -p 10.88.0.121:18005:18005 -p 10.88.0.121:18010:18010 -v /nexus-data:/nexus-data --name krynn-nexus3 --ip 10.88.0.121 krynn-nexus3:latest
07a4a0e8ab779cfa1248d2ee3a308073a47a8511e25ef268509d4a6ebd530af6

looking at the start logs, I could see a lot of:

nexus - Creating schema for ...

And then it failed with:

2024-08-13 14:20:32,738+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.datastore.mybatis.MyBatisDataStore - nexus - Creating schema for P2BrowseNodeDAO
2024-08-13 14:20:33,247+0000 WARN  [FelixStartLevel] *SYSTEM Sisu - Problem adding: org.eclipse.sisu.inject.LazyBeanEntry@2c2f683d to: org.sonatype.nexus.datastore.mybatis.MyBatisDataStore$$EnhancerByGuice$$14b20128@33ec698a via: org.sonatype.nexus.datastore.internal.DataStoreManagerImpl$DataAccessMediator@5b8ced3f
org.sonatype.nexus.datastore.api.DataAccessException: 
### Error updating database.  Cause: org.h2.jdbc.JdbcSQLNonTransientException: The database is read only; SQL statement:
--
    -- each format has a browse node table
[...]
2024-08-13 14:20:33,294+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusContextListener - Uptime: 1 minute, 1 second and 188 milliseconds (nexus-oss-edition/3.70.0.03)
2024-08-13 14:20:33,295+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Shutting down
2024-08-13 14:20:33,296+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusLifecycleManager - Stop KERNEL

After that, restarting the container always resulted in the DB trying to upgrade and failing on the same steps so I stopped there and restored from backups.

Any ideas?

I was able to see these errors:

2024-08-13 14:24:54,296+0000 ERROR [FelixStartLevel] *SYSTEM org.sonatype.nexus.extender.NexusContextListener - Failed to start nexus
org.sonatype.nexus.datastore.api.DataAccessException:
### Error querying database.  Cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "NODE_ID" not found; SQL statement:
SELECT node_id FROM node_id WHERE id = 1; [42102-224]
### The error may exist in org/sonatype/nexus/internal/node/datastore/NodeIdDAO.xml
### The error may involve org.sonatype.nexus.internal.node.datastore.NodeIdDAO.get
### The error occurred while executing a query
### SQL: SELECT node_id FROM node_id WHERE id = 1;
### Cause: org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "NODE_ID" not found; SQL statement:
SELECT node_id FROM node_id WHERE id = 1; [42102-224]

Perhaps the migrator was run as a different user than Nexus thus the DB is not writable by the service running Nexus.

I ended up re-creating the Nexus registry from scratch on 3.71.0 using H2, problem solved.