Problem with nexus server migration to new hardware

Sonatype Nexus Repository
1

Hello
I’m using OSS 3.12.0-01 and trying to migrate from old server (aix 6.1) to new server (CentOS Linux).

I’ve copied all my data folders (whole sonatype-work) on new hardware while old server was stopped, but
every time I’m trying to start new server there are problems with orientdb:

*2019-04-10 08:20:49,907+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Magic number verification failed for page ‘0’ of ‘internal.pcl’.

I’ve tried remove all old orientdb and restore them from backup as written in documentation but errors with magic number still exists:

*2019-04-10 08:20:46,125+0200 INFO [FelixStartLevel] SYSTEM ROOT - Storage ‘plocal:/nexus/sonatype-work/nexus3/db/config’ is created under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
*2019-04-10 08:20:46,134+0200 INFO [ForkJoinPool.commonPool-worker-2] SYSTEM ROOT - Storage ‘plocal:/nexus/sonatype-work/nexus3/db/security’ is created under OrientDB distribution : 2.2.34 (build f340442755a31eabc91b87cb3ef99eda5cee6ebd, branch 2.2.x)
*2019-04-10 08:20:49,606+0200 INFO [FelixStartLevel] SYSTEM org.sonatype.nexus.orient.internal.DatabaseRestorerImpl - restoration of database config from file /nexus/sonatype-work/nexus3/restore-from-backup/config-2019-04-07-10-31-52-3.12.0-01.bak starting
*2019-04-10 08:20:49,606+0200 INFO [ForkJoinPool.commonPool-worker-1] SYSTEM org.sonatype.nexus.orient.internal.DatabaseRestorerImpl - restoration of database analytics from file /nexus/sonatype-work/nexus3/restore-from-backup/analytics-2019-04-07-10-31-52-3.12.0-01.bak starting
*2019-04-10 08:20:49,606+0200 INFO [ForkJoinPool.commonPool-worker-3] SYSTEM org.sonatype.nexus.orient.internal.DatabaseRestorerImpl - restoration of database accesslog from file /nexus/sonatype-work/nexus3/restore-from-backup/accesslog-2019-04-07-10-31-52-3.12.0-01.bak starting
*2019-04-10 08:20:49,854+0200 WARN [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Page 0 is broken in WAL segment 0 and will be truncated
*2019-04-10 08:20:49,855+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} 1 pages in WAL segment analytics.31.wal are broken and will be truncated, some data will be lost after restore.
*2019-04-10 08:20:49,875+0200 WARN [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Storage ‘analytics’ was not closed properly. Will try to recover from write ahead log…
*2019-04-10 08:20:49,876+0200 WARN [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Restore is not possible because write ahead log is empty.
*2019-04-10 08:20:49,901+0200 INFO [ForkJoinPool.commonPool-worker-2] SYSTEM org.sonatype.nexus.orient.internal.DatabaseRestorerImpl - restoration of database security from file /nexus/sonatype-work/nexus3/restore-from-backup/security-2019-04-07-10-31-52-3.12.0-01.bak starting
*2019-04-10 08:20:49,906+0200 INFO [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Storage data recover was completed
*2019-04-10 08:20:49,907+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Magic number verification failed for page ‘0’ of ‘internal.pcl’.
*2019-04-10 08:20:49,908+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Magic number verification failed for page ‘0’ of ‘index.pcl’.
*2019-04-10 08:20:49,909+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Magic number verification failed for page ‘0’ of ‘manindex.pcl’.
*2019-04-10 08:20:49,909+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Magic number verification failed for page ‘0’ of ‘default.pcl’.
*2019-04-10 08:20:49,909+0200 ERROR [ForkJoinPool.commonPool-worker-1] SYSTEM ROOT - $ANSI{green {db=analytics}} Magic number verification failed for page ‘0’ of ‘orole.pcl’.

I though that something is wrong with my Linux machine, so I’ve tried set up nexus server on windows pc. Unfortunately the same error with magic number as soon as restore complete.

Any advices ?

2

The database looks to be corrupt. Does it boot on the original system? If so, I suggest trying the copy again, something went wrong during this attempt.

Rich

3

It is boot on original system. There were some problems on the original system (unexpected shutdown, database in ready only mode, but I’ve fixed this with rebuilding indexes), but right now original system works, but there are some errors from time to time about rolling back transactions.

I’ve tried copy this several times, using different tools (rsync, scp), the same result. Moreover the same magic number error appears from copying all nexus directories, or restoring database from backup (original system complete backup with success). I’ve even tried to export/import database using orientdb console. The same.

Am I missing something obvious here ? I’s moving nexus between different systems (unix, linux) allowed ?

4

I’ve tried remove all old orientdb and restore them from backup as written in documentation but errors with magic number still exists:

There is no need to do that. Just copy the work directory while Nexus is not running. See here for more information:

5

Hope it’s ok to dig up and old issue, but I had the same problem and managed to get it working.
I am trying to copy our Nexus 3.34.1 from AIX 7.2 to RedHat 7.

I copied everything and when trying to start Nexus I got the same issues with the OrientDB.
The approach with backing up the OrientDB from the scheduled task and then restoring them from backup on RedHat didn’t work either. The same magic number errors, caused by the difference in hardware probably.

The solution that worked for me was to export the OrientDB databases with the orientdb console as JSON files. Or did you try that option as well?
Not sure if you need to do it for all databases or just component, config and security… I did it for 3 others as well.

Export the databases as JSON files via the orientdb console, replacing /app/nexus with your own path to the sonatype-work directory.
CONNECT PLOCAL:/app/nexus/sonatype-work/nexus3/db/accesslog admin admin
EXPORT DATABASE accesslog-export
DISCONNECT
CONNECT PLOCAL:/app/nexus/sonatype-work/nexus3/db/analytics admin admin
EXPORT DATABASE analytics-export
DISCONNECT
CONNECT PLOCAL:/app/nexus/sonatype-work/nexus3/db/audit admin admin
EXPORT DATABASE audit-export
DISCONNECT
CONNECT PLOCAL:/app/nexus/sonatype-work/nexus3/db/component admin admin
EXPORT DATABASE component-export
DISCONNECT
CONNECT PLOCAL:/app/nexus/sonatype-work/nexus3/db/config admin admin
EXPORT DATABASE config-export
DISCONNECT
CONNECT PLOCAL:/app/nexus/sonatype-work/nexus3/db/security admin admin
EXPORT DATABASE security-export
DISCONNECT

Copy the entire installation to the new Linux platform.
Now on the new Linux platform.

Remove these directories from the copy:

  • sonatype-work/nexus3/db/accesslog
  • sonatype-work/nexus3/db/analytics
  • sonatype-work/nexus3/db/audit
  • sonatype-work/nexus3/db/component
  • sonatype-work/nexus3/db/config
  • sonatype-work/nexus3/db/security
    Import the gzipped JSON files via the orientdb console to create new databases.
    It needs the path to the json.gz file, I started my orientdb console session from the same path where I placed the backup files.

CREATE DATABASE PLOCAL:/app/nexus/sonatype-work/nexus3/db/accesslog
IMPORT DATABASE accesslog-export.json.gz
DISCONNECT
CREATE DATABASE PLOCAL:/app/nexus/sonatype-work/nexus3/db/analytics
IMPORT DATABASE analytics-export.json.gz
DISCONNECT
CREATE DATABASE PLOCAL:/app/nexus/sonatype-work/nexus3/db/audit
IMPORT DATABASE audit-export.json.gz
DISCONNECT
CREATE DATABASE PLOCAL:/app/nexus/sonatype-work/nexus3/db/component
IMPORT DATABASE component-export.json.gz
DISCONNECT
CREATE DATABASE PLOCAL:/app/nexus/sonatype-work/nexus3/db/config
IMPORT DATABASE config-export.json.gz
DISCONNECT
CREATE DATABASE PLOCAL:/app/nexus/sonatype-work/nexus3/db/security
IMPORT DATABASE security-export.json.gz
DISCONNECT

After that I was able to start the copied Nexus without any problems.