N3: blob store corruption

PetrH · December 6, 2022, 2:20am

Hello,
we’re running Nexus 3.43.0 in a VM that recently had a storage issue.
After getting it back to work some users reported strange issues when getting artifacts, so we decided to run the Nexus repair tasks.

We run all of these:

Repair - Rebuild npm metadata (on all repositories)
Repair - Rebuild repository browse (on all affected repositories)
Repair - Rebuild repository search (on all affected repositories)
Repair - Reconcile component database from blob store (on the affected blob store)
Repair - Reconcile date metadata from blob store
Repair - Reconcile npm /-/v1/search metadata (on all repositories)

The “Rebuild npm metadata” task failed and few others produced many exceptions so we even run them all twice to verify that everything is cleared up in the end.

However the task Rebuild npm metadata still fails with the following exceptions:

2022-12-06 02:35:19,034+0100 WARN  [quartz-13-thread-17]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob b717b506-25b4-4da6-b77a-c37a50124261 attributes: /sonatype-work/nexus3/blobs/npm/content/vol-01/chap-45/b717b506-25b4-4da6-b77a-c37a50124261.properties {deletedDateTime=1670288876079, deleted=true, @BlobStore.created-by=deployment, creationTime=1499591283882, @BlobStore.content-type=application/json, sha1=fdcde583a9fc192bf32e4fc0faa15c32c1cdbf4e, @BlobStore.blob-name=core-ingredients, deletedReason=Updating asset AttachedEntityId{asset->#18:67849}, @Bucket.repo-name=npm-internal, size=19274}
2022-12-06 02:35:19,055+0100 ERROR [quartz-13-thread-17]  *SYSTEM com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask - Failed to run task 'Rebuilding npm metadata of *' on repository 'npm-internal'
org.sonatype.nexus.repository.storage.MissingBlobException: Blob npm@D5048240-C5940CCF-CD744B03-6E311437-47305B4A:b717b506-25b4-4da6-b77a-c37a50124261 exists in metadata, but is missing from the blobstore
        at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:1067)
        at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
        at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
        at com.sun.proxy.$Proxy225.requireBlob(Unknown Source)
        at com.sonatype.nexus.repository.npm.internal.orient.NpmFacetUtils.loadPackageRoot(NpmFacetUtils.java:358)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.rebuildMetadataAsset(OrientRebuildNpmMetadataTask.java:77)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.lambda$1(OrientRebuildNpmMetadataTask.java:62)
        at com.google.common.collect.Iterables$4.lambda$forEach$0(Iterables.java:568)
        at com.google.common.collect.Iterables$5.lambda$forEach$0(Iterables.java:704)
        at java.lang.Iterable.forEach(Iterable.java:75)
        at com.google.common.collect.Iterables$5.forEach(Iterables.java:704)
        at com.google.common.collect.Iterables$4.forEach(Iterables.java:565)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.rebuildMetadata(OrientRebuildNpmMetadataTask.java:62)
        at com.sonatype.nexus.repository.npm.internal.tasks.BaseRebuildNpmMetadataTask.execute(BaseRebuildNpmMetadataTask.java:45)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.lambda$0(OrientRebuildNpmMetadataTask.java:53)
        at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
        at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
        at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
        at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
        at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
        at org.sonatype.nexus.transaction.Operations.run(Operations.java:175)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.execute(OrientRebuildNpmMetadataTask.java:53)
        at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:79)
        at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
2022-12-06 02:35:19,056+0100 WARN  [quartz-13-thread-17]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 9798373a-030e-4404-bebe-f8186cfff868 : 'Manual - Repair - Rebuild npm metadata - ALL' [repository.npm.rebuild-metadata] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task 'Rebuilding npm metadata of *'; 1 failure
        at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
        at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:90)
        at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)
        Suppressed: org.sonatype.nexus.repository.storage.MissingBlobException: Blob npm@D5048240-C5940CCF-CD744B03-6E311437-47305B4A:b717b506-25b4-4da6-b77a-c37a50124261 exists in metadata, but is missing from the blobstore
                at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:1067)
                at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
                at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
                at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
                at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
                at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
                at com.sun.proxy.$Proxy225.requireBlob(Unknown Source)
                at com.sonatype.nexus.repository.npm.internal.orient.NpmFacetUtils.loadPackageRoot(NpmFacetUtils.java:358)
                at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.rebuildMetadataAsset(OrientRebuildNpmMetadataTask.java:77)
                at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.lambda$1(OrientRebuildNpmMetadataTask.java:62)
                at com.google.common.collect.Iterables$4.lambda$forEach$0(Iterables.java:568)
                at com.google.common.collect.Iterables$5.lambda$forEach$0(Iterables.java:704)
                at java.lang.Iterable.forEach(Iterable.java:75)
                at com.google.common.collect.Iterables$5.forEach(Iterables.java:704)
                at com.google.common.collect.Iterables$4.forEach(Iterables.java:565)
                at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.rebuildMetadata(OrientRebuildNpmMetadataTask.java:62)
                at com.sonatype.nexus.repository.npm.internal.tasks.BaseRebuildNpmMetadataTask.execute(BaseRebuildNpmMetadataTask.java:45)
                at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.lambda$0(OrientRebuildNpmMetadataTask.java:53)
                at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
                at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
                at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
                at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
                at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
                at org.sonatype.nexus.transaction.Operations.run(Operations.java:175)
                at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientRebuildNpmMetadataTask.execute(OrientRebuildNpmMetadataTask.java:53)
                at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:79)
                ... 13 common frames omitted
2022-12-06 02:35:19,056+0100 INFO  [quartz-13-thread-17]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Manual - Repair - Rebuild npm metadata - ALL' [repository.npm.rebuild-metadata] state change RUNNING -> WAITING (FAILED)

And while the task Repair - Reconcile npm /-/v1/search metadata doesn’t fail, it still produces exceptions such as:

2022-12-06 02:30:51,454+0100 WARN  [quartz-13-thread-17]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob attributes file /sonatype-work/nexus3/blobs/npm/content/vol-41/chap-04/a853a613-a8f8-4697-bb47-95e714429b30.properties for blob a853a613-a8f8-4697-bb47-95e714429b30
2022-12-06 02:30:51,455+0100 ERROR [quartz-13-thread-17]  *SYSTEM com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientReindexNpmRepositoryTask - Error occurred while reindexing npm asset rbx-web/-/rbx-web-2.2.14-rc4.tgz in repository npm-internal, will not be searchable
org.sonatype.nexus.repository.storage.MissingBlobException: Blob npm@D5048240-C5940CCF-CD744B03-6E311437-47305B4A:a853a613-a8f8-4697-bb47-95e714429b30 exists in metadata, but is missing from the blobstore
        at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:1067)
        at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
        at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
        at com.sun.proxy.$Proxy225.requireBlob(Unknown Source)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientReindexNpmRepositoryTask.maybeUpdateAsset(OrientReindexNpmRepositoryTask.java:149)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientReindexNpmRepositoryTask.updateAssets(OrientReindexNpmRepositoryTask.java:133)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientReindexNpmRepositoryTask.lambda$1(OrientReindexNpmRepositoryTask.java:111)
        at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
        at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
        at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
        at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
        at org.sonatype.nexus.transaction.Operations.call(Operations.java:166)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientReindexNpmRepositoryTask.processBatch(OrientReindexNpmRepositoryTask.java:109)
        at com.sonatype.nexus.repository.npm.internal.tasks.orient.OrientReindexNpmRepositoryTask.execute(OrientReindexNpmRepositoryTask.java:88)
        at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:79)
        at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

and

2022-12-06 02:30:51,595+0100 INFO  [elasticsearch[D5048240-C5940CCF-CD744B03-6E311437-47305B4A][clusterService#updateTask][T#1]]  *SYSTEM org.elasticsearch.cluster.metadata - [D5048240-C5940CCF-CD744B03-6E311437-47305B4A] [9c3fbc7965c4e68e814cc86408e3608f54f497c8] update_mapping [component]
2022-12-06 02:30:51,626+0100 INFO  [elasticsearch[D5048240-C5940CCF-CD744B03-6E311437-47305B4A][clusterService#updateTask][T#1]]  *SYSTEM org.elasticsearch.cluster.metadata - [D5048240-C5940CCF-CD744B03-6E311437-47305B4A] [9c3fbc7965c4e68e814cc86408e3608f54f497c8] update_mapping [component]
2022-12-06 02:30:52,350+0100 WARN  [event-17-thread-1349]  *SYSTEM org.sonatype.nexus.transaction.RetryController - Exceeded retry limit: 8/8 (org.sonatype.nexus.repository.browse.internal.orient.BrowseNodeCollisionException: Node already has an asset
        DB name="component")
2022-12-06 02:30:52,353+0100 WARN  [event-17-thread-1349]  *SYSTEM org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeManager - Problem generating browse nodes for Asset{metadata=AttachedEntityMetadata{schema=asset, document=#17:5010{bucket:#13:8,format:npm,last_updated:Tue Dec 06 02:30:51 CET 2022,attributes:[4],component:#15:3052,name:@core/auth/-/auth-0.0.1.tgz,size:55788,content_type:application/x-tgz,blob_ref:npm@D5048240-C5940CCF-CD744B03-6E311437-47305B4A:a0d8f7d7-18d5-4446-adb6-263f6b2c84e4,last_downloaded:Mon Jul 03 13:14:50 CEST 2017,blob_created:Mon Jul 03 13:14:50 CEST 2017,blob_updated:Mon Jul 03 13:14:50 CEST 2017,created_by:null,created_by_ip:null} v4}, name=@core/auth/-/auth-0.0.1.tgz}
org.sonatype.nexus.repository.browse.internal.orient.BrowseNodeCollisionException: Node already has an asset
        DB name="component"
        at org.sonatype.nexus.repository.browse.internal.orient.BrowseNodeEntityAdapter.createAssetNode(BrowseNodeEntityAdapter.java:308)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeStore.lambda$1(OrientBrowseNodeStore.java:168)
        at org.sonatype.nexus.orient.transaction.OrientOperations.lambda$2(OrientOperations.java:62)
        at org.sonatype.nexus.transaction.OperationPoint.lambda$0(OperationPoint.java:53)
        at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
        at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:58)
        at org.sonatype.nexus.transaction.Operations.proceedWithTransaction(Operations.java:232)
        at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:223)
        at org.sonatype.nexus.transaction.Operations.run(Operations.java:175)
        at org.sonatype.nexus.orient.transaction.OrientOperations.run(OrientOperations.java:62)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeStore.createAssetNode(OrientBrowseNodeStore.java:168)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeManager.createBrowseNodes(OrientBrowseNodeManager.java:137)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeManager.createFromAsset(OrientBrowseNodeManager.java:80)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeManager.maybeCreateFromUpdatedAsset(OrientBrowseNodeManager.java:102)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeEventHandler.lambda$1(OrientBrowseNodeEventHandler.java:65)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeEventHandler.handle(OrientBrowseNodeEventHandler.java:90)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientBrowseNodeEventHandler.on(OrientBrowseNodeEventHandler.java:65)
        at sun.reflect.GeneratedMethodAccessor65.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.common.eventbus.Subscriber.invokeSubscriberMethod(Subscriber.java:87)
        at com.google.common.eventbus.Subscriber$1.run(Subscriber.java:72)
        at org.sonatype.nexus.internal.event.AffinityBarrier.lambda$1(AffinityBarrier.java:91)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

And there doesn’t appear to be any way how to fully recover from that.
Is there anything that could be done about it?

mmartz · December 6, 2022, 3:44pm

I think your best bet is going to be to restore a backup. That kind of corruption is going to require manually evaluating each problem until you get them all resolved by comparing the database and blobstore. It’s likely to be extremely complex to figure out.

PetrH · December 7, 2022, 1:28am

OK, I restored from the backup taken prior to the storage issue, however the result wasn’t much different and there were still many issues.

If someone in the future encounters something similar, following are steps that I performed (on the restored backup) in order to recover from that:

With Nexus stopped, launch the OrientDB console, connect to the component database, remove duplicate asset entries from the components database (there were many and they prevented the index asset_bucket_component_name_idx rebuild).
Note: The shown SQL statement deletes the older duplicate entries. If you remove the DESC at the end it will remove the new duplicate entries instead. After checking the data I decided that deleting the older duplicates is the better choice.

/usr/lib/jvm/jre-1.8.0/bin/java -jar /opt/nexus3/lib/support/nexus-orient-console.jar
CONNECT PLOCAL:/sonatype-work/nexus3/db/component admin admin
DELETE FROM asset WHERE @rid NOT IN(SELECT FIRST(LIST(@rid)) FROM asset GROUP BY name,bucket ORDER BY @rid DESC)

Still in the OrientDB console, drop the browse_node table (to invoke its full rebuild and avoid the org.sonatype.nexus.repository.browse.internal.orient.BrowseNodeCollisionException: Node already has an asset exceptions).

DROP CLASS browse_node

Still in the OrientDB console, repair the component database and rebuild its indexes (mainly to verify that things are fine now).

REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
REBUILD INDEX *

Still in the OrientDB console, export, drop and reimport the database in order to start in clean state and with a smallest possible database (export&import is currently the only way how to compact the OrientDB).
Note: In some cases the drop isn’t successful, then you will need to DISCONNECT & EXIT from the OrientDB console, delete the database directory on the filesystem and launch the OrientDB console again. and continue.

EXPORT DATABASE /tmp/nexus3-component
DROP DATABASE
CREATE DATABASE PLOCAL:/sonatype-work/nexus3/db/component admin admin
IMPORT DATABASE /tmp/nexus3-component.json.gz -preserveClusterIDs=true
DISCONNECT EXIT

Start Nexus, let it rebuild the browse nodes (example in the log: Task 'Rebuild repository browse tree - (npm-internal,npm-external,npm-all,documentation,npmjs)' [create.browse.nodes] state change RUNNING -> OK).
Login to the Nexus admin console and manually start the following tasks, one at a time and in the shown order:
Repair - Reconcile component database from blob store (on all blob stores)
Repair - Reconcile npm /-/v1/search metadata (on all repositories)
Repair - Rebuild repository browse (on all repositories)
Repair - Rebuild repository search (on all repositories)
Repair - Rebuild npm metadata (on all repositories)

Honestly I hope that when Nexus 4 comes out one day, it uses a completely different storage backends and not this horrible mess that one has only unnecessary troubles with.
We still use Nexus 2 as our main repository system (we use Nexus 3 only where it’s unavoidable) for this very reason. Maintaining Nexus 2 and dealing with any consequences of crashes or other issues is a breeze when compared to Nexus 3.