How to recover from 'Attempt to access non-existent blob'

Hello,
in case of disaster (e.g. filesystem failure or accidental files removal) some blobs may be lost.
In this case, the way to find such blobs is running “Repair - Reconcile component database from blob store” task.
In this task log we can see
2019-10-09 13:55:44,715+0000 WARN [quartz-3-thread-18] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob c43b9dcd-fc4c-40df-937a-2aec28c3391f 'com/group/test-artifact/test-artifact-1.1.1.jar'

How to remove all lost components from DB then? Do you have any ready solution / scripted task or I need to delete components manually?

Thanks!

There are two ways there can be inconsistencies in the case of disaster recovery if the blob storage and database were not backed up at exactly the same time. There can be blobs in blob storage that are not in the component database. There can be references to blob from the component database that no longer exist on disk. That message comes from the second type of inconsistency. The task should have removed that bad reference.

It should, but it’s not:

  1. Started container

[volodymyr.soloviov]~ **[]** $ docker run -it --rm -v /Users/volodymyr.soloviov/nexus-data:/nexus-data -p 8081:8081 sonatype/nexus3:3.19.1

  1. Uploaded artifact
  2. Check it’s on FS:
[volodymyr.soloviov]~/nexus-data **[]** $ grep -ir common-io-3.5.1.jar ./

.//blobs/default/content/vol-40/chap-28/8e6f5f6f-4ebe-40c5-8094-6af8c25f2b2d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.md5
.//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar
.//blobs/default/content/vol-28/chap-29/4c98c926-6404-4d79-9bfd-a3475ba2c20f.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.sha1
  1. Removed blob for test purposes:
    `

[volodymyr.soloviov]~/nexus-data [] $ rm .//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.*

`

  1. Running task
2019-10-10 08:38:30,236+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'maven-releases' with blob store 'default'
2019-10-10 08:38:30,248+0000 WARN  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 80d322d8-5361-4bd1-81f0-a74c81c90b7d (/nexus-data/blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties)
2019-10-10 08:38:30,256+0000 ERROR [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Blob properties missing for asset 'com/test/common-io/3.5.1/common-io-3.5.1.jar
2019-10-10 08:38:30,292+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - ---- Elapsed time: 53.99 ms, processed: 6, failed integrity check: 1 ----
  1. And it is still available in browse and throws exceptions when I’m trying to access it:
[volodymyr.soloviov]~/nexus-data **[]** $ wget http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
--2019-10-10 11:39:49-- http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8081... connected.
HTTP request sent, awaiting response... 500 javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
2019-10-10 11:40:12 ERROR 500: javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit.

nexus log:

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@5441B406-632F1056-850069C0-87C1A9B5-53EBB0B5:80d322d8-5361-4bd1-81f0-a74c81c90b7d exists in metadata, but is missing from the blobstore

What’s wrong ?
I expected 404, but it throws 500

Thank you!

What I’m trying to achieve:

import org.sonatype.nexus.repository.storage.StorageFacet
import org.sonatype.nexus.repository.Repository
import org.sonatype.nexus.repository.storage.Asset
import org.sonatype.nexus.repository.storage.Bucket;
import org.sonatype.nexus.blobstore.api.Blob;
import org.sonatype.nexus.repository.storage.MissingBlobException

repoName = 'maven-releases'

repo = repository.repositoryManager.get(repoName)
StorageFacet storageFacet = repo.facet(StorageFacet)
def tx = storageFacet.txSupplier().get()
try {
    tx.begin()
    Bucket bucket = tx.findBucket(repo);
    tx.browseAssets(tx.findBucket(repo)).each { Asset asset ->

        log.info("${asset}")
        def component
        try {
            component = tx.findComponent(asset.componentId());
        } catch (java.lang.NullPointerException ex) {
            log.info("Skipping null-component")
            return
        }

        try {
            Blob blob = tx.requireBlob(asset.requireBlobRef());
        } catch(MissingBlobException e1) {
            log.info("${e1}")
            log.info("Removing component ${component}")
            try {
                tx.deleteComponent(component)
            } catch (java.io.FileNotFoundException ex) {
                log.info("${ex}")
            }
        }
    }
    tx.commit()
} catch (all) {
    log.info("Exception: ${all}")
    all.printStackTrace()
    log.info("Rolling back components removal...")
    tx.rollback()
} finally {
    tx.close()
    log.info("Transaction closed, task finished. Thank you.")
}

Please open an issue in the ‘dev - nexus’ project at https://issues.sonatype.org

1 Like

https://issues.sonatype.org/browse/NEXUS-21543

Hi, I have the same problem here. I can’t access the issue.

We had a hardware problem and some files were corrupted. We were able to repair the base and the blobs using the following procedure:
Task “Repair - Reconcile component database from blob store”: it failed at first so we did a rebuild:

java -jar /opt/nexus-3.20.1-01/lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/opt/sonatype-work/nexus3/db/component admin admin
REBUILD INDEX *
REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
DISCONNECT

There were still some errors. Then we made an export of the database and a drop :

CONNECT plocal:/opt/sonatype-work/nexus3/db/component admin admin

export database component-export
drop database
create database plocal:/opt/sonatype-work/nexus3/db/component
import database component-export.json.gz

Then we imported the database. It worked and then we restarted Nexus and the “Repair - Reconcile component database from blob store” task this time successfully.

But since then we still have errors on some files :

2020-02-04 16:42:11,126+0100 WARN  [qtp1807541379-2816]  user org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /nexus/repository/SRPMS/test/maven-metadata.xml
org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob test@A77A05EE-7F3C1806-9047BAF8-82339CCF-C53A48C4:ac7becf2-beb1-4be1-a2f3-017ad6c0bbab exists in metadata, but is missing from the blobstore

How can we clean the blob metadata ?

Thanks

Hi Did you find a way to cleanup your metadata ?

Getting the same exception. Did we find a solution for this?

Also facing the same when running “Repair - Rebuild Maven repository metada” task. Any hints???

2020-10-19 20:43:45,123+0300 WARN [quartz-3-thread-20] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 4815834d-4247-476a-9646-cf205e75c8aa : ‘Repair - Rebuild Maven repository metada’ [repository.maven.r
ebuild-metadata] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task ‘Rebuilding Maven Metadata of snapshots’; 1 failure
at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:86)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Suppressed: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:203)
at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy202.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.BatchTransaction.allowRetry(BatchTransaction.java:53)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadataInner(MetadataRebuilder.java:476)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:408)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:126)
at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:111)
at org.sonatype.nexus.repository.maven.tasks.RebuildMaven2MetadataTask.execute(RebuildMaven2MetadataTask.java:60)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:75)
… 13 common frames omitted
Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@D073C097-B81A65AE-6D2C7714-3BAB0A64-4DC5596E:e7c86eff-3860-4dd2-b7a1-be04d9bb6b01 exists in metadata, but is missing from the blobstore
at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:931)
at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
at com.sun.proxy.$Proxy202.requireBlob(Unknown Source)
at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.get(MavenFacetImpl.java:193)
at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:45)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.mayUpdateChecksum(MetadataRebuilder.java:546)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.lambda$3(MetadataRebuilder.java:510)
at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
… 21 common frames omitted
2020-10-19 20:43:45,123+0300 INFO [quartz-3-thread-20] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task ‘Repair - Rebuild Maven repository metada’ [repository.maven.rebuild-metadata] state change RUNNING -> WAITI:

A blob is missing but referenced from the database

Yes I see , but how I can recover from that?

Hi everyone.

Has anyone been able to solve this problem.

We get the same error with npm repositories and we have tried to apply the solution of the reconciliation task from the BlobStore and repair the indexes, but in both cases the result has been negative.

We are considering deleting the repo without remove repo and launching reconciliation.

Has anyone done this test?

Sincerely.