How to recover from 'Attempt to access non-existent blob'

vsoloviov · October 9, 2019, 2:35pm

Hello,
in case of disaster (e.g. filesystem failure or accidental files removal) some blobs may be lost.
In this case, the way to find such blobs is running “Repair - Reconcile component database from blob store” task.
In this task log we can see
2019-10-09 13:55:44,715+0000 WARN [quartz-3-thread-18] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob c43b9dcd-fc4c-40df-937a-2aec28c3391f 'com/group/test-artifact/test-artifact-1.1.1.jar'

How to remove all lost components from DB then? Do you have any ready solution / scripted task or I need to delete components manually?

Thanks!

rseddon · October 9, 2019, 3:07pm

There are two ways there can be inconsistencies in the case of disaster recovery if the blob storage and database were not backed up at exactly the same time. There can be blobs in blob storage that are not in the component database. There can be references to blob from the component database that no longer exist on disk. That message comes from the second type of inconsistency. The task should have removed that bad reference.

vsoloviov · October 10, 2019, 8:49am

It should, but it’s not:

Started container

[volodymyr.soloviov]~ **[]** $ docker run -it --rm -v /Users/volodymyr.soloviov/nexus-data:/nexus-data -p 8081:8081 sonatype/nexus3:3.19.1

Uploaded artifact
Check it’s on FS:

[volodymyr.soloviov]~/nexus-data **[]** $ grep -ir common-io-3.5.1.jar ./

.//blobs/default/content/vol-40/chap-28/8e6f5f6f-4ebe-40c5-8094-6af8c25f2b2d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.md5
.//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar
.//blobs/default/content/vol-28/chap-29/4c98c926-6404-4d79-9bfd-a3475ba2c20f.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.sha1

Removed blob for test purposes:
`

[volodymyr.soloviov]~/nexus-data [] $ rm .//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.*

`

Running task

2019-10-10 08:38:30,236+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'maven-releases' with blob store 'default'
2019-10-10 08:38:30,248+0000 WARN  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 80d322d8-5361-4bd1-81f0-a74c81c90b7d (/nexus-data/blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties)
2019-10-10 08:38:30,256+0000 ERROR [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Blob properties missing for asset 'com/test/common-io/3.5.1/common-io-3.5.1.jar
2019-10-10 08:38:30,292+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - ---- Elapsed time: 53.99 ms, processed: 6, failed integrity check: 1 ----

And it is still available in browse and throws exceptions when I’m trying to access it:

[volodymyr.soloviov]~/nexus-data **[]** $ wget http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
--2019-10-10 11:39:49-- http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8081... connected.
HTTP request sent, awaiting response... 500 javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
2019-10-10 11:40:12 ERROR 500: javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit.

nexus log:

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@5441B406-632F1056-850069C0-87C1A9B5-53EBB0B5:80d322d8-5361-4bd1-81f0-a74c81c90b7d exists in metadata, but is missing from the blobstore

What’s wrong ?
I expected 404, but it throws 500

Thank you!

vsoloviov · October 10, 2019, 10:43am

What I’m trying to achieve:

import org.sonatype.nexus.repository.storage.StorageFacet
import org.sonatype.nexus.repository.Repository
import org.sonatype.nexus.repository.storage.Asset
import org.sonatype.nexus.repository.storage.Bucket;
import org.sonatype.nexus.blobstore.api.Blob;
import org.sonatype.nexus.repository.storage.MissingBlobException

repoName = 'maven-releases'

repo = repository.repositoryManager.get(repoName)
StorageFacet storageFacet = repo.facet(StorageFacet)
def tx = storageFacet.txSupplier().get()
try {
    tx.begin()
    Bucket bucket = tx.findBucket(repo);
    tx.browseAssets(tx.findBucket(repo)).each { Asset asset ->

        log.info("${asset}")
        def component
        try {
            component = tx.findComponent(asset.componentId());
        } catch (java.lang.NullPointerException ex) {
            log.info("Skipping null-component")
            return
        }

        try {
            Blob blob = tx.requireBlob(asset.requireBlobRef());
        } catch(MissingBlobException e1) {
            log.info("${e1}")
            log.info("Removing component ${component}")
            try {
                tx.deleteComponent(component)
            } catch (java.io.FileNotFoundException ex) {
                log.info("${ex}")
            }
        }
    }
    tx.commit()
} catch (all) {
    log.info("Exception: ${all}")
    all.printStackTrace()
    log.info("Rolling back components removal...")
    tx.rollback()
} finally {
    tx.close()
    log.info("Transaction closed, task finished. Thank you.")
}

rseddon · October 10, 2019, 2:15pm

Please open an issue in the ‘dev - nexus’ project at https://issues.sonatype.org

vsoloviov · October 10, 2019, 3:51pm

https://issues.sonatype.org/browse/NEXUS-21543

sysmso · February 4, 2020, 4:05pm

Hi, I have the same problem here. I can’t access the issue.

We had a hardware problem and some files were corrupted. We were able to repair the base and the blobs using the following procedure:
Task “Repair - Reconcile component database from blob store”: it failed at first so we did a rebuild:

java -jar /opt/nexus-3.20.1-01/lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/opt/sonatype-work/nexus3/db/component admin admin
REBUILD INDEX *
REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
DISCONNECT

There were still some errors. Then we made an export of the database and a drop :

CONNECT plocal:/opt/sonatype-work/nexus3/db/component admin admin

export database component-export
drop database
create database plocal:/opt/sonatype-work/nexus3/db/component
import database component-export.json.gz

Then we imported the database. It worked and then we restarted Nexus and the “Repair - Reconcile component database from blob store” task this time successfully.

But since then we still have errors on some files :

2020-02-04 16:42:11,126+0100 WARN  [qtp1807541379-2816]  user org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /nexus/repository/SRPMS/test/maven-metadata.xml
org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob test@A77A05EE-7F3C1806-9047BAF8-82339CCF-C53A48C4:ac7becf2-beb1-4be1-a2f3-017ad6c0bbab exists in metadata, but is missing from the blobstore

How can we clean the blob metadata ?

Thanks

bhanupathak · February 26, 2020, 2:18pm

Hi Did you find a way to cleanup your metadata ?

nyludaya · July 8, 2020, 8:39am

Getting the same exception. Did we find a solution for this?

pete.panula · October 19, 2020, 6:43pm

Also facing the same when running “Repair - Rebuild Maven repository metada” task. Any hints???

2020-10-19 20:43:45,123+0300 WARN [quartz-3-thread-20] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 4815834d-4247-476a-9646-cf205e75c8aa : ‘Repair - Rebuild Maven repository metada’ [repository.maven.r
ebuild-metadata] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task ‘Rebuilding Maven Metadata of snapshots’; 1 failure
at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:86)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:145)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:108)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:143)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Suppressed: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
at org.sonatype.nexus.repository.storage.StorageTxImpl.allowRetry(StorageTxImpl.java:203)
at sun.reflect.GeneratedMethodAccessor267.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:69)
at com.sun.proxy.$Proxy202.allowRetry(Unknown Source)
at org.sonatype.nexus.transaction.BatchTransaction.allowRetry(BatchTransaction.java:53)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:81)
at org.sonatype.nexus.transaction.Operations.transactional(Operations.java:200)
at org.sonatype.nexus.transaction.Operations.call(Operations.java:146)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadataInner(MetadataRebuilder.java:476)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.rebuildMetadata(MetadataRebuilder.java:408)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder.rebuild(MetadataRebuilder.java:126)
at org.sonatype.nexus.repository.maven.internal.hosted.MavenHostedFacetImpl.rebuildMetadata(MavenHostedFacetImpl.java:111)
at org.sonatype.nexus.repository.maven.tasks.RebuildMaven2MetadataTask.execute(RebuildMaven2MetadataTask.java:60)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:75)
… 13 common frames omitted
Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@D073C097-B81A65AE-6D2C7714-3BAB0A64-4DC5596E:e7c86eff-3860-4dd2-b7a1-be04d9bb6b01 exists in metadata, but is missing from the blobstore
at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:931)
at sun.reflect.GeneratedMethodAccessor135.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
at com.sun.proxy.$Proxy202.requireBlob(Unknown Source)
at org.sonatype.nexus.repository.maven.internal.MavenFacetImpl.get(MavenFacetImpl.java:193)
at org.sonatype.nexus.transaction.TransactionInterceptor.invoke(TransactionInterceptor.java:45)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.mayUpdateChecksum(MetadataRebuilder.java:546)
at org.sonatype.nexus.repository.maven.internal.hosted.metadata.MetadataRebuilder$Worker.lambda$3(MetadataRebuilder.java:510)
at org.sonatype.nexus.transaction.OperationPoint.proceed(OperationPoint.java:64)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:56)
… 21 common frames omitted
2020-10-19 20:43:45,123+0300 INFO [quartz-3-thread-20] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task ‘Repair - Rebuild Maven repository metada’ [repository.maven.rebuild-metadata] state change RUNNING -> WAITI:

mpiggott · October 19, 2020, 7:03pm

A blob is missing but referenced from the database

pete.panula · October 19, 2020, 7:13pm

Yes I see , but how I can recover from that?

jmmirand · October 28, 2020, 12:48pm

Hi everyone.

Has anyone been able to solve this problem.

We get the same error with npm repositories and we have tried to apply the solution of the reconciliation task from the BlobStore and repair the indexes, but in both cases the result has been negative.

We are considering deleting the repo without remove repo and launching reconciliation.

Has anyone done this test?

Sincerely.

jaiganesh1991 · November 8, 2021, 11:23am

We run into the same issue when we had to resotre the Nexus database from a backup which is not the latest. This had led to the corruption of multiple snapshot and release repositories which had uploaded newer versions to nexus and which is not recorded in the Nexus DB backup.
We deleted the mavenmetada.xml files of the affects components and uploaded two versions.
One to re-create the maven-metadata.xml and the second to repopulate the maven-metadata.xml with the history of all the verions available.

arvindbraju · May 2, 2022, 2:10am

Hello ,

We ran into a similar issue of - “Attempt to access non-existent blob” for docker images. We got the complete nexus instance restored from backup. However, with the new restored instance, we could see db related issue - may be db and blob were out of sink. We restored the db from the previous instance and this fixed the db issue. However, we are still facing Attempt to access non-existent blob issue.
Copying the missing blobs did not resolve the issue.

Below is the log excerpt.

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@5C626408-557C9F39-DADD033A-223C5F2A-8436241A:41cda9d1-6a73-4fbd-b116-995198db13af exists in metadata, but is missing from the blobstore
at org.sonatype.nexus.repository.storage.StorageTxImpl.requireBlob(StorageTxImpl.java:988)
at sun.reflect.GeneratedMethodAccessor237.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.sonatype.nexus.common.stateguard.SimpleMethodInvocation.proceed(SimpleMethodInvocation.java:53)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:53)
at org.sonatype.nexus.common.stateguard.StateGuardAspect$1.invoke(StateGuardAspect.java:63)
at com.sun.proxy.$Proxy214.requireBlob(Unknown Source)
at org.sonatype.nexus.repository.pypi.internal.orient.OrientPyPiHostedFacetImpl.getIndex(OrientPyPiHostedFacetImpl.java:176)
at org.sonatype.nexus.transaction.TransactionalWrapper.proceedWithTransaction(TransactionalWrapper.java:57)
… 166 common frames omitted
2022-05-02 07:07:41,569+0530 WARN [qtp903999980-1571] mvn_deploy org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 7047f7b4-dca0-464c-bf18-14f4c057ee8c (/Opensource/Nexus_restored/sonatype-work/nexus3/blobs/default/content/vol-11/chap-35/7047f7b4-dca0-464c-bf18-14f4c057ee8c.properties)
2022-05-02 07:07:41,628+0530 WARN [qtp903999980-1449] jiodigitalapi org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob c3430ba7-44a1-4c65-afec-b8c6607aeae2 (/Opensource/Nexus_restored/sonatype-work/nexus3/blobs/default/content/vol-17/chap-39/c3430ba7-44a1-4c65-afec-b8c6607aeae2.properties)
2022-05-02 07:07:42,331+0530 WARN [qtp903999980-1449] jiodigitalapi org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob c3430ba7-44a1-4c65-afec-b8c6607aeae2 (/Opensource/Nexus_restored/sonatype-work/nexus3/blobs/default/content/vol-17/chap-39/c3430ba7-44a1-4c65-afec-b8c6607aeae2.properties)
2022-05-02 07:07:42,501+0530 WARN [qtp903999980-1564] mvn_deploy org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 641711a3-980d-43e2-9727-5eb78a6233c5 (/Opensource/Nexus_restored/sonatype-work/nexus3/blobs/default/content/vol-16/chap-36/641711a3-980d-43e2-9727-5eb78a6233c5.properties)
2022-05-02 07:07:42,672+0530 WARN [qtp903999980-1571] mvn_deploy org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 7047f7b4-dca0-464c-bf18-14f4c057ee8c (/Opensource/Nexus_restored/sonatype-work/nexus3/blobs/default/content/vol-11/chap-35/7047f7b4-dca0-464c-bf18-14f4c057ee8c.properties)
2022-05-02 07:07:42,743+0530 WARN [qtp903999980-1548] jioai-pypi org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob d8818481-4b0b-40e7-9ac5-17e088d15ddd (/Opensource/Nexus_restored/sonatype-work/nexus3/blobs/default/content/vol-04/chap-02/d8818481-4b0b-40e7-9ac5-17e088d15ddd.properties)

sundeep.techie · July 16, 2023, 11:31pm

Is there a solution to this. We are also seeing this error. When we browse the repository in nexus-ui, everything is visible (possibly since metadata is present). But when clicking on any link to download, it fails (timeout). and nexus.log has same errors as above.

When I go to s3, the content is present. How does then nexus say it is deleted from blobstore?