How to recover from 'Attempt to access non-existent blob'

Hello,
in case of disaster (e.g. filesystem failure or accidental files removal) some blobs may be lost.
In this case, the way to find such blobs is running “Repair - Reconcile component database from blob store” task.
In this task log we can see
2019-10-09 13:55:44,715+0000 WARN [quartz-3-thread-18] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob c43b9dcd-fc4c-40df-937a-2aec28c3391f 'com/group/test-artifact/test-artifact-1.1.1.jar'

How to remove all lost components from DB then? Do you have any ready solution / scripted task or I need to delete components manually?

Thanks!

There are two ways there can be inconsistencies in the case of disaster recovery if the blob storage and database were not backed up at exactly the same time. There can be blobs in blob storage that are not in the component database. There can be references to blob from the component database that no longer exist on disk. That message comes from the second type of inconsistency. The task should have removed that bad reference.

It should, but it’s not:

  1. Started container

[volodymyr.soloviov]~ **[]** $ docker run -it --rm -v /Users/volodymyr.soloviov/nexus-data:/nexus-data -p 8081:8081 sonatype/nexus3:3.19.1

  1. Uploaded artifact
  2. Check it’s on FS:
[volodymyr.soloviov]~/nexus-data **[]** $ grep -ir common-io-3.5.1.jar ./

.//blobs/default/content/vol-40/chap-28/8e6f5f6f-4ebe-40c5-8094-6af8c25f2b2d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.md5
.//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar
.//blobs/default/content/vol-28/chap-29/4c98c926-6404-4d79-9bfd-a3475ba2c20f.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.sha1
  1. Removed blob for test purposes:
    `

[volodymyr.soloviov]~/nexus-data [] $ rm .//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.*

`

  1. Running task
2019-10-10 08:38:30,236+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'maven-releases' with blob store 'default'
2019-10-10 08:38:30,248+0000 WARN  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 80d322d8-5361-4bd1-81f0-a74c81c90b7d (/nexus-data/blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties)
2019-10-10 08:38:30,256+0000 ERROR [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Blob properties missing for asset 'com/test/common-io/3.5.1/common-io-3.5.1.jar
2019-10-10 08:38:30,292+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - ---- Elapsed time: 53.99 ms, processed: 6, failed integrity check: 1 ----
  1. And it is still available in browse and throws exceptions when I’m trying to access it:
[volodymyr.soloviov]~/nexus-data **[]** $ wget http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
--2019-10-10 11:39:49-- http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8081... connected.
HTTP request sent, awaiting response... 500 javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
2019-10-10 11:40:12 ERROR 500: javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit.

nexus log:

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@5441B406-632F1056-850069C0-87C1A9B5-53EBB0B5:80d322d8-5361-4bd1-81f0-a74c81c90b7d exists in metadata, but is missing from the blobstore

What’s wrong ?
I expected 404, but it throws 500

Thank you!

What I’m trying to achieve:

import org.sonatype.nexus.repository.storage.StorageFacet
import org.sonatype.nexus.repository.Repository
import org.sonatype.nexus.repository.storage.Asset
import org.sonatype.nexus.repository.storage.Bucket;
import org.sonatype.nexus.blobstore.api.Blob;
import org.sonatype.nexus.repository.storage.MissingBlobException

repoName = 'maven-releases'

repo = repository.repositoryManager.get(repoName)
StorageFacet storageFacet = repo.facet(StorageFacet)
def tx = storageFacet.txSupplier().get()
try {
    tx.begin()
    Bucket bucket = tx.findBucket(repo);
    tx.browseAssets(tx.findBucket(repo)).each { Asset asset ->

        log.info("${asset}")
        def component
        try {
            component = tx.findComponent(asset.componentId());
        } catch (java.lang.NullPointerException ex) {
            log.info("Skipping null-component")
            return
        }

        try {
            Blob blob = tx.requireBlob(asset.requireBlobRef());
        } catch(MissingBlobException e1) {
            log.info("${e1}")
            log.info("Removing component ${component}")
            try {
                tx.deleteComponent(component)
            } catch (java.io.FileNotFoundException ex) {
                log.info("${ex}")
            }
        }
    }
    tx.commit()
} catch (all) {
    log.info("Exception: ${all}")
    all.printStackTrace()
    log.info("Rolling back components removal...")
    tx.rollback()
} finally {
    tx.close()
    log.info("Transaction closed, task finished. Thank you.")
}

Please open an issue in the ‘dev - nexus’ project at https://issues.sonatype.org

1 Like

https://issues.sonatype.org/browse/NEXUS-21543

Hi, I have the same problem here. I can’t access the issue.

We had a hardware problem and some files were corrupted. We were able to repair the base and the blobs using the following procedure:
Task “Repair - Reconcile component database from blob store”: it failed at first so we did a rebuild:

java -jar /opt/nexus-3.20.1-01/lib/support/nexus-orient-console.jar

CONNECT PLOCAL:/opt/sonatype-work/nexus3/db/component admin admin
REBUILD INDEX *
REPAIR DATABASE --fix-graph
REPAIR DATABASE --fix-links
REPAIR DATABASE --fix-ridbags
REPAIR DATABASE --fix-bonsai
DISCONNECT

There were still some errors. Then we made an export of the database and a drop :

CONNECT plocal:/opt/sonatype-work/nexus3/db/component admin admin

export database component-export
drop database
create database plocal:/opt/sonatype-work/nexus3/db/component
import database component-export.json.gz

Then we imported the database. It worked and then we restarted Nexus and the “Repair - Reconcile component database from blob store” task this time successfully.

But since then we still have errors on some files :

2020-02-04 16:42:11,126+0100 WARN  [qtp1807541379-2816]  user org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /nexus/repository/SRPMS/test/maven-metadata.xml
org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob test@A77A05EE-7F3C1806-9047BAF8-82339CCF-C53A48C4:ac7becf2-beb1-4be1-a2f3-017ad6c0bbab exists in metadata, but is missing from the blobstore

How can we clean the blob metadata ?

Thanks

Hi Did you find a way to cleanup your metadata ?