How to recover from 'Attempt to access non-existent blob'

Hello,
in case of disaster (e.g. filesystem failure or accidental files removal) some blobs may be lost.
In this case, the way to find such blobs is running “Repair - Reconcile component database from blob store” task.
In this task log we can see
2019-10-09 13:55:44,715+0000 WARN [quartz-3-thread-18] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob c43b9dcd-fc4c-40df-937a-2aec28c3391f 'com/group/test-artifact/test-artifact-1.1.1.jar'

How to remove all lost components from DB then? Do you have any ready solution / scripted task or I need to delete components manually?

Thanks!

There are two ways there can be inconsistencies in the case of disaster recovery if the blob storage and database were not backed up at exactly the same time. There can be blobs in blob storage that are not in the component database. There can be references to blob from the component database that no longer exist on disk. That message comes from the second type of inconsistency. The task should have removed that bad reference.

It should, but it’s not:

  1. Started container

[volodymyr.soloviov]~ **[]** $ docker run -it --rm -v /Users/volodymyr.soloviov/nexus-data:/nexus-data -p 8081:8081 sonatype/nexus3:3.19.1

  1. Uploaded artifact
  2. Check it’s on FS:
[volodymyr.soloviov]~/nexus-data **[]** $ grep -ir common-io-3.5.1.jar ./

.//blobs/default/content/vol-40/chap-28/8e6f5f6f-4ebe-40c5-8094-6af8c25f2b2d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.md5
.//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar
.//blobs/default/content/vol-28/chap-29/4c98c926-6404-4d79-9bfd-a3475ba2c20f.properties:@BlobStore.blob-name=com/test/common-io/3.5.1/common-io-3.5.1.jar.sha1
  1. Removed blob for test purposes:
    `

[volodymyr.soloviov]~/nexus-data [] $ rm .//blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.*

`

  1. Running task
2019-10-10 08:38:30,236+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Checking integrity of assets in repository 'maven-releases' with blob store 'default'
2019-10-10 08:38:30,248+0000 WARN  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob 80d322d8-5361-4bd1-81f0-a74c81c90b7d (/nexus-data/blobs/default/content/vol-25/chap-39/80d322d8-5361-4bd1-81f0-a74c81c90b7d.properties)
2019-10-10 08:38:30,256+0000 ERROR [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - Blob properties missing for asset 'com/test/common-io/3.5.1/common-io-3.5.1.jar
2019-10-10 08:38:30,292+0000 INFO  [quartz-3-thread-1] *SYSTEM org.sonatype.nexus.blobstore.restore.DefaultIntegrityCheckStrategy - ---- Elapsed time: 53.99 ms, processed: 6, failed integrity check: 1 ----
  1. And it is still available in browse and throws exceptions when I’m trying to access it:
[volodymyr.soloviov]~/nexus-data **[]** $ wget http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
--2019-10-10 11:39:49-- http://localhost:8081/repository/maven-releases/com/test/common-io/3.5.1/common-io-3.5.1.jar
Resolving localhost (localhost)... ::1, 127.0.0.1
Connecting to localhost (localhost)|::1|:8081... connected.
HTTP request sent, awaiting response... 500 javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit
2019-10-10 11:40:12 ERROR 500: javax.servlet.ServletException: org.sonatype.nexus.repository.storage.RetryDeniedException: Exceeded retry limit.

nexus log:

Caused by: org.sonatype.nexus.repository.storage.MissingBlobException: Blob default@5441B406-632F1056-850069C0-87C1A9B5-53EBB0B5:80d322d8-5361-4bd1-81f0-a74c81c90b7d exists in metadata, but is missing from the blobstore

What’s wrong ?
I expected 404, but it throws 500

Thank you!

What I’m trying to achieve:

import org.sonatype.nexus.repository.storage.StorageFacet
import org.sonatype.nexus.repository.Repository
import org.sonatype.nexus.repository.storage.Asset
import org.sonatype.nexus.repository.storage.Bucket;
import org.sonatype.nexus.blobstore.api.Blob;
import org.sonatype.nexus.repository.storage.MissingBlobException

repoName = 'maven-releases'

repo = repository.repositoryManager.get(repoName)
StorageFacet storageFacet = repo.facet(StorageFacet)
def tx = storageFacet.txSupplier().get()
try {
    tx.begin()
    Bucket bucket = tx.findBucket(repo);
    tx.browseAssets(tx.findBucket(repo)).each { Asset asset ->

        log.info("${asset}")
        def component
        try {
            component = tx.findComponent(asset.componentId());
        } catch (java.lang.NullPointerException ex) {
            log.info("Skipping null-component")
            return
        }

        try {
            Blob blob = tx.requireBlob(asset.requireBlobRef());
        } catch(MissingBlobException e1) {
            log.info("${e1}")
            log.info("Removing component ${component}")
            try {
                tx.deleteComponent(component)
            } catch (java.io.FileNotFoundException ex) {
                log.info("${ex}")
            }
        }
    }
    tx.commit()
} catch (all) {
    log.info("Exception: ${all}")
    all.printStackTrace()
    log.info("Rolling back components removal...")
    tx.rollback()
} finally {
    tx.close()
    log.info("Transaction closed, task finished. Thank you.")
}

Please open an issue in the ‘dev - nexus’ project at https://issues.sonatype.org

1 Like

https://issues.sonatype.org/browse/NEXUS-21543