Task "Compact Blob Store" does not reclaim disk space

Sonatype Nexus Repository
1

We have a large maven-snapshots repository with over 3 TB of reclaimable bytes, but nothing is deleted from disk when we run the “Compact blob store” task.

Output from the nx-blob-repo-space-report script (https://support.sonatype.com/hc/en-us/article_attachments/6138506837267/nx-blob-repo-space-report-20220510.groovy):

"maven-snapshots": {
	"reclaimableBytes": 3381780825366,
	"totalBytes": 3464385015422
}

Task log:

2023-02-03 14:55:21,685+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task information:
2023-02-03 14:55:21,688+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  ID: e39ab0a2-cd8d-4594-b21a-bedbe47b00b1
2023-02-03 14:55:21,688+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Type: blobstore.compact
2023-02-03 14:55:21,688+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Name: Compact blob store (delete artifacts from disk)
2023-02-03 14:55:21,688+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask -  Description: Compacting default blob store
2023-02-03 14:55:21,688+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Begin deleted blobs processing
2023-02-03 14:55:34,998+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Removed 0 empty directories from /nexus-data/blobs/default/content/directpath
2023-02-03 14:55:34,999+0100 INFO  [quartz-11-thread-19]  *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task complete

Any known causes for the task to not find these artifacts? Are there other ways of reclaiming bytes than running the “Compact blob store” task?

Thanks in advance!
Best regards,
Marius

2

I have this problem too… I made a copy of the server and deleted every repository that references that blob store. Even when there are no references (the blob store could be deleted) the compact blob store tasks recovers nothing and the blob store stays with 165000 blobs and 740GB storage. This is going to be a big problem if it doesn’t recover unused space … we can’t just keep adding disk endlessly.

3

Try this fix: https://support.sonatype.com/hc/en-us/articles/360025325653-How-to-rebuild-the-Admin-Compact-blobstore-deletions-index-file-if-soft-deleted-blobs-are-orphaned

4

Thank you Rich. The solution described in the link worked like charm:

Locate the file based blobstore containing the soft-deleted blobs which were not deleted by the Admin - Compact blobstore task.

Add the following line to the metadata.properties file in the root directory of the blobstore.

rebuildDeletedBlobIndex=true

For example, the default file based blobstore metadata properties file is relative to the install dir at …/sonatype-work/nexus3/blobs/default/metadata.properties

The next time that the Admin - Compact Blobstore task is run, it will iterate over all of the properties files in the blobstore, looking for the deleted=true attribute and rebuild the index file.

5

Thanks Rich, yes, that also worked for me. I have the feeling that the problem is caused by the Maven cleanup tasks that we use to remove snapshots after 30 days.

Shouldn’t this cleanup mode be the default “compact blob store” task behaviour if the standard cleanup is missing large chunks of storage? Or at least an additional option? Having to manually intervene in the filesystem isn’t ideal for such a fundamental cleanup action.

6

Sorry to resurrect an old post, but I am facing this problem with the blobstore used for docker images. I have followed all the steps (cleanup/compact), restarted nexus, ran a repair - recalculate blob store size, but still the blob store is running out of space (~50GB free). Is there anything else to try?

7

Have you checked out the cleanup page on the help site - Cleanup Policies

There are additional steps you may need to take to ensure unreferenced layers are removed.