Failed task "Delete unused manifests" due to "invalid asset"?

Sonatype Nexus Repository
,
1

Does anyone face this issue? my Task " Delete unused manifests and images" does not work anymore because of one invalid asset. I assume the side effect is a never ending growing blob storage.

I am on the most recent version from September 3.72.0 with a H2 database.

My question to the community, does anyone have experience how to deal with this? I can’t even find this asset, by searching by sha XY.

I also tried this with no success - the main task “delete unused manifests” still fails after.

  • Repair - Reconcile component database from blob store
  • Repair - Recalculate blob store storage
  • Rebuilded the index of the blob storage

Happy for any hint :slight_smile:

This is a part of the stack trace

2024-10-04 12:25:59,752+0000 ERROR [quartz-9-thread-10] *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask - Failed to run task ‘Docker - Delete unused manifests and images’ on repository ‘docker-hosted’
org.sonatype.nexus.repository.InvalidContentException: Invalid asset: AssetData{assetId=77478, path=‘/v2/-/blobs/sha256:098cb24285eb179125c9118cab8b75f60edb80c3f0b23a160a4d79d8f8fa91a3’, kind=‘BLOB’, componentId=null, component=null, assetBlobId=86034, assetBlob=AssetBlobData{assetBlobId=86034, blobRef=default@5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7, blobSize=246020316, contentType=‘application/vnd.docker.image.rootfs.diff.tar.gzip’, checksums={sha256=f68cd13cef79a280aba70afec3d67d8b1b6aad51d76246dd76148f3a4703b93d, sha1=2b25fb351262cd31a0251b881e340136436efb06}, blobCreated=2024-09-20T23:56:23.034Z, createdBy=‘anonymous’, createdByIp=‘172.20.22.4’}, lastDownloaded=null, assetSize=246020316} AbstractRepositoryContent{repositoryId=1, attributes=NestedAttributesMap{parent=null, key=‘attributes’, backing={docker={}}}, created=2024-09-20T23:55:44.098466Z, lastUpdated=2024-09-20T23:56:23.036568Z}
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.lambda$5(DockerGCFacetImpl.java:267)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.findUnusedAssets(DockerGCFacetImpl.java:265)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:206)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:131)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:114)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:287)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
at org.sonatype.nexus.repository.docker.tasks.DockerGCTask.execute(DockerGCTask.java:41)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:86)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-04 12:25:59,754+0000 WARN [quartz-9-thread-10] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 7d77ab62-7591-46a0-a51d-2c0a63a31e55 : ‘Docker - Delete unused manifests’ [repository.docker.gc] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task ‘Docker - Delete unused manifests and images’; 1 failure
at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:97)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)

2

It appears the asset is missing required docker attributes.

3

Thank you for your reply.
Indeed - all attributes are null. Any idea how I can get ridge of it? (still only assuming this might be the root cause of the not running task)

4

So far, I could not find the related “problematic asset” and assign it to a specific blob on the file system. Now I run the job Repair - Reconcile componen database from blob store

I hoped the job would help an solve the issue. It didn’t my initial job Delete unused manifests and images still dont work.

But

The reconcile job printed the blob ID on the file system. Now my big question. Is it safe to delete those blobs on the file system level?

Output from the reconcile job:
*SYSTEM com.sonatype.nexus.blobstore.restore.internal.datastore.DockerRestoreBlobStrategy - Skipping as asset already exists, blob store: default, repository: docker-hosted, path: /v2/-/blobs/sha256:098cb24285eb179125c9118cab8b75f60edb80c3f0b23a160a4d79d8f8fa91a3, blob name: /v2/-/blobs/sha256:098cb24285eb179125c9118cab8b75f60edb80c3f0b23a160a4d79d8f8fa91a3, blob id: 5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7

With the blob ID I was able to find them:

find ./* -name "*5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7*"
./blobs/default/content/vol-13/chap-30/5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7.properties
./blobs/default/content/vol-13/chap-30/5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7.bytes

Any opinions? Or did someone already tried it?

5

So for all who asks themselves if we can delete blobs from the file system (I am not the only one but no one posted his experience)

I deleted them this morning.

The good news first was after I deleted two failing assets by their blob ID’s I run “Repair - Reconcile component database from blob store” and “Docker - Delete unused manifests and images” than “Admin - Cleanup unused asset blobs” and "Admin - compact blob store" I was successful!

The error about invalid asset in the job “Docker - Delete unused manifests and images” doesn’t appear anymore and freed up 500 GB disk space again!

So my initial issue is solved. But -

Now its my integrity check job which fails!

2024-10-09 06:20:01,804+0000 WARN  [quartz-9-thread-11]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access non-existent blob attributes file /nexus-data/blobs/default/content/vol-10/chap-20/d12ff8aa-d102-4c9b-b008-ee280d13128c.properties for blob d12ff8aa-d102-4c9b-b008-ee280d13128c
2024-10-09 06:20:01,804+0000 ERROR [quartz-9-thread-11]  *SYSTEM org.sonatype.nexus.blobstore.restore.datastore.DefaultIntegrityCheckStrategy - Blob properties missing for asset '/v2/-/blobs/sha256:e1d5934e5a806db3b0f5dd17bd2d3cf43d855b1d99cdd0b7733b78ea9a1ab5a8'.
2024-10-09 06:20:01,804+0000 INFO  [quartz-9-thread-11]  *SYSTEM org.sonatype.nexus.blobstore.restore.datastore.RestoreMetadataTask - Removing asset /v2/-/blobs/sha256:e1d5934e5a806db3b0f5dd17bd2d3cf43d855b1d99cdd0b7733b78ea9a1ab5a8 from repository docker-hosted, blob integrity check failed
2024-10-09 06:20:13,088+0000 INFO  [quartz-9-thread-11]  *SYSTEM org.sonatype.nexus.blobstore.restore.datastore.DefaultIntegrityCheckStrategy - Elapsed time: 21s, processed: 29311, failed integrity check: 1

In short words, my disk space is back, but now I have the integrity check job which fails. Any idea how I can solve this?

Thank you

6

At the moment that issue isn’t fixable by Nexus, but you may be able to do something manually.

When blobs are written they have a file with the extension bytes for the content, and a properties file which contains metadata about the file.

The log indicates which file and the associated asset, you could attempt to recreate the properties file using another as an example utilizing information Nexus will provide about the asset in the UI.

7

Thanks Matthew for your reply.
I understand what you mean, an interesting idea.

So I picked one of the inconsistent blobs, but whether the .bytes nor the .properties file is present on the file system.

Then I observed the following about the compact DB job:

On the summary on at the end, the WARN (Attempt to access non-existent blob attributes file) and ERROR (Blob properties missing for asset) points on every run to other Assets (or files)

Run on 2024-10-14
org.sonatype.nexus.blobstore.restore.datastore.DefaultIntegrityCheckStrategy - Elapsed time: 1m 12s, processed: 50241, failed integrity check: 9

Run on 2024-10-15
org.sonatype.nexus.blobstore.restore.datastore.DefaultIntegrityCheckStrategy - Elapsed time: 24s, processed: 48504, failed integrity check: 12

Presuming, there are a lot going (pushing and deleting docker files) is this result reasonable?
I am just asking because on my last post the compact DB job execution it printed my blob, I deleted by hand, so I was afraid I messed up everything.
Is the result of the compact DB job normal during normal day to day operations?

Would be interesting if yes - I can close this case. If no I guess I need to reach out the support.

I would be thankful for a last hint.

Regards
Daniel