freitaucher
(Armand de Sillègue Athos Comte de la Fère)
1
Does anyone face this issue? my Task " Delete unused manifests and images" does not work anymore because of one invalid asset. I assume the side effect is a never ending growing blob storage.
I am on the most recent version from September 3.72.0 with a H2 database.
My question to the community, does anyone have experience how to deal with this? I can’t even find this asset, by searching by sha XY.
I also tried this with no success - the main task “delete unused manifests” still fails after.
Repair - Reconcile component database from blob store
Repair - Recalculate blob store storage
Rebuilded the index of the blob storage
Happy for any hint
This is a part of the stack trace
2024-10-04 12:25:59,752+0000 ERROR [quartz-9-thread-10] *SYSTEM org.sonatype.nexus.repository.docker.tasks.DockerGCTask - Failed to run task ‘Docker - Delete unused manifests and images’ on repository ‘docker-hosted’
org.sonatype.nexus.repository.InvalidContentException: Invalid asset: AssetData{assetId=77478, path=‘/v2/-/blobs/sha256:098cb24285eb179125c9118cab8b75f60edb80c3f0b23a160a4d79d8f8fa91a3’, kind=‘BLOB’, componentId=null, component=null, assetBlobId=86034, assetBlob=AssetBlobData{assetBlobId=86034, blobRef=default@5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7, blobSize=246020316, contentType=‘application/vnd.docker.image.rootfs.diff.tar.gzip’, checksums={sha256=f68cd13cef79a280aba70afec3d67d8b1b6aad51d76246dd76148f3a4703b93d, sha1=2b25fb351262cd31a0251b881e340136436efb06}, blobCreated=2024-09-20T23:56:23.034Z, createdBy=‘anonymous’, createdByIp=‘172.20.22.4’}, lastDownloaded=null, assetSize=246020316} AbstractRepositoryContent{repositoryId=1, attributes=NestedAttributesMap{parent=null, key=‘attributes’, backing={docker={}}}, created=2024-09-20T23:55:44.098466Z, lastUpdated=2024-09-20T23:56:23.036568Z}
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.lambda$5(DockerGCFacetImpl.java:267)
at java.base/java.util.Optional.orElseThrow(Optional.java:403)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.findUnusedAssets(DockerGCFacetImpl.java:265)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.handleV2Assets(DockerGCFacetImpl.java:206)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.processRepository(DockerGCFacetImpl.java:131)
at org.sonatype.nexus.repository.docker.internal.datastore.recipe.DockerGCFacetImpl.deleteUnusedManifestsAndImages(DockerGCFacetImpl.java:114)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:287)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
at org.sonatype.nexus.repository.docker.tasks.DockerGCTask.execute(DockerGCTask.java:41)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:86)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
2024-10-04 12:25:59,754+0000 WARN [quartz-9-thread-10] *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskJob - Task 7d77ab62-7591-46a0-a51d-2c0a63a31e55 : ‘Docker - Delete unused manifests’ [repository.docker.gc] execution failure
org.sonatype.goodies.common.MultipleFailures$MultipleFailuresException: Failed to run task ‘Docker - Delete unused manifests and images’; 1 failure
at org.sonatype.goodies.common.MultipleFailures.maybePropagate(MultipleFailures.java:95)
at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:97)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:105)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
It appears the asset is missing required docker attributes.
freitaucher
(Armand de Sillègue Athos Comte de la Fère)
3
Thank you for your reply.
Indeed - all attributes are null. Any idea how I can get ridge of it? (still only assuming this might be the root cause of the not running task)
freitaucher
(Armand de Sillègue Athos Comte de la Fère)
4
So far, I could not find the related “problematic asset” and assign it to a specific blob on the file system. Now I run the job Repair - Reconcile componen database from blob store
I hoped the job would help an solve the issue. It didn’t my initial job Delete unused manifests and images still dont work.
But
The reconcile job printed the blob ID on the file system. Now my big question. Is it safe to delete those blobs on the file system level?
Output from the reconcile job: *SYSTEM com.sonatype.nexus.blobstore.restore.internal.datastore.DockerRestoreBlobStrategy - Skipping as asset already exists, blob store: default, repository: docker-hosted, path: /v2/-/blobs/sha256:098cb24285eb179125c9118cab8b75f60edb80c3f0b23a160a4d79d8f8fa91a3, blob name: /v2/-/blobs/sha256:098cb24285eb179125c9118cab8b75f60edb80c3f0b23a160a4d79d8f8fa91a3, blob id: 5e1e0482-22ef-4d5c-8ff0-df6d6da2cfc7
freitaucher
(Armand de Sillègue Athos Comte de la Fère)
5
So for all who asks themselves if we can delete blobs from the file system (I am not the only one but no one posted his experience)
I deleted them this morning.
The good news first was after I deleted two failing assets by their blob ID’s I run “Repair - Reconcile component database from blob store” and “Docker - Delete unused manifests and images” than “Admin - Cleanup unused asset blobs” and "Admin - compact blob store" I was successful!
The error about invalid asset in the job “Docker - Delete unused manifests and images” doesn’t appear anymore and freed up 500 GB disk space again!
At the moment that issue isn’t fixable by Nexus, but you may be able to do something manually.
When blobs are written they have a file with the extension bytes for the content, and a properties file which contains metadata about the file.
The log indicates which file and the associated asset, you could attempt to recreate the properties file using another as an example utilizing information Nexus will provide about the asset in the UI.
freitaucher
(Armand de Sillègue Athos Comte de la Fère)
7
Thanks Matthew for your reply.
I understand what you mean, an interesting idea.
So I picked one of the inconsistent blobs, but whether the .bytes nor the .properties file is present on the file system.
Then I observed the following about the compact DB job:
On the summary on at the end, the WARN (Attempt to access non-existent blob attributes file) and ERROR (Blob properties missing for asset) points on every run to other Assets (or files)
Run on 2024-10-14 org.sonatype.nexus.blobstore.restore.datastore.DefaultIntegrityCheckStrategy - Elapsed time: 1m 12s, processed: 50241, failed integrity check: 9
Run on 2024-10-15 org.sonatype.nexus.blobstore.restore.datastore.DefaultIntegrityCheckStrategy - Elapsed time: 24s, processed: 48504, failed integrity check: 12
Presuming, there are a lot going (pushing and deleting docker files) is this result reasonable?
I am just asking because on my last post the compact DB job execution it printed my blob, I deleted by hand, so I was afraid I messed up everything.
Is the result of the compact DB job normal during normal day to day operations?
Would be interesting if yes - I can close this case. If no I guess I need to reach out the support.