Docker upload cleaned up

pit.artamonov · September 5, 2019, 8:49am

Periodically, nexus automatically cleans one or more of the docker-image approximately 3 days after it is added.

How can I find out why he does it? Perhaps there is some way to enable advanced logging of this process.

Nexus 3.18.1-01
log:

2019-09-03 05:18:27,112+0300 WARN  [quartz-3-thread-7]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob path$nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/0 attributes: /mnt/nexus_data/nexus/nexus-data/blobs/default/content/directpath/nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/0.properties {deletedDateTime=1567407472074, deleted=true, @BlobStore.created-by=nexus-repository-docker, @BlobStore.direct-path=true, creationTime=1567407471329, @docker-range=null, @docker-name=saas-crm/dotnet-build, sha1=da39a3ee5e6b4b0d3255bfef95601890afd80709, @BlobStore.blob-name=nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/0, deletedReason=Docker upload cleaned up., size=0}
2019-09-03 05:18:27,117+0300 WARN  [quartz-3-thread-7]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob path$nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471475 attributes: /mnt/nexus_data/nexus/nexus-data/blobs/default/content/directpath/nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471475.properties {deletedDateTime=1567407472168, deleted=true, @BlobStore.created-by=nexus-repository-docker, @BlobStore.direct-path=true, creationTime=1567407471511, @docker-range=null, @docker-name=saas-crm/dotnet-build, sha1=7202f4c990ac83ff07782fd25680fc0530cda690, @BlobStore.blob-name=nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471475, deletedReason=Docker upload cleaned up., size=4966}
2019-09-03 05:18:27,119+0300 WARN  [quartz-3-thread-7]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob path$nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471618 attributes: /mnt/nexus_data/nexus/nexus-data/blobs/default/content/directpath/nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471618.properties {deletedDateTime=1567407472307, deleted=true, @BlobStore.created-by=nexus-repository-docker, @BlobStore.direct-path=true, creationTime=1567407471648, @docker-range=null, @docker-name=saas-crm/dotnet-build, sha1=da39a3ee5e6b4b0d3255bfef95601890afd80709, @BlobStore.blob-name=nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471618, deletedReason=Docker upload cleaned up., size=0}
2019-09-03 05:58:35,669+0300 WARN  [quartz-3-thread-7]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob path$nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/0 attributes: /mnt/nexus_data/nexus/nexus-data/blobs/default/content/directpath/nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/0.properties {deletedDateTime=1567407472074, deleted=true, @BlobStore.created-by=nexus-repository-docker, @BlobStore.direct-path=true, creationTime=1567407471329, @docker-range=null, @docker-name=saas-crm/dotnet-build, sha1=da39a3ee5e6b4b0d3255bfef95601890afd80709, @BlobStore.blob-name=nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/0, deletedReason=Docker upload cleaned up., size=0}
2019-09-03 05:58:35,675+0300 WARN  [quartz-3-thread-7]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob path$nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471475 attributes: /mnt/nexus_data/nexus/nexus-data/blobs/default/content/directpath/nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471475.properties {deletedDateTime=1567407472168, deleted=true, @BlobStore.created-by=nexus-repository-docker, @BlobStore.direct-path=true, creationTime=1567407471511, @docker-range=null, @docker-name=saas-crm/dotnet-build, sha1=7202f4c990ac83ff07782fd25680fc0530cda690, @BlobStore.blob-name=nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471475, deletedReason=Docker upload cleaned up., size=4966}
2019-09-03 05:58:35,679+0300 WARN  [quartz-3-thread-7]  *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Attempt to access soft-deleted blob path$nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471618 attributes: /mnt/nexus_data/nexus/nexus-data/blobs/default/content/directpath/nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471618.properties {deletedDateTime=1567407472307, deleted=true, @BlobStore.created-by=nexus-repository-docker, @BlobStore.direct-path=true, creationTime=1567407471648, @docker-range=null, @docker-name=saas-crm/dotnet-build, sha1=da39a3ee5e6b4b0d3255bfef95601890afd80709, @BlobStore.blob-name=nexus-repository-docker/86755c8d-5346-40e3-8603-a8c083e7b25d/1567407471618, deletedReason=Docker upload cleaned up., size=0}

ado · September 5, 2019, 11:21am

I suggest that you have some tags and manifests that are not associated with layers.

Nexus scan all of this, them marked that layers as orphaned and then delete them.

But in my case I have a special tasks for that (one for scan and one for compact blobstore). Do you have any tasks that running periodicly or some cleanup policies that assigned for your docker-repository?

Also you can enable debug log level following this article: Support Features

pit.artamonov · September 10, 2019, 6:12pm

UPD
The strange thing is that the “Purge incomplete docker uploads” task is launched every day. And normal should remove the artifact on the same day.

ado · September 11, 2019, 4:40am

Well about this task:
Pushing large images can result in failures due to network interruptions and other issues. These partial uploads result in temporary storage for these transfers in the repository manager filling up. The task Purge incomplete docker uploads can be configured to delete these files. If you also tend to upload images to the same tag repeatedly, this can leave a lot of dangling images around, consuming a lot of space. The task Purge unused docker manifests and images can be configured to remove these files. Further information about these tasks can be found in Configuring and Executing Tasks.

Link: Pushing Images

And a little more about tasks and how to configure them: System Configuration

In two words - you can edit settings of how often this task executed. You say that “Purge incomplete uploads” run every day so you may configure to run it, let’s say - once a week. It’s up to you to decide how often you need to execute this task.

Honestly I don’t remember that this task was enabled by default, but may be in some version of Nexus it does. Still I don’t see nothing criminal that these task runs every day and cleans up “incomplete uploads”. The main purpose of this task is not to delete some of your artifacts but delete layers of docker image that was pushed “no right way”(As example: you’re trying to push image and for whatever reason network stop working) and to free up your space because they are useless and you can’t properly use them.