Issue with Cleanup Policies: Preview displays artifact but they are not cleaned when the cleanup task run

[anthony]

Hello,
I have an issue with Nexus Cleanup Policies. I can’t make it work properly. Maybe I’m missing something.

General information

• Nexus version: Community 3.80.0-06
• Saturated: No
• Read-only: No
• Database: H2
• Blobstore type: File (> 50 GB free)

The issue, what I’ve done, what I expect, what is happening

I created a cleanup policy named raw_[REDACTED]. The policy (raw type) consist of cleaning artifact that are 7 days old, keep them if they have been downloaded last 7 days and there is also a regex enabled (3 criterion in total). When I preview it, I have artifacts that displays:

Then I enable the policy in the raw repository:

Then I run the “Admin - Cleanup repositories using their associated policies” task:

Then I check if the artifacts are cleaned:

What I see: artifacts are not deleted.
What I expect: artifacts are soft deleted, can’t be listed. Disk space is not reclaimed yet.

Troubleshooting done

1. Is the regex the issue?
I tried to remove the regex criterion and do all the steps above, the result is the same.

2. What does the logs says?
I can’t access to the log of the cleanup for an unknown reason. I guess the log is here:

I suppose that the log is not empty, UI reports 6 KB of data. When I click on it, the log is empty. When I try to download it I have a 404 error:

3. Is the other policy enabled on the repo is causing an issue?
I tried to disable the other policy and follow all the steps above, but the result is the same.

4. Any errors in Nexus logs?
I didn’t see any errors in Nexus logs.

5. What if I compact the blob storage or run other cleaning related tasks?
The result remains the same.

Thank you for reading! Hopping that there is a solution to make the policy work. This feature is crucial to keep the memory usage controlled.

Regards,
Anthony

[itperfecto]

I have a very similar issue with this version of Nexus 3.80.

I can also see in the preview the assets that will be deleted, but after running the Cleanup Service task, the task log shows:

TASK org.sonatype.nexus.cleanup.internal.content.service.CleanupServiceImpl - 0 assets cleaned up for repository

It doesn’t find and delete them…

[impkos]

I have a similar problem

[Maximilian_Raab]

Hello,

the same for our instance. It does not work with Nexus OSS 3.80.0.

[Maximilian_Raab]

Still the same problem with 3.81.0.

[sollers]

Same, with Nexus OSS 3.80.0.

[philipsen.sergi (Sergi Philipsen)]

We are encountering the same issue. It is really annoying to us because we have a large test matrix that creates a lot of Docker images. At first, we could make the Virtual Disk bigger, but we are running very low. We are considering to downgrade, but we are not sure if this will break stuff.

[TwizzyDizzy]

Hi folks,

let’s bring a bit of data to the party. I am, like you, also affected and the issue can be traced back to the update to 3.80.0.

So here’s a diff of 3.79.1…3.80.0.

As one can see, there are indeed several changes to files related to the cleanup and, it seems, also regex-functionality.

I’m not qualified to judge those changes, but it is a data point that Sonatype might wanna consider.

Cheers
Thomas

[philipsen.sergi (Sergi Philipsen)]

The day we started to ship arm64 images, we also upgraded. So at first I thought this was related.

[Wolfgang_Seifert]

I also had this problem after upgrading to 3.80.

I downgraded to 3.79 but lost all data because these have been migrated already to 3.80 and cannot be read by 3.79.

I created a github issue here: Cleanup task does not clean anymore since 3.80 · Issue #672 · sonatype/nexus-public · GitHub

[Yakir_Magriso]

Hi.

We are facing the same issue as well with version 3.81.1.
Cleanup policies are not respected anymore. For example, cleanup of components last downloaded more than 7 days ago does not work anymore and we need to manually cleanup such components.
Any udpate on the issue from Sonatype is really appreciated.

Thanks,
Yakir

[jzora (Jonathan Zora)]

Hi, all! Our team is aware of some issue(s) with cleanup policies not targeting components correctly. They’re being actively investigated due to the potential for disruption in our users’ workflows.

I’ll share updates on the issue as they’re made available to me.

[jzora (Jonathan Zora)]

Hi, all,

We’ve identified solutions to a number of issues that were affecting cleanup policies. Fixes will be included as part of release 3.83.0.

[anthony]

Hello @jzora,
Thank you for keeping us updated on the subject !
I’ll wait for the new version.
Thank you.
Regards.

[anthony]

Hello,

I noticed that the version 3.83.0 is out now. I’ll test it from September 1st and I’ll keep this thread updated once I’ve been able to test the version.

Thank you !

Regards,
Anthony

[Wolfgang_Seifert]

It is still the same using 3.83.0!

The 3.83.0 release notes state that

NEXUS-47563 & NEXUS-47159
Cleanup policies using asset matchers with the option to retain a select number of versions now correctly identify and retain the expected number of Maven assets. The CSV preview and cleanup task execution now return accurate results when used with PostgreSQL-backed repositories.
NEXUS-47553
Cleanup policies using the Component age criteria now correctly remove eligible components when applied to repositories backed by an H2 database.

but these do not target my problem (asset matcher “.*”).

As mentioned here Cleanup task does not clean anymore since 3.80 · Issue #672 · sonatype/nexus-public · GitHub there seems to be a race-condition which prevents the repositories from being cleaned.
This is a more “subtle” problem, while the problems mentioned in the release notes are more “simple” ones.

[Yakir_Magriso]

Adding some more observations:

• We upgraded to version 3.82.0 and the issue is still encountered (I know that there are posts here that the issue is expected to be resolved on version 3.83.0, but the issues is still encountered there).
• We migrated the internal H2 DB to an external PostgreSQL DB and the issue is still encountered.
• We noticed that cleanup policies are respected only for assets located in the root directory of a repository - i.e. anything in a sub-directory is not deleted even if it is expected to be deleted by a cleanup policy.

Thanks,
Yakir

[plynch (Peter Lynch)]

@Wolfgang_Seifert

NEXUS-47553

Please provide the complete cleanup policy settings, the repo format it is being run on, the component path in the repo that doesn’t get deleted, the exact version of repo 3 and database being used.

[plynch (Peter Lynch)]

@Yakir_Magriso

Please provide the complete cleanup policy settings, the repo format it is being run on, the component path in the repo that doesn’t get deleted, the exact version of repo 3 and database being used.

[Wolfgang_Seifert]

Comparing nexus-oss version 3.79.1-04 (which works) with version 3.83.2-01 (which does not work) I found that changing browseEager back to browse made my cleanup tasks work again.

Sonatype adopted now one of my proposed changes in their new version 3.84.0-03. So my issue is solved.

See Cleanup task does not clean anymore since 3.80 · Issue #672 · sonatype/nexus-public · GitHub for details.