Switching to Read-Only Mode BEFORE Running Out of Storage?

Sonatype Nexus Repository
1

Over the past month, I’ve had to restore my Nexus Repo instance from backups because when it ran out of space, the databases got corrupted somehow. I know that Nexus goes into read-only mode when the server is out of storage, but is it possible to fire off a warning email and also go into read-only mode at a user-defined percentage storage remaining… (e.g. when it’s nearing 5% storage remaining or even 5GB remaining, switch to read-only mode for component database and blobstore whilst also send an email to a user-defined email).

Thanks for any info y’all can provide!

2

I answered this on a support ticket for you, but I’ll post it again here so others can benefit.

You can configure this by adding a line like this to sonatype-work/nexus3/etc/nexus.properties:

storage.diskCache.diskFreeSpaceLimit=8192

That will set the limit to 8G.

In version 3.14.0 we increased the default from 256Mb to 4Gb, so if you’re on an older version you can just upgrade. That would be a good idea in any case.

Also note that it is a very good idea to put blob storage on a different disk than the rest of the work directory. That will prevent someone uploading a very large file and causing database corruption.

Rich

3

This doesn’t seem to work for me. Even though I added the following line to /nexus-data/etc/nexus.properties, the Nexus startup log shows it’s still using the default of 4096:

storage.diskCache.diskFreeSpaceLimit=1048

Nexus startup shows this:

2020-01-16 18:59:38,674+0000 INFO  [FelixStartLevel] *SYSTEM org.sonatype.nexus.bootstrap.ConfigurationBuilder -   storage.diskCache.diskFreeSpaceLimit='4096'
4

Sorry, I had the wrong file. You can find that setting in $installdir/etc/karaf/system.properties.

Rich

5

Hi Rich,

I’ve tried both /etc/karaf/system.properties as well as /etc/nexus.properties. Neither of these locations appear to impact the threshold.

If you have any additional information, I would greatly appreciate it. I’m trying to delete content, but can’t do to the read-only database.

Nexus 3.22.0

6

Hi @mvolkhart, Welcome to the Sonatype Community!

You can check that your setting took by looking at the Nexus UI:

image
image1145×132 23.5 KB

You may also find this KB article useful as an option to temporarily get the application back up and running while you clean up the data:

7

Also note that $installdir/etc/karaf/system.properties is definitely the right place, that is what we changed when we bumped it to 4Gb:

8

system.properties file exists under more than one directory path and some of the files will have the parameter storage.diskCache.diskFreeSpaceLimit. I ran the linux command “find / -name system.properties” and got results for 4 paths (including the one mentioned by Rich Seddon). Only 2 files from the results had the parameter storage.diskCache.diskFreeSpaceLimit set at ~4GB. Please note that changing only 1 file, will not take effect on the system as in the Nexus UI showed by Michael Worthington. Therefore, you need to check and edit under all files named system.properties in your results. Only then will the change take effect.

My database locked at free space of approx. 400MB, I changed the storage.diskCache.diskFreeSpaceLimit setting to 300MB and it allowed me privileges again to delete files and clear my database.

My comment is very late, but I hope it will help someone. Myself and a colleague were struggling with the same problem the whole day and this thread was extremely useful. I just wanted to share what worked for me and hope my observation will be of help to someone else. :+1:

9

Thanks for the confirmation @chabedi.shadrack! Glad you found this thread helpful. We are super excited to have you here in the Sonatype Community!