Page in file browse_node_2.pcl with index 22604 was placed in wrong free list, this error will be fixed automatically

OSS Nexus version : OSS 3.37.3-02

Our log file nexus.log i being flooded with orientDB warnings about browse_node_2.pcl like this: - $ANSI{green {db=component}} Page in file browse_node_2.pcl with index 22604 was placed in wrong free list, this error will be fixed automatically

The log entries seems to start after restart of Nexus as soon as some build in our CI/CD-pipeline tries to access the Nexus repository
After a sort while Nexus does not respond to any uploads or downloads. The communication with the Nexus servers hangs.

First Step
We have investigated the orientDB, and we find that we are not able to rebuild the index asset_bucket_component_name_idx in the component DB due to duplicate key.
Logging in to the orientDB and using command REBUILD INDEX * gives us many (one at a time) errors about duplicate keys. Cannot index record #36:17372989: found duplicated key 'OCompositeKey{keys=[#28:4, null, no/ourorg/app/frontend/maven-metadata.xml.sha512]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #36:17372986

When looking at the two duplicate records we see that they have the same index key data (bucket, component, name) and that they were created at the same time, but otherwise they are different. One was was created by “anonymous” and was created by “cicduser”.

orientdb {db=component}> load record #36:17372989 DOCUMENT @class:asset @rid:#36:17372989 @version:1 +----+---------+----------------------------------------------------------------------------------------------------------------------------------+ |# |NAME |VALUE | +----+---------+----------------------------------------------------------------------------------------------------------------------------------+ |0 |bucket |#28:4 | |1 |format |maven2 | |2 |last_u...|2023-01-23 19:14:50 | |3 |attrib...|[{hashes_not_verified=false},{sha1=932d6eb15b34eba32de8cf0a6546e156c265bbf77, sha256=c4414b69d26a7dbc7652f2abc93ffd15bd1776f333c...| |4 |component| | |5 |name |no/ourorg/app/frontend/maven-metadata.xml.sha512 | |6 |size |128 | |7 |conten...|text/plain | |8 |create...|anonymous | |9 |create...| | |10 |blob_ref |default@1FBE9CBD-EBFC7202-65C22B6D-7312RM25-091D49DF:9594ef6c-767d-4394-92a9-c4f10f4c1234 | |11 |last_d...| | |12 |blob_c...|2023-01-23 19:14:50 | |13 |blob_u...|2023-01-23 19:14:50 | +----+---------+----------------------------------------------------------------------------------------------------------------------------------+

orientdb {db=component}> load record #36:17372986 DOCUMENT @class:asset @rid:#36:17372986 @version:4 +----+---------+-----------------------------------------------------------------------------------------------------------------------------------+ |# |NAME |VALUE | +----+---------+-----------------------------------------------------------------------------------------------------------------------------------+ |0 |bucket |#28:4 | |1 |format |maven2 | |2 |last_u...|2023-01-27 14:20:10 | |3 |attrib...|[{hashes_not_verified=false},{sha1=d33cc78f8db51519458b516deea6206684f7affd, sha256=f4c3b713e8c04a59d3781173b362c4c0693b1779365a...| |4 |component| | |5 |name |no/ourorg/app/frontend/maven-metadata.xml.sha512 | |6 |size |128 | |7 |conten...|text/plain | |8 |create...|nxbamboo | |9 |create...| | |10 |blob_ref |default@1RMA9CBD-EBFC7303-65C22B6D-7312EA25-051C49DF:6c2e0714-498c-6512-a0e5-ce088b36d76b | |11 |last_d...| | |12 |blob_c...|2023-01-23 19:14:50 | |13 |blob_u...|2023-01-27 14:20:10 | +----+---------+-----------------------------------------------------------------------------------------------------------------------------------+

There are many such instances of duplicate records.
We have tried to manually remove one of the duplicates, but just find another duplicate when trying to rebuild the index.

Second Step
We have cleaned out a lot of old snapshot from our locally hosted Maven2 reopsitory.
We used the a task of type Maven - Delete unused SNAPSHOTS, where we set the unused before days to 1.
That reduced the size of the database directory /sonatype-work/nexus3/db from 150Gb to about 50Gb.
We also dropped the index asset_bucket_component_name_idx from the database.
This also seemed to touch the file browse_node_2.pcl in some way, because when trying to rebuild the indexes again we did not get the error about duplicate keys.

Problem solved…?.. unfortunately not, for after a few days of “stable” running of Nexus, we got the same problem again.
We noticed Nexus responding slower (possibly due to a huge log file) and finally not responding at all.
Our log fil was again filling up with the same message:
Page in file browse_node_2.pcl with index XXXX was placed in wrong free list, this error will be fixed automatically.

Doen anybody have any suggestions about the cause of this?
We have tried quite a few more things since this, and face other issues at the moment, but we fear we don’t understand the root cause, and are just trying to treat the symptoms.
Any help would be greatly appreciated!

A quick followup.

The solution that has worked (so far) for us was to run the the task
Repair - Reconcile component database from blob store.
Our blob store was about 450 Gb and the task took 130 hours to complete.

After that we have not seen orientDB warnings about browse_node_2.pcl

As a general maintenance task we are planning to clean out a lot of old and unused artifacts from our repository in order to get a more maintainable size of our blob store.