OSS Nexus version : OSS 3.37.3-02
Our log file nexus.log i being flooded with orientDB warnings about browse_node_2.pcl like this:
com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster - $ANSI{green {db=component}} Page in file browse_node_2.pcl with index 22604 was placed in wrong free list, this error will be fixed automatically
The log entries seems to start after restart of Nexus as soon as some build in our CI/CD-pipeline tries to access the Nexus repository
After a sort while Nexus does not respond to any uploads or downloads. The communication with the Nexus servers hangs.
First Step
We have investigated the orientDB, and we find that we are not able to rebuild the index asset_bucket_component_name_idx in the component DB due to duplicate key.
Logging in to the orientDB and using command REBUILD INDEX * gives us many (one at a time) errors about duplicate keys.
com.orientechnologies.orient.core.storage.ORecordDuplicatedException: Cannot index record #36:17372989: found duplicated key 'OCompositeKey{keys=[#28:4, null, no/ourorg/app/frontend/maven-metadata.xml.sha512]}' in index 'asset_bucket_component_name_idx' previously assigned to the record #36:17372986
When looking at the two duplicate records we see that they have the same index key data (bucket, component, name) and that they were created at the same time, but otherwise they are different. One was was created by “anonymous” and was created by “cicduser”.
orientdb {db=component}> load record #36:17372989 DOCUMENT @class:asset @rid:#36:17372989 @version:1 +----+---------+----------------------------------------------------------------------------------------------------------------------------------+ |# |NAME |VALUE | +----+---------+----------------------------------------------------------------------------------------------------------------------------------+ |0 |bucket |#28:4 | |1 |format |maven2 | |2 |last_u...|2023-01-23 19:14:50 | |3 |attrib...|[{hashes_not_verified=false},{sha1=932d6eb15b34eba32de8cf0a6546e156c265bbf77, sha256=c4414b69d26a7dbc7652f2abc93ffd15bd1776f333c...| |4 |component| | |5 |name |no/ourorg/app/frontend/maven-metadata.xml.sha512 | |6 |size |128 | |7 |conten...|text/plain | |8 |create...|anonymous | |9 |create...|172.25.4.231 | |10 |blob_ref |default@1FBE9CBD-EBFC7202-65C22B6D-7312RM25-091D49DF:9594ef6c-767d-4394-92a9-c4f10f4c1234 | |11 |last_d...| | |12 |blob_c...|2023-01-23 19:14:50 | |13 |blob_u...|2023-01-23 19:14:50 | +----+---------+----------------------------------------------------------------------------------------------------------------------------------+
orientdb {db=component}> load record #36:17372986 DOCUMENT @class:asset @rid:#36:17372986 @version:4 +----+---------+-----------------------------------------------------------------------------------------------------------------------------------+ |# |NAME |VALUE | +----+---------+-----------------------------------------------------------------------------------------------------------------------------------+ |0 |bucket |#28:4 | |1 |format |maven2 | |2 |last_u...|2023-01-27 14:20:10 | |3 |attrib...|[{hashes_not_verified=false},{sha1=d33cc78f8db51519458b516deea6206684f7affd, sha256=f4c3b713e8c04a59d3781173b362c4c0693b1779365a...| |4 |component| | |5 |name |no/ourorg/app/frontend/maven-metadata.xml.sha512 | |6 |size |128 | |7 |conten...|text/plain | |8 |create...|nxbamboo | |9 |create...|172.25.4.228 | |10 |blob_ref |default@1RMA9CBD-EBFC7303-65C22B6D-7312EA25-051C49DF:6c2e0714-498c-6512-a0e5-ce088b36d76b | |11 |last_d...| | |12 |blob_c...|2023-01-23 19:14:50 | |13 |blob_u...|2023-01-27 14:20:10 | +----+---------+-----------------------------------------------------------------------------------------------------------------------------------+
There are many such instances of duplicate records.
We have tried to manually remove one of the duplicates, but just find another duplicate when trying to rebuild the index.
Second Step
We have cleaned out a lot of old snapshot from our locally hosted Maven2 reopsitory.
We used the a task of type Maven - Delete unused SNAPSHOTS, where we set the unused before days to 1.
That reduced the size of the database directory /sonatype-work/nexus3/db from 150Gb to about 50Gb.
We also dropped the index asset_bucket_component_name_idx from the database.
This also seemed to touch the file browse_node_2.pcl in some way, because when trying to rebuild the indexes again we did not get the error about duplicate keys.
Problem solved…?.. unfortunately not, for after a few days of “stable” running of Nexus, we got the same problem again.
We noticed Nexus responding slower (possibly due to a huge log file) and finally not responding at all.
Our log fil was again filling up with the same message:
Page in file browse_node_2.pcl with index XXXX was placed in wrong free list, this error will be fixed automatically.
Doen anybody have any suggestions about the cause of this?
We have tried quite a few more things since this, and face other issues at the moment, but we fear we don’t understand the root cause, and are just trying to treat the symptoms.
Any help would be greatly appreciated!