Hi everyone,
We are experiencing database corruption on our Nexus Repository 3 instance (OSS 3.61.0-02, OrientDB v2.2.37).
Root Cause: Data files and folders were deleted directly from the backend S3 blob store instead of through the Nexus UI. As a result, the metadata in OrientDB became desynchronized and corrupted.
Observed Errors:
-
Web UI: Navigating through folders throws
"A database error occurred". -
OrientDB Record Access: Running
LOAD RECORDorDELETE FROM #...on corrupted records (such as inbrowse_node) fails withjava.nio.BufferUnderflowException. -
Storage Status: Storage was forced into
read-onlymode due toOPageIsBrokenExceptionand"Magic number verification failed"errors across page files. -
DB Check Failure: Running
CHECK DATABASEreported 3,214 storage errors and crashed at 35% while checking thecomponent_bucket_group_name_version_idxindex on clustercomponentwith the following stack trace:
Plaintext
Exception `782A4FFF` in storage `plocal:/data3/db/component/component`: 2.2.37
java.lang.NullPointerException
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.findBucket(OSBTree.java:1674)
at com.orientechnologies.orient.core.index.sbtree.local.OSBTree.get(OSBTree.java:194)
at com.orientechnologies.orient.core.index.engine.OSBTreeIndexEngine.get(OSBTreeIndexEngine.java:128)
at com.orientechnologies.orient.core.storage.impl.local.OAbstractPaginatedStorage.doGetIndexValue(OAbstractPaginatedStorage.java:2289)
at com.orientechnologies.orient.core.db.tool.OCheckIndexTool.checkThatRecordIsIndexed(OCheckIndexTool.java:153)
...
Questions / Proposed Solution: Due to the database size (~1.5TB across ~450 million records), a full JSON export/import would require 20+ hours of downtime and carries a high risk of failure during index recreation.
We are considering stopping Nexus, running TRUNCATE CLASS browse_node UNSAFE directly in the OrientDB console, and letting Nexus rebuild the browse tree and Elasticsearch indexes upon restart.
-
Is truncating
browse_nodesufficient? -
What is the recommended procedure to resolve these broken index pages and clear the
read-onlystate without a full database re-import?
Any guidance would be greatly appreciated.
Kind Regards,
Branka