Over the past few days we have been struggling with an issue where we are unable to read or push anything towards our Nexus instance and the only way to recover is to restore it from a backup.
At the time tons of the below message appear (like 2-3 every milli second).
2022-06-23 07:49:30,635+0000 WARN [qtp1258398841-613] ci.user com.orientechnologies.orient.core.storage.impl.local.paginated.OPaginatedCluster - $ANSI{green {db=component}} Page in file asset_14.pcl with index 154 was placed in wrong free list, this error will be fixed automatically.
The issue re-appears randomly after a restore on the same asset file and the same indexes are reported.
Around a month ago we have upgraded our instance to 3.37.3, where the issue initially occurred. Yesterday we have upgraded to 3.39.0, hoping that it would sort this out.
Looking at OrientDB’s github repository, there was a similar issue reported and there seems to be a solution in version 3, however Nexus is still using v2 of the database.
opened 08:16AM - 14 Feb 18 UTC
closed 11:52AM - 12 Jun 20 UTC
### OrientDB Version: 2.2.18
### Java Version: 1.8.0_144
### OS: FreeBSD 11.0-… RELEASE-p1
Hi, I have got a problem when I created new class in database.
I created "drives" class and then inserted approximately 100 records.
After rebooting my server, I found that cluster segments for drives class were missing.
Some queries for listing drive class do not work anymore.
It showed an error like this:
> Cluster segment #142 does not exist in database 'FTO-Structure-2.0'"
#### Database Information
I checked the database by console.sh with info command.
Drive clusters were missing and the index of 3 of class list was also gone.
The followings are the parts of a result that are invalid:
> CLUSTERS (collections)
> +----+--------------+----+------------+-----------------+-----+----------+
> |# |NAME | ID|CLASS |CONFLICT-STRATEGY|COUNT|SPACE-USED|
> +----+--------------+----+------------+-----------------+-----+----------+
> |0 |_studio | 97|_studio | | 6| 786b|
> |1 |_studio_1 | 98|_studio | | 0| 0b|
> |2 |_studio_2 | 99|_studio | | 0| 0b|
> |3 |_studio_3 | 100|_studio | | 0| 0b|
> |4 |_studio_4 | 101|_studio | | 0| 0b|
> |5 |_studio_5 | 102|_studio | | 0| 0b|
> |6 |_studio_6 | 103|_studio | | 0| 0b|
> |7 |_studio_7 | 104|_studio | | 0| 0b|
> |8 |alias | 89|alias | | 11| 810b|
> |9 |alias_1 | 90|alias | | 5| 430b|
> |10 |alias_2 | 91|alias | | 5| 438b|
> |11 |alias_3 | 92|alias | | 5| 429b|
> |12 |alias_4 | 93|alias | | 4| 366b|
> |13 |alias_5 | 94|alias | | 5| 427b|
> |14 |alias_6 | 95|alias | | 3| 300b|
> |15 |alias_7 | 96|alias | | 3| 300b|
> |16 |default | 3| |version | 0| 0b|
> |17 |domains | 25|domains | | 2| 330b|
> |18 |domains_1 | 26|domains | | 1| 148b|
> |19 |domains_2 | 27|domains | | 0| 0b|
> |20 |domains_3 | 28|domains | | 0| 0b|
> |21 |domains_4 | 29|domains | | 0| 0b|
> |22 |domains_5 | 30|domains | | 0| 0b|
> |23 |domains_6 | 31|domains | | 0| 0b|
> |24 |domains_7 | 32|domains | | 0| 0b|
> |25 |e | 17|E | | 0| 0b|
> |26 |e_1 | 18|E | | 0| 0b|
> |27 |e_2 | 19|E | | 0| 0b|
> |28 |e_3 | 20|E | | 0| 0b|
> |29 |e_4 | 21|E | | 0| 0b|
> |30 |e_5 | 22|E | | 0| 0b|
> |31 |e_6 | 23|E | | 0| 0b|
> |32 |e_7 | 24|E | | 0| 0b|
>
> CLASSES
> +----+-----+----------+-----------+--------------------------------------------------------------------------------------------------------------------+
> |# |COUNT|NAME |SUPER-CLASS|CLUSTERS |
> +----+-----+----------+-----------+--------------------------------------------------------------------------------------------------------------------+
> |0 | 6|_studio | |_studio(97),_studio_1(98),_studio_2(99),_studio_3(100),_studio_4(101),_studio_5(102),_studio_6(103),_studio_7(104) |
> |1 | 41|alias |[E] |alias(89),alias_1(90),alias_2(91),alias_3(92),alias_4(93),alias_5(94),alias_6(95),alias_7(96) |
> |2 | 3|domains |[V] |domains(25),domains_1(26),domains_2(27),domains_3(28),domains_4(29),domains_5(30),domains_6(31),domains_7(32) |
> |3 | | | | |
> |4 | 0|E | |e(17),e_1(18),e_2(19),e_3(20),e_4(21),e_5(22),e_6(23),e_7(24) |
#### Database Directory
However, the database's directory still contains files for drives class.
> ...
> -rwxr-xr-x 1 orientdb orientdb 1024 Mar 6 2017 domains_7.cpm
> -rwxr-xr-x 1 orientdb orientdb 66560 Mar 6 2017 domains_7.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_1.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_1.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_2.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_2.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_3.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_3.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_4.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_4.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_5.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_5.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_6.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_6.pcl
> -rwxr-xr-x 1 orientdb orientdb 66560 Feb 14 12:46 drives_7.cpm
> -rwxr-xr-x 1 orientdb orientdb 263168 Feb 14 12:46 drives_7.pcl
> -rwxr-xr-x 1 orientdb orientdb 1024 Mar 6 2017 e.cpm
> -rwxr-xr-x 1 orientdb orientdb 66560 Mar 6 2017 e.pcl
> ...
#### orient-server.log
I have just noticed some errors in orient-server.log. The warning happened after I created new class.
> 2018-02-12 10:28:09:667 WARNI {db=FTO-Structure-2.0} Page in file drives.pcl with index 0 was placed in wrong free list, this error will be fixed automatically [OPaginatedCluster]
> 2018-02-12 10:28:13:320 WARNI {db=FTO-Structure-2.0} Page in file drives_7.pcl with index 0 was placed in wrong free list, this error will be fixed automatically [OPaginatedCluster]
I have already tried repairing by "repair database --fix-graph" and "repair database --fix-ridbags", but it did not work.
Is there any other ways to repair my database?
So far we have checked the filesystem for consistency checks, ran checks on the orientdb, rebuilt the indexes. All seems to be in order, no issues were noted.
Do you have idea what could be going wrong here? Any feedback would be appreciated.
Thank you.
Adrian
mpiggott
June 27, 2022, 11:39pm
#2