For a brand new deployment (no sonatype-work directory) you will have to create the path and file: /opt/<nexus directory>/sonatype-work/nexus3/etc/nexus.properties - in this file add the line ‘nexus.datastore.enabled=true’ and then start the application. This will start Nexus with the H2 database.
It is likely you were adding this in to nexus.properties found in the application directory such as /opt/<nexus directory>/nexus-3.70.1-02/etc which will not work.
If you have an existing deployment you would need to also run a DB migration to move the data from Orient to H2.
https://help.sonatype.com/en/migrating-to-a-new-database.html#migrating-from-orientdb-to-h2-162010