A good source of how to do it using a plugin is the implementation of default realm:
https://github.com/sonatype/nexus-public/tree/main/plugins/nexus-default-role-plugin
The problem I see in your approach is the anonymous access. If you want everything authenticated, it think might work as you want to, but I see a problem with repositories you want to access with no authentication (but being able to update using authentication for example). I do not know how you would do it nicely.
Possibly forcing the remote token authentication for web UI and to require generated tokens from Nexus for updates (they are sent as basic authentication). But it creates a hell of configuration on the reverse proxy side (some URLs authenticated, some not…).
Creating your own plugin fixes this. You need to resolve the part with web access authentication and then you need to implement the authorization part. The SAML implementation by Sonatype is not very agile. It only works on a single domain - if you use multiple domains to access Nexus, you have a problem. If your identity provider changes configuration, you also have a problem and must fix it manually. In my case the biggest hurdle of it is the single-domain support.
Edit: I strongly suggest you do NOT try to update Sonatype code. I do not know how you would compile everything and create a working application, because starting with 3.80 the free and professional version were merged so you would have to compile parts you do not have (or add them through build hacking). And I am not considering the legality of this attempt.