Leveraging Nexus Repository for Efficient AI Model Management

Hello Nexus Sonatype Community,

As a long-time Nexus user, I’ve recently ventured into AI and encountered a significant challenge that I believe Nexus could help solve. AI model files are notoriously large, ranging from hundreds of megabytes to tens of gigabytes. Each version is a separate file, leading to two major issues:

  1. Storage Capacity: These large files quickly consume local machine storage.
  2. Network Efficiency: If team members require the same model, they must individually download it, which is a slow and bandwidth-intensive process.

I propose utilizing a local network-based Nexus repository specifically for AI models. This approach offers several benefits:

  • Single Download: Models are downloaded from the internet once, saving time and bandwidth.
  • Local Network Access: Team members can access these models at LAN speeds.
  • Storage Optimization: Users can delete local copies without worrying about re-download times.

I’m keen to hear your thoughts on this approach. Have any of you implemented something similar, or do you have suggestions on optimizing this setup using Nexus? Looking forward to a fruitful discussion!

Thank you!

I don’t see why you couldn’t do this? Seems like putting the files in a Raw (Hosted) repository would do the trick. There’s not any “metadata” around them but if you controlled the directory structure it should be relatively idiomatic.

Model > version_folder > model_filename_with_version

Take a look at JFrog Artifactory(requires license) provides versioning for models & acts a package manager for models, it directly interfaces with HuggingFace and acts as a proxy for any requested models. Simply specify the artificactory repo as an endpoint locally.

DVC is free & sits on top of git allowing you to version models, it provides support for several backends, ideally a local shared volume would be most optimal. It does support Cloud Storage but it is slower.