Leveraging Nexus Repository for Efficient AI Model Management

Sonatype Nexus Repository
1

Hello Nexus Sonatype Community,

As a long-time Nexus user, I’ve recently ventured into AI and encountered a significant challenge that I believe Nexus could help solve. AI model files are notoriously large, ranging from hundreds of megabytes to tens of gigabytes. Each version is a separate file, leading to two major issues:

  1. Storage Capacity: These large files quickly consume local machine storage.
  2. Network Efficiency: If team members require the same model, they must individually download it, which is a slow and bandwidth-intensive process.

I propose utilizing a local network-based Nexus repository specifically for AI models. This approach offers several benefits:

  • Single Download: Models are downloaded from the internet once, saving time and bandwidth.
  • Local Network Access: Team members can access these models at LAN speeds.
  • Storage Optimization: Users can delete local copies without worrying about re-download times.

I’m keen to hear your thoughts on this approach. Have any of you implemented something similar, or do you have suggestions on optimizing this setup using Nexus? Looking forward to a fruitful discussion!

Thank you!

2

I don’t see why you couldn’t do this? Seems like putting the files in a Raw (Hosted) repository would do the trick. There’s not any “metadata” around them but if you controlled the directory structure it should be relatively idiomatic.

Model > version_folder > model_filename_with_version