Error in Nexus 3.47.1

Sonatype Nexus Repository
1

I updated our 2 Nexus instances from 3.46.0 to 3.47.1 (read the info about 3.47.0 before starting the update), but I now have issues with one of the instances.

On the first instance, we have a Docker registry, this one updated just fine.
On the second instance, we have PyPi and Raw repos. The update itself seemed to work just fine. Nothing in the logs pointed to any errors whatsoever.

Today I got a call from my colleagues that they aren’t able to download any files from the repos on the second instance anymore. Checking the logs I found lots of stacktraces like these:

2023-02-13 01:10:47,517+0100 WARN  [qtp206947563-1125]  *UNKNOWN org.sonatype.nexus.repository.httpbridge.internal.ViewServlet - Failure servicing: GET /repository/site/index.html
java.lang.IllegalArgumentException: Not a valid blob reference
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
        at org.sonatype.nexus.blobstore.api.BlobRef.parse(BlobRef.java:80)
        at org.sonatype.nexus.repository.storage.AssetEntityAdapter.readFields(AssetEntityAdapter.java:239)
        at org.sonatype.nexus.repository.storage.AssetEntityAdapter.readFields(AssetEntityAdapter.java:1)
        at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.readFields(MetadataNodeEntityAdapter.java:1)
        at org.sonatype.nexus.orient.entity.EntityAdapter.readEntity(EntityAdapter.java:286)
        at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.findByProperty(MetadataNodeEntityAdapter.java:167)
        at org.sonatype.nexus.repository.storage.StorageTxImpl.findAssetWithProperty(StorageTxImpl.java:396)
        at sun.reflect.GeneratedMethodAccessor82.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
-- cut --

Trying to examine a file in Nexus’ browser gave a popup with the same error: ‘Not a valid blob reference’.

I tried to run some tasks like ‘Repair - Rebuild repository browse’, after which Nexus’ browser was completely empty! From the log:

org.sonatype.nexus.repository.browse.node.RebuildBrowseNodeFailedException: Could not re-create browse nodes
        at org.sonatype.nexus.repository.browse.internal.orient.OrientRebuildBrowseNodeService.rebuild(OrientRebuildBrowseNodeService.java:132)
        at org.sonatype.nexus.repository.browse.node.RebuildBrowseNodesTask.execute(RebuildBrowseNodesTask.java:54)
        at org.sonatype.nexus.repository.RepositoryTaskSupport.execute(RepositoryTaskSupport.java:79)
        at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
        at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
        at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
        at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
        at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
        at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Not a valid blob reference
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:144)
        at org.sonatype.nexus.blobstore.api.BlobRef.parse(BlobRef.java:80)
        at org.sonatype.nexus.repository.storage.AssetEntityAdapter.readFields(AssetEntityAdapter.java:239)
        at org.sonatype.nexus.repository.storage.AssetEntityAdapter.readFields(AssetEntityAdapter.java:1)
        at org.sonatype.nexus.repository.storage.MetadataNodeEntityAdapter.readFields(MetadataNodeEntityAdapter.java:1)
        at org.sonatype.nexus.orient.entity.EntityAdapter.readEntity(EntityAdapter.java:286)
        at org.sonatype.nexus.orient.entity.action.ReadEntityByIdAction.execute(ReadEntityByIdAction.java:46)
        at org.sonatype.nexus.orient.entity.IterableEntityAdapter.read(IterableEntityAdapter.java:89)
        at org.sonatype.nexus.repository.storage.AssetStoreImpl.getById(AssetStoreImpl.java:71)
        at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
        at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
        at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
        at org.sonatype.nexus.repository.browse.internal.orient.OrientRebuildBrowseNodeService.rebuild(OrientRebuildBrowseNodeService.java:113)
        ... 15 common frames omitted

I then also tied ‘Repair - Reconcile component database from blob store’, which didn’t seem to do anything. Log is similar to the above stack-traces.

Hoping that my files were actually still there, I reverted back to 3.46.0 on the second instance. Thankfully everything was still there.

To me, it sounds like the fix for 3.47.0 may be having some side effects in 3.47.1 – but only on some types of repository?

2

If the search index is still intact, if you could search for a component you have trouble downloading and browse to the specific asset (file) then look for the Blob reference and reply with it.

3

Assuming you are an OSS user, please create a jira issue in our issue tracker with a support zip and full logs from the instance. Your attachments there will remain private between Sonatype and yourself.

Jira Project: NEXUS at Loading...

If you are a licensed paid customer, open a support ticket at https://support.sonatype.com

4

@plynch Will do. I assume it would be better to create the pack with 3.47.1 installed? I’d need to find a good time to shut down Nexus…

5

I assume it would be better to create the pack with 3.47.1 installed?

You can simply attach the archived logs from the upgrade and subsequent time from when the errors were reported. Archived and dated logs are in the sonatype-work/nexus3/log directory

6

Issue created: Log in - Sonatype JIRA

Thanks guys!

7

What appears unique in your case is you have a blobstore with a colon “:” in the name. This is causing the problem. We don’t expect this to be typical naming convention and is not desired to have special character names in blobstore names, but Nexus Repo presently lacks enough validation to prevent creating blobstores with special characters in the name. Until we fix this, we recommend not upgrading to 3.47.0 or newer if one has any other characters than alpha-numeric in your blobstore names.

8

We have recorded a public jira for the reported issue with blobstore names containing special characters: Loading...

9

Wow, that was fast!

We had a convention for naming our repos like this in Nexus2, not sure why this was transferred to Nexus3’s blobstores… And only in this one instance of Nexus as well…

Anyway, good to know what the issue is!

1 Like