We have an instance of Nexus 3.58.1-02 serving as an internal NuGet repository. We have 3 separate hosted NuGet feeds all on a common blob store. The blob store is on our internal network drive. Every night, we run 3 tasks - backup, then cleanup, then compact. Starting this past Friday night, the Compact task has been failing. The error message is “An unexpected network error occurred.” The whole log is pasted at the bottom.
On Thursday, I did a couple things that I wouldn’t expect to cause an error. I added a git-lfs hosted repository, which had 1 file added to it. I also deleted the “v3” folder from one of the nuget repositories. That was because we’ve seen some weird behavior with the v3 index and I have seen that alleviate the problem in the past.
The tasks ran successfully Thursday night, which I believe was after I had done both of those things. We aren’t aware of any changes that happened Friday that could have impacted our server health.
Today, I’ve tried:
- Rebuilding the deletions index file
- Running a “Reconcile component database from blob store task”
- Clicking “Rebuild Index” on the nuget feed that I previously deleted the v3 folder from
None of these seemed to have an effect.
Task log:
2023-09-03 19:10:52,020-0500 INFO [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - ID: 2eee85fa-ed63-4d9c-a970-5052c13c31b8
2023-09-03 19:10:52,020-0500 INFO [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Type: blobstore.compact
2023-09-03 19:10:52,020-0500 INFO [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Name: Compact blob store
2023-09-03 19:10:52,036-0500 INFO [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Description: Compacting default blob store
2023-09-03 19:10:52,036-0500 INFO [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.file.FileBlobStore - Begin deleted blobs processing
2023-09-03 19:10:52,036-0500 ERROR [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Failed to run task 'Compacting default blob store'
org.sonatype.nexus.blobstore.api.BlobStoreException: java.io.IOException: An unexpected network error occurred, Cause: An unexpected network error occurred
at org.sonatype.nexus.blobstore.file.FileBlobStore.doCompact(FileBlobStore.java:650)
at org.sonatype.nexus.blobstore.BlobStoreSupport.compact(BlobStoreSupport.java:222)
at org.sonatype.nexus.common.stateguard.MethodInvocationAction.run(MethodInvocationAction.java:39)
at org.sonatype.nexus.common.stateguard.StateGuard$GuardImpl.run(StateGuard.java:272)
at org.sonatype.nexus.common.stateguard.GuardedInterceptor.invoke(GuardedInterceptor.java:54)
at org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask.execute(CompactBlobStoreTask.java:102)
at org.sonatype.nexus.scheduling.TaskSupport.call(TaskSupport.java:100)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.doExecute(QuartzTaskJob.java:143)
at org.sonatype.nexus.quartz.internal.task.QuartzTaskJob.execute(QuartzTaskJob.java:106)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.sonatype.nexus.quartz.internal.QuartzThreadPool.lambda$0(QuartzThreadPool.java:145)
at org.sonatype.nexus.thread.internal.MDCAwareRunnable.run(MDCAwareRunnable.java:40)
at org.apache.shiro.subject.support.SubjectRunnable.doRun(SubjectRunnable.java:120)
at org.apache.shiro.subject.support.SubjectRunnable.run(SubjectRunnable.java:108)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: An unexpected network error occurred
at java.io.RandomAccessFile.readBytes(Native Method)
at java.io.RandomAccessFile.read(RandomAccessFile.java:377)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:436)
at com.squareup.tape.QueueFile.ringRead(QueueFile.java:269)
at com.squareup.tape.QueueFile.peek(QueueFile.java:415)
at org.sonatype.nexus.blobstore.file.internal.orient.OrientFileBlobDeletionIndex.readOldestRecord(OrientFileBlobDeletionIndex.java:107)
at org.sonatype.nexus.blobstore.file.FileBlobStore.doCompactWithDeletedBlobIndex(FileBlobStore.java:870)
at org.sonatype.nexus.blobstore.file.FileBlobStore.doCompact(FileBlobStore.java:643)
... 18 common frames omitted
2023-09-03 19:10:52,036-0500 INFO [quartz-10-thread-20] *SYSTEM org.sonatype.nexus.blobstore.compact.internal.CompactBlobStoreTask - Task complete