As part of a test with NEXUS3 as nuget package repo, we ran into the problem of vanishing assets. For our users it is unclear if this is a system or a user error, and even though there’s a gut feeling, there is no proof that it wasn’t the system.
Apparently, restoring a soft-deleted asset in NEXUS 2 was as simple as moving it out of the .trash folder and back into the repo tree. What is the equivalent of this in NEXUS 3?
Hi Edwin,
Sorry to hear you’re having some unexplained problems. It’d be helpful for me to understand exactly what you mean by “vanishing assets” - Does this mean that you can no longer see the asset in the browse UI / search screens or is it that the NuGet client cannot see it (i.e. it is missing from the NuGet feed for some reason)?
What version of NXRM3 are you running?
In NXRM3 the data is stored in two places; the binaries are stored on disk in the blob-store and then a reference to those binaries is stored in our internal database in the asset table. So there are some possibilities here:
In the first two possibilities you can get extra information by finding the blob .properties file for a removed asset and find the property “deletedReason”.
Joe
Hi Joe,
I’m running NEXUS 3:
|version|3.10.0-04|
|edition|OSS|
|buildRevision|f7b56bf088e8e313a210acbccc8b0739f446b962|
|buildTimestamp|2018-04-03-1355-42621|
Multiple packages were uploaded to our nexus hosted Nuget repo. These packages were older versions of packages already in the repo. A couple of days later many of these packages were gone from the UI. I did find these packages in the filesystem with their properties file stating deleted=true and deletedReason=Deleting asset AttachedEntityId{asset->#19:26}
I can not find a log entry telling me who (or what) deleted these packages.
I am not able to restore these packages, aside from having them completely reuploaded.
I did try removing the deletion entries from the properties files and a reindex, but as you said this was futile. How would I rebuild the asset record?
Kind regards,
Edwin
Edwin, when you say, “older versions of packages already in the repo,” are you saying that you republished older versions of components that had the exact same coordinates? Name, namespace and version?
If so, only one of those can exist at any one time in a given npm [edit: I mean NuGet, though npm has the same restriction] repository. It’s like overwriting a new file with an older copy; the previous (newer) version is deleted, because they share the same spot.
Is this what’s happening?
What is happening is: 2. The blob has been marked as soft-deleted and the asset record has also been deleted
By older version, I mean for example:
Asset 1 is still there.
Asset 2 vanished form the UI and now has “deleted=true” in its .properties file.
Asset 3 is the package that was reuploaded after a new hire couldn’t get Foo-3.5.2.nupkg from the repository.
All systems keep the packages in their cache for efficiency, which is why the missing packages weren’t discovered sooner.
Another user might have deleted these “older version” assets unknowingly, but I can’t find this in the log, and I am unable to undo such deletion.
A limited amount of users had deletion rights, until we find a root cause this has been revoked.
That’s clear, thanks—and it’s what I was concerned about.
There can be only one NuGet package called ‘Foo’ of version 3.5.2 in a single repository at any given time. Every time you publish something new to (“Foo”, “3.5.2”), you’re overwriting whatever’s alraedy there, just as if you were overwriting a file. You can’t have two files with the same path and file name on the same file system; you can’t have two packages with the same coordinate in the same repository. For NuGet, the “coordinate” is packageId and version.
Although many ecosystems allow it, we highly recommend against republishing binaries with reused coordinates. If at all possible, adjust your development practices to prevent this from happening.
When you visualize the whole pipeline from development, publication, to downstream caching (possibly in multiple business units, or in a tier of load-balanced read proxies), it can be impossible to know if consumers are referencing the “right one,” or (as happened to you) to tell if the wrong one is still lurking around.
Also, it means your historical record is lossy, since you’re not keeping all of the versions you may care about (for audit reasons, etc.)
In this specific case, you might want to hand-edit the July 13 package to have a unique version number. NuGet allows pre-release qualifiers, so you could have these two packages:
Foo-3.5.2.nupkg
Foo-3.5.2-weForgotToSignItOrWhatever.nupkg
Then you can use the UI Upload feature to upload the ‘old’ version to the repository, where it can live alongside the other one because it has a different coordinate.
Actually, this is the desired situation:
We only reuploaded the package after it was already gone from the web UI and “deleted=true” (soft-deleted).
So the question remains:
Hi everyone! Sorry for reviving an 1 year old topic…
@Edwin, did you manage to restore the soft-deleted assets? I’m facing a similar problem and I didn’t find how to do it anywhere.
Thanks
Disclaimer… the following instructions use an implementation detail of the current blob store. As such, they are not officially supported, and they won’t work if the implementation changes.
Associated with each blob in the blob store there is a properties file. So for instance A particular blob file will consist of two files:
fac5e64d-831b-4b58-98c9-3dd0e4bd1b59.bytes
fac5e64d-831b-4b58-98c9-3dd0e4bd1b59.properties
Within the properties files you’ll see content like this:
#2018-08-22 08:33:33,965-0500
#Wed Aug 22 08:33:33 CDT 2018
@BlobStore.created-by=admin
size=40
@Bucket.repo-name=maven-releases
creationTime=1534944813965
@BlobStore.created-by-ip=127.0.0.1
@BlobStore.content-type=text/plain
@BlobStore.blob-name=org/foo/bar/project/1.2.3/project-1.2.3.jar
sha1=30d7edff3bc3f5e601be9529460b7bd573b1a8b7
deleted=true
The “@Bucket.repo-name” property shows which repository the blob is associated with. The “deleted=true” indicates the blob is in soft deleted state.
To restore the soft deleted file, locate the properties file and remove the “deleted=true” line. Then schedule and run a “repair: reconcile component database from blobstore” task.
There are some limitations associated with the reconcile task, see here for details:
https://help.sonatype.com/display/NXRM3/System+Configuration
And then, you should apply this solution for preventing deletion task.