NPM Health Check report does not correlate with local npm audit results

lionelsaliou · May 20, 2021, 10:42am

Hello all,

My organisation has a Sonatype Nexus Repository Manager PRO 3.29.2-02 instance. As the fresh starting Application Security engineer for the organisation, I am trying to get a sense of how good (or how bad) the 3rd party libraries we rely upon are.

We are not at a maturity stage where policies can be enforced and developers’ mind changed. Therefore, it is key for me to ensure that we are have an accurate measure of the problem.

With this in mind, I thought that Nexus RPM could give us this data via the Health Checker. In order to verify that the Health Check report is accurate, I did the following:

Obtained an old(ish) NPM project;
Install its dependencies;
Run the npm audit and count the findings.

Then:

Configure the NPM proxy and group on Nexus Repo Manager;
Re-clone my NPM project;
Define a .npmrc to use Nexus as the proxy;
Wait for the health check to complete and count the findings.

For reference, my local npm audit using version 6.14.12 found a total of 17 issues. Much to my surprise (and disappointment) the Nexus Health Checker only identified 2 issues. What is most baffling is that the report states that only 50% of the packages were scanned. There is no explanation why that is and/or to fix it. For all I know, this lack of coverage is the root cause of the incorrect results.

In my view, this is an unacceptable discrepancy. I am creating this entry today in order to figure out what is possibly wrong here. As I am not allowed to attach files, here are extracts from our logs, showing that Health Check is running without errors.

2021-05-18 10:51:07,459+0000 INFO  [quartz-9-thread-12]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'System - Repository Health     Check: npm' [healthcheck] state change WAITING -> RUNNING
2021-05-18 10:51:07,718+0000 WARN  [quartz-9-thread-12]  *SYSTEM com.sonatype.insight.scan.client.ClientScanner - Could not locate client info descriptor
2021-05-18 10:51:08,191+0000 INFO  [quartz-9-thread-12]  *SYSTEM org.sonatype.nexus.scheduling.TaskSchedulerImpl - Task 'System - Repository Health Check: npm' [healthcheck] scheduled: hourly
2021-05-18 10:51:08,192+0000 INFO  [quartz-9-thread-12]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'System - Repository Health Check: npm' [healthcheck] state change RUNNING -> WAITING (OK)
2021-05-18 10:51:18,187+0000 INFO  [quartz-9-thread-12]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'System - Repository Health Check: npm' [healthcheck] state change WAITING -> RUNNING
2021-05-18 10:51:23,544+0000 INFO  [quartz-9-thread-12]  *SYSTEM com.sonatype.nexus.plugins.healthcheck.task.HealthCheckTask - Received health check report for repository npm
2021-05-18 10:51:23,549+0000 INFO  [quartz-9-thread-12]  *SYSTEM org.sonatype.nexus.scheduling.TaskSchedulerImpl - Task 'System - Repository Health Check: npm' [healthcheck] scheduled: hourly
2021-05-18 10:51:23,550+0000 INFO  [quartz-9-thread-12]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'System - Repository Health Check: npm' [healthcheck] state change RUNNING -> WAITING (OK)

Finally, I would like to stress that this ticket is NOT about having Nexus support npm audit for developers when using Nexus without the Firewall or IQ. I am aware there is a ticket for that already here:

My assertion is that if I am to recommend developers to use Nexus, then the Health Checker must be as accurate as dependency-check or npm audit or equivalent for your tool of choice. If it is not meant to be, please point me to the resource/documentation or otherwise that justify/explain the differences. If they are supposed to match, then please let me know what configuration is missing/incorrect.

Thank you for reading.

mmartz · May 20, 2021, 2:42pm

The npm health check and the repository health check are using two very different data sources. Repository health check uses our manually vetted security data which is generally more accurate and removes many false positives from the security vulnerabilities reported against projects. It’s the same data that powers our paid IQ server product and is the reason many organizations choose to use our paid solutions to track down vulnerable dependencies over our competition. If you manually check the npm audit results vs the health check results I’d expect you to find the same thing we usually find when comparing security scanning tools to IQ server - most of the “vulnerabilities” that npm finds don’t actually apply to the software you are looking at.

rseddon · May 20, 2021, 2:57pm

There is a known issue around the percentages shown for npm: https://issues.sonatype.org/browse/NEXUS-23754

mpiggott · May 20, 2021, 3:44pm

I would suggest contacting support to assist with diagnosing the issue.

lionelsaliou · May 20, 2021, 4:01pm

Hi Mmartz,

Thank you for taking the time to respond to my post!

Just to be clear, Nexus Health Check found 2 issues in my example repo, whereas npm audit found 17 total. For the 2 defects Nexus found, I did cross reference with the npm results and they matched: correct CVE, version and so on.

How can I validate your claim that the vulnerabilities that are found locally do not apply? In this case, many come with dedicated URLs on the NPM official repository where I can clearly see that the “version” of the package I am referencing is “listed” as vulnerable. I can accept the scenario where a reference to a vulnerable component is being made, but it is not used in actuality, however I cannot imagine the Health Checker being able to reach such conclusion. In order to provide feedback to my colleagues, I am happy to reference an official piece of SonaType/Product documentation!

mmartz · May 20, 2021, 4:18pm

I’m not sure that there is any documentation I can point to for your specific situation, though maybe someone from the sales organization could help. Typically we’d need to know the specific components/vulnerabilities being flagged by npm audit - usually it involves reading through the code for the dependency to see if the vulnerability still applies (which our security research team does). My understanding is often when the sales team gets involved in helping an organization make a purchasing decision related to IQ Server they’ll help do a deep analysis on stuff like this to give you confidence that our data is correct and that the results from npm audit include a lot of false positives.