Maven build fails intermittently due to artificats not found error

vishnuvp · November 4, 2019, 4:39am

Hi All,

I have a sonatype nexus 3.19.01 setup with maven-proxy repository. We are behind corp proxy and I think thats the issue. I have proxy configured and it is working because some of the artificats are downloaded
I setup maven repo referring this link : Nexus Maven proxy setup

I also setup an atlassian proxy repo .

I added both these proxy repos to a maven group called maven-public.

In the build machine, my maven settings is as below.

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
                      https://maven.apache.org/xsd/settings-1.0.0.xsd">
   <mirrors>
    <mirror>
     <id>nexus</id>
     <url>http://mynexushost:8080/repository/maven-public/</url>
     <mirrorOf>*</mirrorOf>
    </mirror>
   </mirrors>
   <proxies>
    <proxy>
      <id>http</id>
      <active>true</active>
      <protocol>http</protocol>
      <host>myproxy</host>
      <port>8080</port>
      <username>myuser</username>
      <password>mypass</password>
      <nonProxyHosts>*.mycomp.com|127.0.0.1|10.16.193.141|10.228.4.80|mynexushost|testhost</nonProxyHosts>
    </proxy>
    <proxy>
      <id>https</id>
      <active>true</active>
      <protocol>https</protocol>
      <host>myproxy</host>
      <port>8080</port>
      <username>myuser</username>
      <password>mypass</password>
      <nonProxyHosts>*.mycomp.com|127.0.0.1|10.16.193.141|10.228.4.80|mynexushost|testhost</nonProxyHosts>
    </proxy>
  </proxies>
  <profiles>
    <profile>
      <id>nexus</id>
      <!--Enable snapshots for the built in central repo to direct -->
      <!--all requests to nexus via the mirror -->
      <repositories>
        <repository>
          <id>central</id>
          <url>http://central</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>true</enabled></snapshots>
        </repository>
      </repositories>
     <pluginRepositories>
        <pluginRepository>
          <id>central</id>
          <url>http://central</url>
          <releases><enabled>true</enabled></releases>
          <snapshots><enabled>true</enabled></snapshots>
        </pluginRepository>
      </pluginRepositories>
    </profile>
  </profiles>
  <activeProfiles>
    <!--make the profile active all the time -->
    <activeProfile>nexus</activeProfile>
  </activeProfiles>
</settings>

I am using this repo for maven build :Sample github repo

I am getting this error.
[ERROR] Failed to execute goal on project dailytaskcreator: Could not resolve dependencies for project com.vishnu.automation:dailytaskcreator:jar:1.0: Failed to collect dependencies at com.atlassian.jira:jira-rest-java-client-core:jar:5.1.6 -> com.atlassian.sal:sal-api:jar:3.0.7: Failed to read artifact descriptor for com.atlassian.sal:sal-api:jar:3.0.7: Could not find artifact com.atlassian.sal:sal-parent:pom:3.0.7 in nexus (http://mynexushost:8080/repository/maven-public/) -> [Help 1]

Complete maven trace is below.
[ERROR] Failed to execute goal on project dailytaskcreator: Could not resolve dependencies for project com.vishnu.automation:dailytaskcreator:jar:1.0: Failed to collect dependencies at com.atlassian.jira:jira-rest-java-client-core:jar:5.1.6 -> com.atlassian.event:atlassian-event:jar:3.0.0: Failed to read artifact descriptor for com.atlassian.event:atlassian-event:jar:3.0.0: Could not find artifact com.atlassian.platform:platform-poms:pom:3.0.0-m073 in nexus (http://nexushost:8080/repository/maven-public/) -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal on project dailytaskcreator: Could not resolve dependencies for project com.vishnu.automation:dailytaskcreator:jar:1.0: Failed to collect dependencies at com.atlassian.jira:jira-rest-java-client-core:jar:5.1.6 -> com.atlassian.event:atlassian-event:jar:3.0.0 at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:269) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147) at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347)

Upon re-execution of command “mvn package -U” the build passes. Any idea whats going wrong. I am puzzled why some artifacts gets downloaded in first attempt and some fails ?

Thanks,
Vishnu

jtom · November 4, 2019, 7:30pm

Have you checked your logs for failures/failure codes?
-U pretty much tries again despite failures, so could be pretty much anything that causes them.
If intermittant could be something like network, or as you say corp proxy.
All you’ve shown here is that it is failing.
-Joe

vishnuvp · November 5, 2019, 2:49am

I tailed nexus log and saw this.

2019-11-04 16:00:00,004+1100 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change WAITING -> RUNNING
2019-11-04 16:00:00,005+1100 INFO  [quartz-3-thread-20]  *SYSTEM org.sonatype.nexus.quartz.internal.task.QuartzTaskInfo - Task 'Storage facet cleanup' [repository.storage-facet-cleanup] state change RUNNING -> WAITING (OK)
2019-11-04 16:00:17,700+1100 INFO  [qtp1007621157-1074]  *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for atlassian-proxy changed from AVAILABLE to AUTO_BLOCKED_UNAVAILABLE until 2019-11-04T16:00:57.700+11:00 - reason Unauthorized for https://packages.atlassian.com
2019-11-04 16:00:17,701+1100 WARN  [qtp1007621157-1074]  *UNKNOWN org.sonatype.nexus.repository.maven.internal.proxy.MavenProxyFacet - Exception org.sonatype.nexus.repository.proxy.ProxyServiceException: HTTP/1.1 401 Unauthorized checking remote for update, proxy repo atlassian-proxy failed to fetch com/atlassian/httpclient/atlassian-httpclient-api/2.0.0/atlassian-httpclient-api-2.0.0.pom.sha1 with status line HTTP/1.1 401 Unauthorized, content not in cache.
2019-11-04 16:00:17,723+1100 INFO  [qtp1007621157-1074]  *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for maven-central changed from AVAILABLE to UNAVAILABLE - reason Unauthorized for https://repo1.maven.org
2019-11-04 16:00:17,723+1100 WARN  [qtp1007621157-1074]  *UNKNOWN org.sonatype.nexus.repository.maven.internal.proxy.MavenProxyFacet - Exception org.sonatype.nexus.repository.proxy.ProxyServiceException: HTTP/1.1 401 Unauthorized checking remote for update, proxy repo maven-central failed to fetch com/atlassian/httpclient/atlassian-httpclient-api/2.0.0/atlassian-httpclient-api-2.0.0.pom.md5 with status line HTTP/1.1 401 Unauthorized, content not in cache.
2019-11-04 16:00:19,235+1100 INFO  [qtp1007621157-1074]  *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for maven-central changed from UNAVAILABLE to AVAILABLE - reason n/a for n/a
2019-11-04 16:00:58,215+1100 INFO  [Check Status https://packages.atlassian.com]  *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for atlassian-proxy changed from AUTO_BLOCKED_UNAVAILABLE to AVAILABLE - reason n/a for n/a

rseddon · November 5, 2019, 2:21pm

Maven Central does not return 401 responses for GET requests. It looks like your company’s firewall is blocking access. The next step here would be to check with your firewall administrators.

vishnuvp · November 7, 2019, 5:38am

Thanks.I will check with them. However can there be instances where Nexus fails without even being attempting to contact Maven group.

I was executing another maven project today and this is what I got as build log. Corresponding Nexus was nil.
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: The following artifacts could not be resolved: nl.captcha:simplecaptcha:jar:1.2.1, org.apache.woden:woden:jar:1.0-incubating-M7b: Could not find artifact nl.captcha:simplecaptcha:jar:1.2.1 in nexus (http://lnnxrepo:8080/repository/maven-public/) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolve (DefaultArtifactResolver.java:413) at org.eclipse.aether.internal.impl.DefaultArtifactResolver.resolveArtifacts (DefaultArtifactResolver.java:215) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.resolveDependencies (DefaultRepositorySystem.java:325) at org.apache.maven.project.DefaultProjectDependenciesResolver.resolve (DefaultProjectDependenciesResolver.java:202) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.getDependencies (LifecycleDependencyResolver.java:243) at org.apache.maven.lifecycle.internal.LifecycleDependencyResolver.resolveProjectDependencies (LifecycleDependencyResolver.java:147) at org.apache.maven.lifecycle.internal.MojoExecutor.ensureDependenciesAreResolved (MojoExecutor.java:248) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:202) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156) at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288) at org.apache.maven.cli.MavenCli.main (MavenCli.java:192) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356) Caused by: org.eclipse.aether.transfer.ArtifactNotFoundException: Could not find artifact nl.captcha:simplecaptcha:jar:1.2.1 in nexus (http://lnnxrepo:8080/repository/maven-public/) at org.eclipse.aether.connector.basic.ArtifactTransportListener.transferFailed (ArtifactTransportListener.java:48) at org.eclipse.aether.connector.basic.BasicRepositoryConnector$TaskRunner.run (BasicRepositoryConnector.java:368) at org.eclipse.aether.util.concurrency.RunnableErrorForwarder$1.run (RunnableErrorForwarder.java:75) at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624) at java.lang.Thread.run (Thread.java:748)

rseddon · November 7, 2019, 2:39pm

Maven will cache previous download failures, by default this cache lasts 24 hours. To clear this cache run the build with “mvn -U”.

vishnuvp · November 7, 2019, 11:12pm

Sorry I didn’t mention earlier, I am executing always with mvn -U -X for logs with debugging.

vishnuvp · November 12, 2019, 1:39am

I think the issue is there for all proxy type repositories. Today I was doing a .Net build with nuget repository and I got same issue.

I checked with my firewall admins. They needed info on how Nexus uses my proxy credentials. Does nexus use the proxy auth credentials on every request ?

2019-11-11 14:58:17,306+1100 INFO [Check Status https://www.nuget.org] *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for nuget_proxy changed from AUTO_BLOCKED_UNAVAILABLE to AVAILABLE - reason n/a for n/a 2019-11-11 14:58:22,402+1100 INFO [elasticsearch[13E81695-81631669-25B736E4-B4BC35F5-191C8F10][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.metadata - [13E81695-81631669-25B736E4-B4BC35F5-191C8F10] [03ae57836ec4ae2297f2739b8b86af35f6b9f92a] update_mapping [component] 2019-11-11 14:58:23,412+1100 INFO [elasticsearch[13E81695-81631669-25B736E4-B4BC35F5-191C8F10][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.metadata - [13E81695-81631669-25B736E4-B4BC35F5-191C8F10] [03ae57836ec4ae2297f2739b8b86af35f6b9f92a] update_mapping [component] 2019-11-11 14:58:23,631+1100 INFO [elasticsearch[13E81695-81631669-25B736E4-B4BC35F5-191C8F10][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.metadata - [13E81695-81631669-25B736E4-B4BC35F5-191C8F10] [03ae57836ec4ae2297f2739b8b86af35f6b9f92a] update_mapping [component] 2019-11-11 14:58:24,027+1100 INFO [elasticsearch[13E81695-81631669-25B736E4-B4BC35F5-191C8F10][clusterService#updateTask][T#1]] *SYSTEM org.elasticsearch.cluster.metadata - [13E81695-81631669-25B736E4-B4BC35F5-191C8F10] [03ae57836ec4ae2297f2739b8b86af35f6b9f92a] update_mapping [component] 2019-11-11 14:58:42,211+1100 INFO [qtp1007621157-3926] *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for nuget_proxy changed from AVAILABLE to AUTO_BLOCKED_UNAVAILABLE until 2019-11-11T14:59:22.211+11:00 - reason Unauthorized for https://www.nuget.org 2019-11-11 14:58:42,211+1100 WARN [qtp1007621157-3926] *UNKNOWN com.sonatype.nexus.repository.nuget.internal.NugetFeedFetcher - Status code 401 contacting https://www.nuget.org/api/v2/FindPackagesById?id='Microsoft.Owin.Testing'&$top=80 2019-11-11 14:58:43,140+1100 INFO [SessionValidationThread-1] *UNKNOWN org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions... 2019-11-11 14:58:43,140+1100 INFO [SessionValidationThread-1] *UNKNOWN org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation. No sessions were stopped. 2019-11-11 14:59:23,399+1100 INFO [Check Status https://www.nuget.org] *UNKNOWN org.sonatype.nexus.repository.httpclient.internal.HttpClientFacetImpl - Repository status for nuget_proxy changed from AUTO_BLOCKED_UNAVAILABLE to AVAILABLE - reason n/a for n/a

jtom · November 12, 2019, 3:22pm

Hello,
I may be missing your Q but NXRM just does what you have configured.
So it can always use auth if you configure it too via System HTTP settings (see https://help.sonatype.com/display/NXRM3/System+Configuration#SystemConfiguration-HTTPandHTTPSRequestandProxySettings).
Otherwise it’s the individual proxy configuration which you could setup via the repository settings or not.
Otherwise it’s what your proxy expects.
If I’m not missing your Q, only you can answer what your system is setup to do=)
-Joe

rseddon · November 12, 2019, 7:21pm

I checked with my firewall admins. They needed info on how Nexus uses my proxy credentials. Does nexus use the proxy auth credentials on every request ?

Yes, it will use the configured HTTP proxy settings for all outbound requests. You’ll need the administrators of your firewall to check and see why it is rejecting access.

vishnuvp · November 13, 2019, 12:36am

Thank Joe. I am doing exactly as mentioned in the link you shared. The main issue is the proxy allowing artifact downloads only occasionally and blocking requests to download artifacts with 401 other times.
The proxy setup is shown below.

vishnuvp · November 13, 2019, 12:54am

Agree. Thanks Rich. So Nexus sends proxy credentials with every outbound request.

I told them “on every outbound request, the proxy details I set in the Nexus system is used. Is the account getting locked out due to successive requests.”

The firewall team investigated and told me

"I can see that that is not correct, the nexus system is NOT sending credentials with every outbound request.
Today the proxy sent 4x 401s to 10.18.192.225. This was due to request being sent without any credentials
Today I also see 40 successful requests where the credentials were provided "

So it appears like nexus is sending requests without creds. How do we debug and make sure whether that’s the case.

vishnuvp · November 21, 2019, 2:01am

Can you please advise whether I should be raising this as a bug since Nexus is sending requests without credentials sporadically.