Certificate Issues with new Maven Repo?

mogoodrich · October 27, 2017, 1:45pm

Hello all–

fyi, we seem to running into some certificate issues contacting the Maven Repo when running builds, which I assume is happening for some reason due to the Maven migration?

27-Oct-2017 03:11:38	[ERROR] The build could not read 1 project -> [Help 1]
27-Oct-2017 03:11:38	[ERROR]   
27-Oct-2017 03:11:38	[ERROR]   The project org.openmrs.module:webservices.rest-omod:2.22.0-SNAPSHOT (/opt/bamboo5.9/xml-data/build-dir/MIREBALAIS-RWSM-JOB1/omod/pom.xml) has 1 error
27-Oct-2017 03:11:38	[ERROR]     Unresolveable build extension: Plugin org.openmrs.maven.plugins:maven-openmrs-plugin:1.0.1 or one of its dependencies could not be resolved: Could not transfer artifact org.openmrs.maven.plugins:maven-openmrs-plugin:jar:1.0.1 from/to openmrs-repo (http://mavenrepo.openmrs.org/nexus/content/repositories/public): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 2]

Workaround is to add artifactory as a profile in:

<?xml version="1.0" encoding="UTF-8"?>
<settings xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd" xmlns="http://maven.apache.org/SETTINGS/1.1.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <servers>
    <server>
      <username>${env.BINTRAY_USER}</username>
      <password>${env.BINTRAY_API_KEY}</password>
      <id>bintray-sdk</id>
    </server>
  </servers>
  <profiles>
    <profile>
      <repositories>
        <repository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>public</name>
          <url>https://openmrs.jfrog.io/openmrs/public</url>
        </repository>
        <repository>
          <snapshots />
          <id>snapshots</id>
          <name>public</name>
          <url>https://openmrs.jfrog.io/openmrs/public</url>
        </repository>
      </repositories>
      <pluginRepositories>
        <pluginRepository>
          <snapshots>
            <enabled>false</enabled>
          </snapshots>
          <id>central</id>
          <name>public</name>
          <url>https://openmrs.jfrog.io/openmrs/public</url>
        </pluginRepository>
        <pluginRepository>
          <snapshots />
          <id>snapshots</id>
          <name>public</name>
          <url>https://openmrs.jfrog.io/openmrs/public</url>
        </pluginRepository>
      </pluginRepositories>
      <id>artifactory</id>
    </profile>
  </profiles>
  <activeProfiles>
    <activeProfile>artifactory</activeProfile>
  </activeProfiles>
  <pluginGroups>
    <pluginGroup>org.openmrs.maven.plugins</pluginGroup>
  </pluginGroups>
</settings>

Take care, Mark

wyclif · October 27, 2017, 3:05pm

I think I ran into a similar issue with the uicommons module, but seems like it’s a local because the travis build passed.

david5780 · October 27, 2017, 5:03pm

I believe we ran into this same issue today with all of our module builds. The Bamboo server was upgraded yesterday, which is when it started happening for us.

Thanks, Dave

cintiadr · October 28, 2017, 5:01am

By any chance you have a JAVA < 8u101?

cintiadr · October 28, 2017, 5:09am

Chances are it’s related to Maven repository outage/migration

david5780 · October 30, 2017, 11:48am

We are using java 7.0.79. The builds run fine when building locally, just not when running from Bamboo.

We were able to get them to run from Bamboo by pointing the m2 settings file to

https://openmrs.jfrog.io/openmrs/public

raff · October 30, 2017, 12:02pm

@david5780, any chance you could upgrade to java 7.0.111 (7u111) at least and report, if it fixed the issue?

raff · October 30, 2017, 12:03pm

@cintiadr, didn’t we use letsencrypt for mavenrepo-old.openmrs.org ?

david5780 · October 30, 2017, 12:19pm

We would not be able to update to java 7.0.111. The Bamboo server is not maintained by our group. Also, I think 7.0.80 is the latest publicly available version. My understanding is that 7.0.111 is a paid version. This doesn’t seem like a java issue since we are able to build fine locally.

mogoodrich · October 30, 2017, 3:38pm

This is still happening for us, unfortunately, but only in some cases, and it’s unclear what the pattern is. Most of the time it works, but when it fails, it fails consistently. For instance, every time our bamboo tries to deploy the Visit Documents module snapshot we are seeing this error:

build 30-Oct-2017 11:25:56 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-deploy-plugin:2.7:deploy (default-deploy) on project visitdocumentsui: Failed to retrieve remote metadata org.openmrs.module:visitdocumentsui:1.4-SNAPSHOT/maven-metadata.xml: Could not transfer metadata org.openmrs.module:visitdocumentsui:1.4-SNAPSHOT/maven-metadata.xml from/to openmrs-repo-snapshots (http://mavenrepo.openmrs.org/nexus/content/repositories/snapshots): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]

However, it’s able to deploy other module snapshots correctly, and the “snapshot” section in the distribution management section of both poms appear to be identical.

On a related note, in case this is helpful, at times our builds are failing because it can’t find a dependency (which does exist):

[ERROR] Failed to execute goal on project mirebalais-api: Could not resolve dependencies for project org.openmrs.module:mirebalais-api:jar:1.2-SNAPSHOT: The following artifacts could not be resolved: org.openmrs.module:pacsintegration-api:jar:1.5-SNAPSHOT, org.openmrs.module:emr-api:jar:2.1-SNAPSHOT, org.openmrs.module:coreapps-api:jar:1.18.0-SNAPSHOT, org.openmrs.module:allergyapi-api:jar:1.5-SNAPSHOT, org.openmrs.module:edtriageapp-api:jar:1.0-SNAPSHOT, org.openmrs.module:labtrackingapp-api:jar:1.0.0-SNAPSHOT: Could not find artifact org.openmrs.module:pacsintegration-api:jar:1.5-SNAPSHOT → [Help 1]

These failures appear to be transitory though… a rerun and generally things work.

Hope this helps and doesn’t just make things more confusing…

Take care, Mark

sjmckee · October 30, 2017, 3:56pm

We found changing references from the old repository URL to the new one helped.

http://mavenrepo.openmrs.org/nexus/content/repositories/snapshots -> https://openmrs.jfrog.io/openmrs/snapshots/

I’m not sure if it will help in your case or not, but it might be worth a try. I think there is something strange happening how Bamboo handles redirects.

We had the old URL specified in our settings.xml file, but as soon as we switched them to the new one all the builds started working again.

Steve

cintiadr · October 30, 2017, 11:25pm

Nope, before all the automation, we are using a paid/manual certificate. You know, those you pay and get the certificate back a couple of days later, without autorenew and stuff. We sorta need to get rid of it. But letsencrypt is so standard these days that it didn’t even occur to me that some java truststores wouldn’t have it.

It’s a little tricky for me to diagnose problems when I don’t have access to the machines. I suppose bamboo is not accessible from the internet, right? I suppose you might have multiple bamboo agents. Would you be able to check if there’s any pattern (a certain agent always fails, or always succeeds)? Also, if you have multiple agents, there’s a chance they have slightly different maven versions or java updates.

Can you please run a build ‘mvn --version ; java -version’ in all of them? Also, could you run that locally for me please? I will try to reproduce it locally, but I’d need the very specific version to attempt.

I suppose there’s a pretty good reason why you are running it with java 7, and you wouldn’t be able to try it with java 8.

The URL on the settings.xml should only affect downloads, not uploads. The only thing relevant there for uploads are the credentials. So it’s very weird. It could indeed be a problem with maven, but it’s very confusing the whole thing.

Last by not least, I’m really sorry for all the problems. When we automate and upgrade all the systems, some teething problem is bound to happen. It’s a shame it’s with an external system, so I don’t really have the power to fix it by myself.

mogoodrich · October 31, 2017, 3:34pm

Cintia,

First of, no worries! Generally the upgrade has gone really smoothly, and it’s worked in most cases. I realize that problems are to be expected.

And I think I’ve figured this one out… was an issue on my end… I think there was a old version of Java 8 (1.8.0_60) that it was picking up, even though I had also installed 1.8.0_151… I’ve cleaned it up and confirmed that I can now deploy the Visit Documents module… so weird though that it was only happening in some cases, though…

I’m running through our whole pipeline now to make sure everything still works, fingers crossed!

Take care, Mark

dkayiwa · November 1, 2017, 9:23pm

@mogoodrich does the whole pipeline work well? It looks like am running into the same problem while trying to build the latest version of the reference application distro.

@deacostac seems to be running into the same here: Problems with the installation of openmrs openmrs-api

mogoodrich · November 1, 2017, 9:53pm

Yes, our whole pipeline is now working, after I made sure I was running Java 1.8_151.

I think it relates to the letsencrypt bug mentioned above… and you need to make sure you are using a high enough build of Java 8. (I’m assuming Java 7 would still work, as long as you have a more recent build of it as well).

I’m still having problems deploying from my laptop as I mentioned in another thread, but that seems to be a different issue.

Take care, Mark

dkayiwa · November 1, 2017, 10:22pm

The problem got fixed after upgrading from jdk1.8.0_66 to jdk1.8.0_152

Thanks @mogoodrich for sharing the fix!

tahira · February 7, 2018, 6:17am

For me, it worked on jdk1.8.0_162. Just in case if anyone is struggling like me

Thanks everyone!

vamsi77 · March 12, 2018, 12:05pm

How to resolve this if we are using the JDK 1.7 (7u80)?

raff · March 12, 2018, 12:14pm

See https://stackoverflow.com/a/35454903

vamsi77 · March 12, 2018, 12:29pm

Thank you, that worked.