UNABLE TO VERIFY FIRST CERTIFICATE

Certificate Issue Breaking Our Automation

I recently ran into an issue with a customer where we were monitoring the public internet connectivity to one of their public facing websites. One day the network tests were executing as expected, and the next we saw all tests failing due to some obscure SSL error.

This didn’t seem to be affecting their production users, but those users would be navigating to the website via a browser like Chrome. This was, however, breaking our monitoring tool and potentially any other automation that may be calling this site as well.

When I ran a curl command towards this domain from the terminal, I would see the error below. To determine what was causing this, I started collecting packet captures from both the browser and command line use case.

curl: (60) SSL certificate problem: unable to get local issuer certificate

More details here: https://curl.se/docs/sslcerts.html



curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it.  To learn more about this situation and how to fix it, please visit the web page mentioned above.

In the packet capture output from my curl command, I noticed that the TLS negotiation completed but was immediately followed by an alert and a client side TCP reset.

Below is the browser based connectivity. As you can see, my client IP (172.16.16.31) resolved the domain to IP > Client Hello sent to server > Server Hello sent back to Client > Certificate, Server Key Exchange, Server Hello Done all completes as expected.

I started digging into the certificate itself to see if anything popped out. The certificate appeared to be issued by DigiCert SHA2 Secure Server CA. This is one of DigiCert’s Intermediate CAs. Typically I would expect to see a few different certificate entries in this step, the largest of which being the entire certificate chain. This would include the DigiCert Root CA, the DigiCert SHA2 Secure Server CA (signed by the root CA), and the server certificate (signed by the intermediate CA).

In this case the Root CA was not present, meaning our monitoring tool needed to have the Intermediate CA in its host trust store otherwise it would see the server certificate as well as the intermediate CA as untrustworthy.

This makes sense to me so far, but why would the same not be true for a browser based user coming from the same machine?

In the certificate itself, I noticed an extension (id-pe-authorityInfoAccess) which listed a URI. This URI can be called by the browser automatically to validate the Intermediate CA in the event it is not listed in the trust store already. To validate this is what was happening, I ran a ping to the FQDN “cacerts.digicert.com” and saw it resolved to 192.229.211.108.

mymacbook ~ % ping cacerts.digicert.com

PING fp2e7a.wpc.phicdn.net (192.229.211.108): 56 data bytes

64 bytes from 192.229.211.108: icmp_seq=0 ttl=55 time=33.565 ms

64 bytes from 192.229.211.108: icmp_seq=1 ttl=54 time=21.520 ms

64 bytes from 192.229.211.108: icmp_seq=2 ttl=54 time=31.673 ms

^C 

I then went back to my pcap and filtered based on this IP. I saw that during the TLS negotiation, the browser automatically called this URI to validate the Intermediate CA as trusted.

This last step was missing from my curl-based use case. In that event, the client would show a “Fatal” alert message due to Unknown CA before sending a TCP RST. This is because it had no logic built-in/no way to check the validity of the Intermediate CA other than what existed in it’s own trust store. In order to fix this, I could either:

  1. upload the Intermediate CA and/or server certificate to my host’s trust store
  2. tell the client to fix the issue on their end by ensuring the complete certificate chain was configured correctly (preferred method)

For option 1, I considered this as the least preferred option. Certificates have an expiry date, and this would potentially come up again if the customer did not fix the cert chain issue in their next certificate rotation.

Option 2 would be the “correct” way of addressing this, as we would not need to be manually managing the certificates on a per host basis.

Leave a Comment

Your email address will not be published. Required fields are marked *