Let's Encrypt Revoking Certificates

Happening Now: Over 2 Percent of Sites Using a Let’s Encrypt TLS Certificate May Throw Security Warnings

On Wednesday, March 4, 2020, 3 million Transport Layer Security (TLS) certificates issued by Let’s Encrypt will be revoked because of a Certificate Authority Authorization (CAA) bug. This is 2.6% of the over 116 million active certificates issued by Let’s Encrypt.

Let’s Encrypt has contacted all certificate holders affected by this bug, and they’ve created a tool and a list of serial numbers to determine if your TLS certificate is affected by the bug.

Let’s Encrypt have not set an exact time for revocation of the certificates, however, they say that the earliest timeframe will be UTC 00:00.

Some certificate holders have received emails that they’re affected, but they may have received that alert erroneously, either because the certificate was issued in the last few days after the bug was fixed, or by not meeting certain timing criteria necessary for the bug to trigger, adding to confusion.

How to tell if you’re affected

Let’s Encrypt created a tool where you can check your site’s host name and determine if your Let’s Encrypt-issued certificate is affected by this bug.

Let’s Encrypt can also see the list of all affected serial numbers.

On a Linux/BSD-like system, you can also run the following command to show your domain’s current certificate serial number. Replace example.com below with your own domain name:
openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

If your hosting provider provided a certificate for your website, they were likely the ones contacted by Let’s Encrypt. Numerous site owners have received notifications from hosting providers that they would be handling the reissuance of those certificates.

If you have created your own Let’s Encrypt certificate, you will need to update yourself if you are affected.

What will happen if I don’t fix this?

A secure TLS certificate ensures that your site visitors have encrypted traffic between their browsers and your website. Site visitors might see a certificate revoked error, a “not secure” warning, or other security warnings in their browser that may erode trust in your site.

What happened in technical terms?

Boulder, the software builder used by Let’s Encrypt’s CA, checks CAA records for a domain name at the same time that it verifies that a certificate requester controls that domain. Most subscribers to the service issue a certificate immediately after they validate domain control, however Let’s Encrypt trusts that validation for 30 days. Due to that trust, they sometimes have to recheck CAA records a second time, just prior to issuing the certificate. The timeframe for rechecking is 8 hours, meaning that any domain name validated more than 8 hours ago requires a recheck.

According to Let’s Encrypt:

The bug: when a certificate request contained N domain names that needed CAA rechecking, Boulder would pick one domain name and check it N times. What this means in practice is that if a subscriber validated a domain name at time X, and the CAA records for that domain at time X allowed Let’s Encrypt issuance, that subscriber would be able to issue a certificate containing that domain name until X+30 days, even if someone later installed CAA records on that domain name that prohibit issuance by Let’s Encrypt.

Let’s Encrypt confirmed the bug at 2020-02-29 03:08 UTC, and halted issuance two minutes later. They deployed a fix at 05:22 UTC and re-enabled certificate issuance at that time.

According to security researcher Scott Helme, who posted his investigation on Twitter:

Does this mean we should use something other than Let’s Encrypt for SSL certificates?

Let’s Encrypt have been very transparent about this bug, both in identifying the problem themselves and reporting the CA incident. They are acting exactly how a certificate authority should act. As such, we are confident that Let’s Encrypt is still a good source for TLS certificates.

You can find details of the bug on the Let’s Encrypt bug tracker.

Did you enjoy this post? Share it!

Comments

5 Comments
  • How about if we use Cloudflare with our LetsEncrypt cert... it makes it more difficult to check against the list using these tools since the request will examin CF's cert, but when the LetsEncrypt certs starts failing Cloudflare will start complaining and also fail?

    • Hi Ben! If your site is behind Cloudflare, you may still be able to test the origin server's certificate by using the origin IP in the openssl command. In the code snippet in the post, replace -connect example.com:443 with -connect [ip_address]:443. Don't use the IP for -servername example.com though, as this tells your server which SNI hostname you're looking up a cert for.

      If that doesn't work, you can still manually attempt to reissue certificates on the server side just to be sure.

  • >openssl s_client -connect example.com:443 -servername example.com -showcerts /dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

    s_client: must not provide both -connect option and target parameter
    s_client: Use -help for summary.
    unable to load certificate

    • Hi there! Some angle brackets in the original code snippet were incorrectly being treated as an HTML tag, so part of the command was stripped out. We've resolved that at this time, so it should behave with the updated content. Thanks for pointing it out!

  • What a relief... Investing in Wordfence is one of the best decisions taken by us for our website.

    Thank you... please keep up the good work...