Earlier today, we experienced a brief service downgrade caused by an SSL certificate issue. As a result, access to the dashboard and some of our APIs were interrupted, and you might have received emails about communication interruption with your servers.
No customer server or service was affected as a result of this issue.
Here is a quick summary of what happened.
We use Let's Encrypt wildcard certificates for some of our primary HTTP endpoints. Today one of our certificates was expired, and the script responsible for renewal failed to do so automatically. Our team had to reissue the certificate manually and restore the service.
LE wildcard certificates use DNS records for verification, and our script failed to update the DNS record initially, which delayed the renewal. Once the script finally updated the record and the certificate was issued, a bug in the script used the TLS key, and the TLS certificate issued instead of the full chain key. Since the root CA at LE was changed recently, the new chain certificate was not valid, which caused some of our services to remain left without a valid certificate on some browsers and clients (including the CX toolbelt).
We've now updated the full chain certificate and valid certificates back all services. If you are experiencing any issues, please let us know.
We are sorry for the inconvenience and the interruption in our service caused by this issue today. We are working to strengthen our process and policies around this to ensure it won't happen again in the future.