OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed - https://knapsackpro.com/?utm_source=github&utm_medium=gist&utm_campaign=github-gist-openssl-certificate-verify-failed

openssl-certificate-verify-failed.md

OpenSSL::SSL::SSLError SSL_connect returned=1 errno=0 state=error: certificate verify failed

root issue

Let's Encrypt certificates may be affected by the recent expiry (2021-09-30) of their old root certificate. https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Requirements:

1. all clients of your API must trust ISRG Root X1 (not just DST Root CA X3), and
2. if clients of your API are using OpenSSL, they must use version 1.1.0 or later.

You may need to upgrade OS and OpenSSL on your CI machine.

Here is a list of compatible OS: https://letsencrypt.org/docs/certificate-compatibility/

How to solve it?

Upgrade your openssl version >= 1.1.0 and OS.

If you are still seeing issues then most likely your application is configured to not use operating system certificates. For instance you use certified gem. See comments to this gist.

More ideas

The other most likely issue could be a change on your CI server. For instance, if you use Docker then maybe something changed in the Docker image or one of its layers and you might use an outdated openssl library. Or you use old Ruby version that depends on the old OpenSSL system library?

You can run the following code to check what TLS version is used in your project.

Please ensure you run the code in the rails environment on your CI. You can do this either through rails console, or by running a script using rails runner

require 'net/https'
require 'json'
uri = URI("https://www.howsmyssl.com/")
http = ::Net::HTTP.new uri.host, uri.port
http.use_ssl = true
data = JSON.parse(http.get('/a/check').body)
puts "TLS Version Negotiated: #{data['tls_version']}"
puts "Open SSL version: #{OpenSSL::OPENSSL_VERSION}"

In my case it prints code: TLS Version Negotiated: TLS 1.3

Another thing you can check is openssl version: OpenSSL::OPENSSL_VERSION

knapsack_pro gem uses Net::HTTP from Ruby standard library to make requests. This depends on OpenSSL library in OS. https://github.com/KnapsackPro/knapsack_pro-ruby/blob/e0b9baea5a5c3c4f65e924fac5d9dddac9f2f711/lib/knapsack_pro/client/connection.rb#L145 So it's the error on openssl/ruby level and we can't change OpenSSL version in your OS from knapsack_pro gem level.

Most likely you need to update openssl in CI server so that Ruby uses the proper openssl library and you need to update your operating system in order to trust ISRG Root X1.

If you use Debian on CI, you may want to update docker image to a newer version or you could use Ubuntu to use proper OpenSSL version.

We were confused by this because we could connect to knapsack via curl and IRB in our CI docker image.
It turned out that the Certified gem was enabled for development and tests. This gem forces Net::HTTP to use a bundled set of certificates instead of the OS certificates. We removed the gem and our test suite connects again.
We discovered this was the root issue by checking what certificate Net::HTTP was using -

require 'net/https'
require 'json'
uri = URI("https://www.howsmyssl.com/")
http = ::Net::HTTP.new uri.host, uri.port
http.use_ssl = true
http.ca_file # should be nil to use the operation system certificates

@MarkyMarkMcDonald How did you verify that you are not using operation system certificates? Thanks.