Certificate Verification Failing with urllib3>=1.25

[joshowen]

The following error happens with urllib3>=1.25, and is not present with urllib<1.25. For what it's worth, this was triggered with the NewRelic Airflow metrics package.

scheduler    | [2020-03-02 21:37:22,914] {harvester.py:74} ERROR - New Relic send_batch failed with an exception.
scheduler    | Traceback (most recent call last):
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
scheduler    |     cnx.do_handshake()
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1934, in do_handshake
scheduler    |     self._raise_ssl_error(self._ssl, result)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/OpenSSL/SSL.py", line 1671, in _raise_ssl_error
scheduler    |     _raise_current_error()
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
scheduler    |     raise exception_type(errors)
scheduler    | OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
scheduler    | 
scheduler    | During handling of the above exception, another exception occurred:
scheduler    | 
scheduler    | Traceback (most recent call last):
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
scheduler    |     chunked=chunked,
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/newrelic/hooks/external_urllib3.py", line 18, in _nr_wrapper_make_request_
scheduler    |     return wrapped(*args, **kwargs)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/newrelic/hooks/external_urllib3.py", line 18, in _nr_wrapper_make_request_
scheduler    |     return wrapped(*args, **kwargs)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
scheduler    |     self._validate_conn(conn)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
scheduler    |     conn.connect()
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connection.py", line 360, in connect
scheduler    |     ssl_context=context,
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
scheduler    |     return context.wrap_socket(sock, server_hostname=server_hostname)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
scheduler    |     raise ssl.SSLError("bad handshake: %r" % e)
scheduler    | ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)
scheduler    | 
scheduler    | During handling of the above exception, another exception occurred:
scheduler    | 
scheduler    | Traceback (most recent call last):
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/newrelic_airflow_plugin/harvester.py", line 67, in _loop
scheduler    |     response = client.send_batch(items, common=common)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/newrelic_telemetry_sdk/client.py", line 170, in send_batch
scheduler    |     return self._pool.urlopen("POST", self.URL, body=payload, headers=headers)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
scheduler    |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/util/retry.py", line 376, in increment
scheduler    |     raise six.reraise(type(error), error, _stacktrace)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
scheduler    |     raise value.with_traceback(tb)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
scheduler    |     chunked=chunked,
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/newrelic/hooks/external_urllib3.py", line 18, in _nr_wrapper_make_request_
scheduler    |     return wrapped(*args, **kwargs)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/newrelic/hooks/external_urllib3.py", line 18, in _nr_wrapper_make_request_
scheduler    |     return wrapped(*args, **kwargs)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 376, in _make_request
scheduler    |     self._validate_conn(conn)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
scheduler    |     conn.connect()
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/connection.py", line 360, in connect
scheduler    |     ssl_context=context,
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/util/ssl_.py", line 370, in ssl_wrap_socket
scheduler    |     return context.wrap_socket(sock, server_hostname=server_hostname)
scheduler    |   File "/opt/conda/envs/airflow_env/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
scheduler    |     raise ssl.SSLError("bad handshake: %r" % e)
scheduler    | urllib3.exceptions.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)```

[a-feld]

Hi @joshowen ! From that stack trace it looks like the SSL certificate received by your application when communicating with New Relic is failing to validate.

urllib3 started requiring certificate verification by default for SSL connections in versions >=1.25.

Do you have a CA certificate bundle installed on the application machine? The SDK currently makes use of the default system certificate stores so if there aren't any certificates, that would explain the failure.

[a-feld]

Hi @joshowen , I'm going to close this issue for now since it appears the the SDK is operating as intended, but feel free to reopen if we haven't properly addressed this.

The preferred way to fix this issue is to install os level certificates. However, another way to work around this issue is to use the certifi package and override the ca_certs argument to the urllib3 connection pool.

pip install certifi

import os

import certifi

from newrelic_telemetry_sdk import GaugeMetric, MetricClient


class CertifiConnectionPool(MetricClient.POOL_CLS):
    def __init__(self, *args, **kwargs):
        # Use certificates from certifi
        kwargs["ca_certs"] = certifi.where()
        super(CertifiConnectionPool, self).__init__(*args, **kwargs)


class CertifiMetricClient(MetricClient):
    POOL_CLS = CertifiConnectionPool


metric = GaugeMetric("hello_world", 1)
client = CertifiMetricClient(os.environ["NEW_RELIC_INSERT_KEY"])
response = client.send(metric)
response.raise_for_status()
print("Metric sent!")

[ashwini-streamotion]

Hi,
I am facing the same issue.
I have OS level certs in my system, but the sdk does not pick it up.
If I change this line from:
self._pool = pool = self.POOL_CLS( host=host, port=443, retries=retries, headers=headers, strict=True )
to
self._pool = pool = self.POOL_CLS( host=host, port=443, retries=retries, headers=headers, strict=True, ca_certs='/etc/ssl/cert.pem' )
every thing starts working normally.
Also, I am hitting the issue only when pyOpenSSL lib is installed. Upon uninstalling that lib, the issue gets resloved. Alternatively, using
urllib3.contrib.pyopenssl.extract_from_urllib3()
also resolves the issue.

[a-feld]

Hi folks, I just wanted to give an update - this looks related to urllib3+pyopenssl. We have a reproduction using pure urllib3+pyopenssl:

import urllib3
from urllib3.contrib.pyopenssl import inject_into_urllib3

inject_into_urllib3()


conn = urllib3.connection_from_url("https://example.com", retries=False)
response = conn.request("GET", "/")
conn.close()
print(response.status)

Unfortunately, we haven't had the opportunity to investigate further but if anybody in the community knows what's going on, feel free to chime in! 😄

[joshowen]

This is likely being monkeypatched for most folks by requests. They're removing the default monkeypatching functionality in the next release (psf/requests#5443), but I'm sure plenty of packages have older versions of requests vendored in, so this will likely be an issue for a while.

[a-feld]

Fixed upstream in psf/requests#5443