SLL Error with "Deep Learning on images" Macro

[brian-walheim]

I'm using the latest version of the "Deep learning on images." When running the macro to install a pretrained model we get the following error:

2022-06-21 18:46:14,741 DEBUG Starting new HTTPS connection (1): storage.googleapis.com:443Traceback (most recent call last):  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen    chunked=chunked,  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request    self._validate_conn(conn)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn    conn.connect()  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect    ssl_context=context,  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket    return context.wrap_socket(sock, server_hostname=server_hostname)  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket    _context=self, _session=session)  File "/usr/lib64/python3.6/ssl.py", line 776, in __init__    self.do_handshake()  File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake    self._sslobj.do_handshake()  File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake    self._sslobj.do_handshake()ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)During handling of the above exception, another exception occurred:Traceback (most recent call last):  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/adapters.py", line 449, in send    timeout=timeout  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/util/retry.py", line 446, in increment    raise MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels.h5 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))During handling of the above exception, another exception occurred:Traceback (most recent call last):  File "/home/dataiku/install/dataiku-dss-9.0.3/python/dataiku/runnables/server.py", line 45, in serve    result = runnable.run(report_progress)  File "/tmp/tmp_folder_UXgxsBgR/dku_code.py", line 103, in run  File "/tmp/tmp_folder_UXgxsBgR/dku_code.py", line 79, in download_files_to_managed_folder  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py", line 75, in get    return request('get', url, params=params, **kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py", line 60, in request    return session.request(method=method, url=url, **kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py", line 533, in request    resp = self.send(prep, **send_kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py", line 646, in send    r = adapter.send(request, **kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/adapters.py", line 514, in send    raise SSLError(e, request=request)requests.exceptions.SSLError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels.h5 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))/home/dataiku/install/dataiku-dss-9.0.3/python/dataiku/runnables/server.py : serve (45)	result = runnable.run(report_progress)/tmp/tmp_folder_UXgxsBgR/dku_code.py : run (103)	/tmp/tmp_folder_UXgxsBgR/dku_code.py : download_files_to_managed_folder (79)	/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py : get (75)	return request('get', url, params=params, **kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py : request (60)	return session.request(method=method, url=url, **kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py : request (533)	resp = self.send(prep, **send_kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py : send (646)	r = adapter.send(request, **kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/adapters.py : send (514)	raise SSLError(e, request=request)

 Has anyone else encountered this issue

---

Operating system used: Linux

[sergeyd]

Hi @brian-walheim 

Considering there is "SSL: CERTIFICATE_VERIFY_FAILED" while accessing storage.googleapis.com there are two possible root causes:

• OS truststore has missing ROOT CA certs that can validate certs signed by Google Trust Services LLC 
• there is an HTTP_PROXY in your env that interfere with SSL certs. 

You will need to work with your sysadmin/network team to fix that. You can check that you can successfully connect to this site with the next CLI command: 

curl -vvv https://storage.googleapis.com

Getting HTTP/1.1 400 Bad Request is fine but if you get SSL verification failed -> something environmental on your side. 

[brian-walheim]

I tried the curl command and got the following output

* About to connect() to storage.googleapis.com port 443 (#0)
*   Trying 142.250.81.208...
* Connected to storage.googleapis.com (142.250.81.208) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=storage.googleapis.com
*       start date: Apr 19 17:49:51 2022 GMT
*       expire date: Jun 27 17:49:51 2024 GMT
*       common name: storage.googleapis.com
*       issuer: E=acsgmgmt@ngc.com,CN=Northrop Grumman Proxy CA,O=Northrop Grumman,L=McLean,C=us
* NSS error -8162 (SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE)
* The certificate issuer's certificate has expired. Check your system date and time.
* Closing connection 0
curl: (60) The certificate issuer's certificate has expired. Check your system date and time.
More details here: http://curl.haxx.se/docs/sslcerts.html

[sergeyd]

Hi @brian-walheim 

Thanks for the test. So this indeed confirmes that that SSL cert in your proxy has expired. Notice the issuer: 

issuer: E=acsgmgmt@ngc.com,CN=Northrop Grumman Proxy CA,O=Northrop Grumman,L=McLean,C=us
* NSS error -8162 (SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE)
* The certificate issuer's certificate has expired. 

While here is the example what it should be like: 

curl -vvv https://storage.googleapis.com
* About to connect() to storage.googleapis.com port 443 (#0)
*   Trying 74.125.193.128...
* Connected to storage.googleapis.com (74.125.193.128) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=storage.googleapis.com
* 	start date: May 30 11:27:43 2022 GMT
* 	expire date: Aug 22 11:27:42 2022 GMT
* 	common name: storage.googleapis.com
* 	issuer: CN=GTS CA 1C3,O=Google Trust Services LLC,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: storage.googleapis.com
> Accept: */*