You now have until September 15th to submit your use case or success story to the 2022 Dataiku Frontrunner Awards!ENTER YOUR SUBMISSION

SLL Error with "Deep Learning on images" Macro

brian-walheim
Level 1
Level 1
SLL Error with "Deep Learning on images" Macro

I'm using the latest version of the "Deep learning on images." When running the macro to install a pretrained model we get the following error:

2022-06-21 18:46:14,741 DEBUG Starting new HTTPS connection (1): storage.googleapis.com:443Traceback (most recent call last):  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 677, in urlopen    chunked=chunked,  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 381, in _make_request    self._validate_conn(conn)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn    conn.connect()  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connection.py", line 371, in connect    ssl_context=context,  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/util/ssl_.py", line 386, in ssl_wrap_socket    return context.wrap_socket(sock, server_hostname=server_hostname)  File "/usr/lib64/python3.6/ssl.py", line 365, in wrap_socket    _context=self, _session=session)  File "/usr/lib64/python3.6/ssl.py", line 776, in __init__    self.do_handshake()  File "/usr/lib64/python3.6/ssl.py", line 1036, in do_handshake    self._sslobj.do_handshake()  File "/usr/lib64/python3.6/ssl.py", line 648, in do_handshake    self._sslobj.do_handshake()ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)During handling of the above exception, another exception occurred:Traceback (most recent call last):  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/adapters.py", line 449, in send    timeout=timeout  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/connectionpool.py", line 727, in urlopen    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/urllib3/util/retry.py", line 446, in increment    raise MaxRetryError(_pool, url, error or ResponseError(cause))urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels.h5 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))During handling of the above exception, another exception occurred:Traceback (most recent call last):  File "/home/dataiku/install/dataiku-dss-9.0.3/python/dataiku/runnables/server.py", line 45, in serve    result = runnable.run(report_progress)  File "/tmp/tmp_folder_UXgxsBgR/dku_code.py", line 103, in run  File "/tmp/tmp_folder_UXgxsBgR/dku_code.py", line 79, in download_files_to_managed_folder  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py", line 75, in get    return request('get', url, params=params, **kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py", line 60, in request    return session.request(method=method, url=url, **kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py", line 533, in request    resp = self.send(prep, **send_kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py", line 646, in send    r = adapter.send(request, **kwargs)  File "/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/adapters.py", line 514, in send    raise SSLError(e, request=request)requests.exceptions.SSLError: HTTPSConnectionPool(host='storage.googleapis.com', port=443): Max retries exceeded with url: /tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels.h5 (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:877)'),))/home/dataiku/install/dataiku-dss-9.0.3/python/dataiku/runnables/server.py : serve (45)	result = runnable.run(report_progress)/tmp/tmp_folder_UXgxsBgR/dku_code.py : run (103)	/tmp/tmp_folder_UXgxsBgR/dku_code.py : download_files_to_managed_folder (79)	/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py : get (75)	return request('get', url, params=params, **kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/api.py : request (60)	return session.request(method=method, url=url, **kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py : request (533)	resp = self.send(prep, **send_kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/sessions.py : send (646)	r = adapter.send(request, **kwargs)/home/dataiku/design/code-envs/python/plugin_deeplearning-image_managed/lib/python3.6/site-packages/requests/adapters.py : send (514)	raise SSLError(e, request=request)

 Has anyone else encountered this issue


Operating system used: Linux

0 Kudos
3 Replies
sergeyd
Dataiker
Dataiker

Hi @brian-walheim 

Considering there is "SSL: CERTIFICATE_VERIFY_FAILED" while accessing storage.googleapis.com there are two possible root causes:

  • OS truststore has missing ROOT CA certs that can validate certs signed by Google Trust Services LLC 
  • there is an HTTP_PROXY in your env that interfere with SSL certs. 

You will need to work with your sysadmin/network team to fix that. You can check that you can successfully connect to this site with the next CLI command: 

curl -vvv https://storage.googleapis.com

Getting HTTP/1.1 400 Bad Request is fine but if you get SSL verification failed -> something environmental on your side. 

0 Kudos
brian-walheim
Level 1
Level 1
Author

I tried the curl command and got the following output

* About to connect() to storage.googleapis.com port 443 (#0)
*   Trying 142.250.81.208...
* Connected to storage.googleapis.com (142.250.81.208) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* Server certificate:
*       subject: CN=storage.googleapis.com
*       start date: Apr 19 17:49:51 2022 GMT
*       expire date: Jun 27 17:49:51 2024 GMT
*       common name: storage.googleapis.com
*       issuer: E=acsgmgmt@ngc.com,CN=Northrop Grumman Proxy CA,O=Northrop Grumman,L=McLean,C=us
* NSS error -8162 (SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE)
* The certificate issuer's certificate has expired. Check your system date and time.
* Closing connection 0
curl: (60) The certificate issuer's certificate has expired. Check your system date and time.
More details here: http://curl.haxx.se/docs/sslcerts.html

 

0 Kudos
sergeyd
Dataiker
Dataiker

Hi @brian-walheim 

Thanks for the test. So this indeed confirmes that that SSL cert in your proxy has expired. Notice the issuer: 

issuer: E=acsgmgmt@ngc.com,CN=Northrop Grumman Proxy CA,O=Northrop Grumman,L=McLean,C=us
* NSS error -8162 (SEC_ERROR_EXPIRED_ISSUER_CERTIFICATE)
* The certificate issuer's certificate has expired

While here is the example what it should be like: 

curl -vvv https://storage.googleapis.com
* About to connect() to storage.googleapis.com port 443 (#0)
*   Trying 74.125.193.128...
* Connected to storage.googleapis.com (74.125.193.128) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* 	subject: CN=storage.googleapis.com
* 	start date: May 30 11:27:43 2022 GMT
* 	expire date: Aug 22 11:27:42 2022 GMT
* 	common name: storage.googleapis.com
* 	issuer: CN=GTS CA 1C3,O=Google Trust Services LLC,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: storage.googleapis.com
> Accept: */*

 

0 Kudos

Labels

?
Labels (1)
A banner prompting to get Dataiku