Python Code Env SSL: CERTIFICATE_VERIFY_FAILED

JIshikawa

Hi,

I am trying to install the required code environment for object detection using Python 3.7 (internal deployment).

The installation almost completes but keeps failing at the same point when Dataiku tries to install packages from torchvision, caused by an SSL Certification error as it seems.

I have already added extra options for 'pip install' to trust hosts but it somehow doesn't work.

 

Is there a way to run something like this on a DSS python env?

Certificates.command

Here is the error log

Installed kernelspec py-dku-venv-internal_object_detection_v1 in /Users/XXXXX/Library/DataScienceStudio/dss_home/jupyter-run/jupyter/kernels/py-dku-venv-internal_object_detection_v1/Users/XXXXX/Library/DataScienceStudio/dss_home/code-envs/python/INTERNAL_object_detection_v1/lib/python3.7/site-packages/sklearn/utils/validation.py:37: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.  LARGE_SPARSE_SUPPORTED = LooseVersion(scipy_version) >= '0.14.0'Downloading: "https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth" to /Users/XXXXX/Library/DataScienceStudio/dss_home/code-envs/resources/python/INTERNAL_object_detection_v1/pytorch/hub/checkpoints/fasterrcnn_resnet50_fpn_coco-258fb6c6.pthTraceback (most recent call last):  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 1350, in do_open    encode_chunked=req.has_header('Transfer-encoding'))  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/http/client.py", line 1281, in request    self._send_request(method, url, body, headers, encode_chunked)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/http/client.py", line 1327, in _send_request    self.endheaders(body, encode_chunked=encode_chunked)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/http/client.py", line 1276, in endheaders    self._send_output(message_body, encode_chunked=encode_chunked)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/http/client.py", line 1036, in _send_output    self.send(msg)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/http/client.py", line 976, in send    self.connect()  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/http/client.py", line 1451, in connect    server_hostname=server_hostname)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/ssl.py", line 423, in wrap_socket    session=session  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/ssl.py", line 870, in _create    self.do_handshake()  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/ssl.py", line 1139, in do_handshake    self._sslobj.do_handshake()ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)During handling of the above exception, another exception occurred:Traceback (most recent call last):  File "/Users/XXXXX/Library/DataScienceStudio/dss_home/tmp/code-env-resources/INTERNAL_object_detection_v1-resources_init-WDk3vjamEY9L.py", line 19, in <module>    torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)  File "/Users/XXXXX/Library/DataScienceStudio/dss_home/code-envs/python/INTERNAL_object_detection_v1/lib/python3.7/site-packages/torchvision/models/detection/faster_rcnn.py", line 371, in fasterrcnn_resnet50_fpn    progress=progress)  File "/Users/XXXXX/Library/DataScienceStudio/dss_home/code-envs/python/INTERNAL_object_detection_v1/lib/python3.7/site-packages/torch/hub.py", line 571, in load_state_dict_from_url    download_url_to_file(url, cached_file, hash_prefix, progress=progress)  File "/Users/XXXXX/Library/DataScienceStudio/dss_home/code-envs/python/INTERNAL_object_detection_v1/lib/python3.7/site-packages/torch/hub.py", line 437, in download_url_to_file    u = urlopen(req)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 222, in urlopen    return opener.open(url, data, timeout)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 525, in open    response = self._open(req, data)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 543, in _open    '_open', req)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 503, in _call_chain    result = func(*args)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 1393, in https_open    context=self._context, check_hostname=self._check_hostname)  File "/Users/XXXXX/Library/DataScienceStudio/Python/python3.7-20220516/lib/python3.7/urllib/request.py", line 1352, in do_open    raise URLError(err)urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)>

---

Operating system used: Mac OS 12.6.2

---

Operating system used: Mac OS 12.6.2

AlexandreV

Hello again @JIshikawa

I believe my previous answer will not solve your issue, I'm sorry about that. Here is a workaround for object detection and image classification code envs. It necessitates a python3.6 from python.org:

On your mac:

• Please install a python3.6 from Python.org
• Make sure that calling /usr/local/bin/python3.6 from a terminal opens a python prompt (you can close the terminal afterwards)

In DSS

• Delete the internal_object_detection_v1 code env (Administration > Code Envs and click on the trash can on the right corresponding to the code env)
• Create a PYTHON36 object detection code env (from Administration > Settings > Misc. > DSS internal code environment)

This should do the trick

Alex

Turribeach

The problem is that Python fails to correctly locate the trust store therefore it can not validate the SSL certificate provided. What do you get if you run this:

python3 -c 'import ssl; print(ssl.get_default_verify_paths())'

JIshikawa

Interesting, it seems it points to python 3.6 for some reason, even if I told the code env to use python 3.7.

DefaultVerifyPaths(cafile='/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem', capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/Library/Frameworks/Python.framework/Versions/3.6/etc/openssl/certs')

AlexandreV

Hello @JIshikawa
,

This is a bug with our python, we're working on it.

A workaround for this is to select "Custom (lookup in PATH)" and set the path to a python3.7 you have installed on your machine (python from python.org, from pyenv, from brew...)

 

 
Hope this solves your issue

Alex

JIshikawa

Using Python 3.6 did the trick. The installation went through without problems.

Many thanks for the great support,

JIshikawa