Upgrading DSS for Ubuntu 18.04: fails with python import error "cannot import name _remove_dead_weakref"

[UserBird]

Hi,

I upgraded ubuntu to 18.04, after reading that dss is compatible with it since version 4.2.2.

Nevertheless, when I try to install dss 4.2.2, it fails with this error:



*********************************************
* Dataiku DSS installer *
*********************************************
[+] /home/inactinique/dataiku-data already exists and is a DSS installation
[+] Saving installation log to /home/inactinique/dataiku-data/run/install.log
[+] Using Java at /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java : openjdk version "1.8.0_162"
[+] Checking required dependencies
+ Detected OS distribution : ubuntu 18.04
+ Checking required packages...
[+] Migrating Python environment
[+] Precompiling Dataiku Python code
[+] Precompiling Jupyter Python code
[+] Precompiling third-party Python code
[+] Migrating data directory
Traceback (most recent call last):
File "/opt/dataiku-dss-4.2.2/scripts/dkuinstall/migrate_auto.py", line 5, in
import migration_base, migrate_dss, migrate_apinode
File "/opt/dataiku-dss-4.2.2/scripts/dkuinstall/migration_base.py", line 7, in
import json, logging
File "/usr/lib/python2.7/logging/__init__.py", line 26, in
import sys, os, time, cStringIO, traceback, warnings, weakref, collections
File "/usr/lib/python2.7/weakref.py", line 14, in
from _weakref import (
ImportError: cannot import name _remove_dead_weakref



Has someone any idea about what's going wrong?

Best,

Frédéric

[pbertin · Accepted solution]

Hi Frédéric,

this is a known limitation of virtualenv with respect to an upgrade of the underlying Python (this has already been seen to happen on previous upgrades of Ubuntu distributions) : the Python virtual environment within the DSS data dir still contains references to the pre-upgrade Python, and breaks with low-level errors on startup.

The solution is most probably to fully rebuild the Python environment used by DSS, as described here:
https://doc.dataiku.com/dss/latest/installation/migrations.html#migrating-the-data-directory

- move away (rename) the DATADIR/pyenv directory
- re-run the DSS installer in upgrade mode : this should rebuild a new virtualenv
- optionally, reinstall any locally-installed package (you can look for them by listing the lib/python2.7/site-packages in the old virtualenv)

Note that any DSS-managed Python code environments will probably have to be rebuilt as well (this can be done from the DSS UI)

Regards
Patrice Bertin
Dataiku

[UserBird]

Dear Patrice,
it just perfectly worked.
Merci!