Installation / DSS Start failure

PGuti · October 2021

Hello!

I had a quite old instance of DSS on my mac (I used it last time in late 2017).

I wanted to run it but I got an error when trying to launch it.

So I thought OK I will just download the latest version and upgrade it.

I did this and when I'm trying to start DSS I get the following error:

[Install] ******************************************
* MIGRATION FAILED
******************************************
* Attempting to rollback
Backing up partially-migrated: dss-version.json -> /Users/pierregutierrez/Library/DataScienceStudio/dss_home/_broken_migration_backup_20211029-103623/dss-version.json

[Install] Restoring backup: _pre_migration_backup_20211029-103623/dss-version.json -> dss-version.json

[Install] Backing up partially-migrated: bin/env-default.sh -> /Users/pierregutierrez/Library/DataScienceStudio/dss_home/_broken_migration_backup_20211029-103623/bin__env-default.sh
Restoring backup: _pre_migration_backup_20211029-103623/bin__env-default.sh -> bin/env-default.sh

[Install] Backing up partially-migrated: install.ini -> /Users/pierregutierrez/Library/DataScienceStudio/dss_home/_broken_migration_backup_20211029-103623/install.ini

[Install] Restoring backup: _pre_migration_backup_20211029-103623/install.ini -> install.ini
Backing up partially-migrated: databases -> /Users/pierregutierrez/Library/DataScienceStudio/dss_home/_broken_migration_backup_20211029-103623/databases

[Install] Restoring backup: _pre_migration_backup_20211029-103623/databases -> databases

[Install] Backing up partially-migrated: config -> /Users/pierregutierrez/Library/DataScienceStudio/dss_home/_broken_migration_backup_20211029-103623/config

[Install] Restoring backup: _pre_migration_backup_20211029-103623/config -> config
Restore complete, removing marker file

[Install] Traceback (most recent call last):
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migrate_auto.py", line 30, in <module>
    migrate_dss.migrate(sys.argv[1])
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migrate_dss.py", line 98, in migrate

[Install]     reraise(*sys.exc_info())
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migrate_dss.py", line 67, in migrate
    migration.execute(dip_home, node_type)
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migration_base.py", line 136, in execute

[Install]     wrapper.execute(diphome)
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migration_base.py", line 79, in execute
    migration.execute(ppaths)
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migration_json.py", line 132, in execute

[Install]     migrate_json_file(self, fp)
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/migration_json.py", line 95, in migrate_json_file
    new_obj = transformer.transform(cobj, fp)
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/steps.py", line 4940, in transform

[Install]     return migrate_ml_spark_params(modeling)
  File "/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/dkuinstall/steps.py", line 4907, in migrate_ml_spark_params

[Install]     spark_params[prop] = obj.get(prop, None)
AttributeError: 'NoneType' object has no attribute 'get'

[install] EOF reached
[install] EOF reached
DSS installer terminated with status 1
DSS installer failed , fatal error

Any idea how to fix this?

Thank you!

Huixian · October 2021

Hi @PGuti
,

Thank you for the information!

Can you please check if the params.json file under "/Users/pierregutierrez/Library/DataScienceStudio/dss_home/config/projects/VELIB/analysis/W9NhaPCz/ml/7iRWiMaj/" is null/empty. If so, can you please delete the file or move it outside of the DSS data directory and try again with the migration?

If the issue persists after that, please attach the full log to the chat you opened, so that we can review and investigate further.

Thank you!

Regards,
Huixian Pan

Sergey · October 2021

Hi @PGuti
,

Can you please send us a couple of dozens of lines before this error happened? We need to see the last step the migration was doing before failure.

Thanks.

PGuti · October 2021

The complete log would show all the project names on a public page which I would prefer avoiding.

The exact line before was:

"""

[Install] Migrating /Users/pierregutierrez/Library/DataScienceStudio/dss_home/config/projects/VELIB/analysis/W9NhaPCz/ml/7iRWiMaj/params.json

"""

The last step written was:

"""

20 -> 21

- Migrates spark params in saved models into a common object

(Wrapped)

"""

I had a bunch of warning messages:
[Install] ** WARNING: Unhandled algorithm: KMEANS

[Install] ** WARNING: Unhandled algorithm: EXTRA_TREES

[Install] ** WARNING: Unhandled algorithm: NEURAL_NETWORK

[Install] ** WARNING: Unhandled algorithm: PY_TWO_STEP

which I'm assume are models which changed and custom models.

The beginning of the log seems to explicit what is going to be done and is:

"""

Starting Data Science Studio.app
dssHome:/Users/pierregutierrez/Library/DataScienceStudio/dss_home dssBin:/Users/pierregutierrez/Library/DataScienceStudio/dss_home/bin/dss
Starting Java check
/Applications/DataScienceStudio.app/Contents/Resources/kit/scripts/_find-java.sh
[Install] [+] Using Java at /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java : java version "1.8.0_121"

[Install] /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java

[Java check] EOF reached
[Java check] EOF reached
Java check terminated with status 0
dss-version exists, doing stop-upgrade-start
Running DSS task for 'stop'
[DSS] dss: DSS supervisor is not running

[dss stop] EOF reached
[dss stop] EOF reached
[Install] *********************************************
* Dataiku DSS installer *
*********************************************

[Install] [+] /Users/pierregutierrez/Library/DataScienceStudio/dss_home already exists and is a DSS installation

[Install] [+] Saving installation log to /Users/pierregutierrez/Library/DataScienceStudio/dss_home/run/install.log

[Install] [+] Using Java at /Library/Internet Plug-Ins/JavaAppletPlugin.plugin/Contents/Home/bin/java : java version "1.8.0_121"

[Install] [+] Checking required dependencies

[Install] + Detected OS distribution : osx 11.6

[Install] + Checking required packages...

[Install] [+] Migrating Python environment

[Install] [+] Precompiling Dataiku Python code

[Install] [+] Precompiling Jupyter Python code

[Install] [+] Precompiling third-party Python 2.7 code

[Install] [+] Migrating data directory

[Install] Select how to migrate from 17 from avail: [8 -> 9, 9 -> 10, 10 -> 11, 11 -> 12, 12 -> 13, 13 -> 14, 14 -> 15, 15 -> 16, 16 -> 17, 17 -> 18, 18 -> 19, 19 -> 20, 20 -> 21, 21 -> 22, 22 -> 23, 23 -> 24, 24 -> 6000, 6000 -> 6020, 6020 -> 6030, 6030 -> 7000, 7000 -> 7020, 7020 -> 8000, 8000 -> 8020, 8999 -> 9000]
Executing the following migrations

17 -> 18
- Clean up the partitioning of SQL datasets
- Upgrade the y/n resize-to-fit setting of dashboard images to new multi-option setting
- Cleanup trained models parameters
- Nest feature generation parameters
- Drop caches
- Drop persistent notifications database

18 -> 19

19 -> 20
- Add params to ML recipes
- Transform comments insight to discussions insights
- Transform dashboard by changing comments insight to discussions insights
- Drop caches
- Drop persistent notifications database

[Install]

20 -> 21
- Migrates spark params in saved models into a common object
- Migrates spark params in analyses' models into a common object
- Migrates spark params in analyses' ml tasks into a common object
- Regroup spark params in recipes
- Drop caches
- Drop persistent notifications database

21 -> 22
- Give NPS Survey Settings to each user

22 -> 23
-
[Install] Setup metadata for code envs in Jupyter kernels' specs
- Upgrade structure of dss_schema_info in H2 databases
- Make CLASS_WEIGHT the default weighting strategy for classification saved models
- Sets a default Git URL whitelist for clones/pulls
- Use smartName in refresh charts steps
- Move ml task param skipExpensiveReports in modeling
- Drop caches
- Drop persistent notifications database

23 -> 24
- Migrating article attachments to use smart IDs

24 -> 6000
- Migrating project path to project folder
- Change project home behavior in user-settings.json
- Migrating article IDs to numeric IDs instead of names
- Migrating homepage articles to ids
[Install]

[Install] - Migrating dashboard articles to ids
- Upgrade Wiki timeline with numeric article IDs
- Change credentials mode in AWS connections
- Upgrade pre-push hook in general settings
- Upgrade pre-push hook in API deployer
- Migrating visual ML execution params
- Migrating listing of code-envs used in visual deep-learning
- Upgrade evaluation recipe metrics and ouputs configurations

6000 -> 6020
- Migrating article IDs to numeric IDs in dashboards

6020 -> 6030
- Migrating Microsoft Teams scenarios reporters

6030 -> 7000
- Renaming connectionCredentials to credentials for each user
- Convert expositions in infras of API deployer
- Convert expositions in deployments of API deployer
- Removing existing Hipchat Channels
- Removing existing Hipchat Reporters
- Removing existing Hipchat Integrations
- Migrating algorithms params structure
- Migrating posttrain computation params
- Migrating SAML SP configuration

7000 -> 7020
- Migrate scoring recipe explanation params

7020 -> 8000
- Initialize audit settings
- Initialize audit settings (API node)
- Migrating SVM algorithms params structure
- Migrating structure of 'proportion of features to sample'
- Migrating prediction algorithms params structure to add range
- Migrating grid length for non-searchable algos
- Migrate code env selection

8000 -> 8020
- Migrating tree visualisations to take into account difference between class and sample weights
- Rename 'hashSVDhashSize' field to 'hashSize' for text features
- Migrating structure of global tags categories

8999 -> 9000
- Migrate categorical hashing method
- Update preparation script FilterOnDateRange steps
- Update preparation script FlagOnDateRange steps
- Migrating of the jupyter storage path to project
[Install]
- Add parameters to evaluation recipes
- Migrate API service params to use type parameter
- Migrating API Deployer settings to be Deployer settings
- Migrating time triggers
- Migrate max depth for tree based models

"""

PGuti · November 2021

Hello!

So when looking at this file, I saw it was only containing "null". So I deleted it and restarted.

I had the problem for several other files. So I deleted all of them. If I had to guess, I would say it's related to the custom models I had (the big WARNINGS on the log).

In case someone has the same issue as I did, here is the python code to generate all file names you want to remove:

import os
import json

def load_json_file(filename):
    with open(filename) as f:
        data = json.load(f)
    return data

mainpath = "YOURPATHHERE"

projects = os.listdir(mainpath)

for p in projects:
    project_path = os.path.join(mainpath,p, "analysis")
    if os.path.isdir(project_path):
        analyses = os.listdir(project_path)
        for analysis in analyses:
            ml_path = os.path.join(project_path, analysis, "ml")
            if os.path.isdir(ml_path):
                models = os.listdir(ml_path)
                for m in models:
                    final_path = os.path.join(ml_path,m, "params.json")
                    if os.path.isfile(final_path):
                        mydata = load_json_file(final_path)
                        if mydata is None:
                            print("rm " + final_path)

The studio is now working!

Thanks for the help!

Huixian · November 2021

This is great! Thank you @PGuti
for the updates and thank you for sharing the additional information!

Regards,
Huixian Pan

Installation / DSS Start failure

Best Answer

Answers

Categories

Setup Info

Tags