Run python script
Hello,
I want to run other open source code in Python with Dataiku platform.
Most open-source code can be run through Python scripts like below link.
https://github.com/kuangliu/pytorch-cifar
python main.py --resume --lr=0.01
I want to know how to use python script in dataiku.
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,226 Dataiker
Hi,
One approach is to
1) import the remote git repo as a project library from git.
https://doc.dataiku.com/dss/latest/collaboration/import-code-from-git.html
2) Then use a Notebook or Python recipe in DSS to run the "main.py" with the arguments.
3) Note the code env used for the Python recipe/Notebook will need to have the required e.g torch, torchvision
Note the code may not be designed to be run as code directly e.g if interact with specific components, you may need to refactor or modify the code.
In your case, I found an issue with a line utils.py which I commend and set a fixed size as suggested here https://github.com/yangze0930/NTS-Net/issues/22.
Once resolving this, I was able to run the script but your sample command fails because there was no "checkpoint" directory which seems expected.import subprocess import sys import os.path as osp def get_path_to_library_path(path_in_lib): for path_chunk in sys.path: searched_path = osp.join(path_chunk, path_in_lib) if osp.exists(searched_path): return searched_path raise Exception("Path %s not found in imported libraries" % path_in_lib) path = get_path_to_library_path("pytorch-cifar/main.py") print("Path:", path) current_python_path = sys.executable print("Current Python Path:", current_python_path) # Specify the command to execute command = [current_python_path, path, '--resume', '--lr=0.01'] # Execute the command try: result = subprocess.run(command, check=True, text=True) sys.exit(0) except subprocess.CalledProcessError as e: print(f"Subprocess execution failed with exit code {e.returncode}.") print("Output:") print(e.stdout) print("Error:") print(e.stderr) sys.exit(1) # Properly exit the script
Hope that helps!