Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

Run python script

eric1
Level 1
Run python script

Hello,

I want to run other open source code in Python with Dataiku platform.
Most open-source code can be run through Python scripts like below link.
https://github.com/kuangliu/pytorch-cifar
python main.py --resume --lr=0.01

I want to know how to use python script in dataiku.

0 Kudos
1 Reply
AlexT
Dataiker

Hi,
One approach is to
1) import the remote git repo as a project library from git.
https://doc.dataiku.com/dss/latest/collaboration/import-code-from-git.html
2) Then use a Notebook or Python recipe in DSS to run the "main.py" with the arguments.
3) Note the code env used for the Python recipe/Notebook will need to have the required e.g torch,  torchvision 

Note the code may not be designed to be run as code directly e.g if interact with specific components, you may need to refactor or modify the code.
In your case, I found an issue with a line utils.py which I commend and set a fixed size as suggested here https://github.com/yangze0930/NTS-Net/issues/22.
Once resolving this, I was able to run the script but your sample command fails because there was no "checkpoint" directory which seems expected. 

import subprocess
import sys
import os.path as osp

def get_path_to_library_path(path_in_lib):
    for path_chunk in sys.path:
        searched_path = osp.join(path_chunk, path_in_lib)
        if osp.exists(searched_path):
            return searched_path
    raise Exception("Path %s not found in imported libraries" % path_in_lib)

path = get_path_to_library_path("pytorch-cifar/main.py")
print("Path:", path)

current_python_path = sys.executable
print("Current Python Path:", current_python_path)

# Specify the command to execute
command = [current_python_path, path, '--resume', '--lr=0.01']

# Execute the command
try:
    result = subprocess.run(command, check=True, text=True)
    sys.exit(0)
except subprocess.CalledProcessError as e:
    print(f"Subprocess execution failed with exit code {e.returncode}.")
    print("Output:")
    print(e.stdout)
    print("Error:")
    print(e.stderr)
    sys.exit(1)

# Properly exit the script

 
Hope that helps!