Run python script

Options
eric1
eric1 Registered Posts: 1

Hello,

I want to run other open source code in Python with Dataiku platform.
Most open-source code can be run through Python scripts like below link.
https://github.com/kuangliu/pytorch-cifar
python main.py --resume --lr=0.01

I want to know how to use python script in dataiku.

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    edited July 17
    Options

    Hi,
    One approach is to
    1) import the remote git repo as a project library from git.
    https://doc.dataiku.com/dss/latest/collaboration/import-code-from-git.html
    2) Then use a Notebook or Python recipe in DSS to run the "main.py" with the arguments.
    3) Note the code env used for the Python recipe/Notebook will need to have the required e.g torch, torchvision

    Note the code may not be designed to be run as code directly e.g if interact with specific components, you may need to refactor or modify the code.
    In your case, I found an issue with a line utils.py which I commend and set a fixed size as suggested here https://github.com/yangze0930/NTS-Net/issues/22.
    Once resolving this, I was able to run the script but your sample command fails because there was no "checkpoint" directory which seems expected.

    import subprocess
    import sys
    import os.path as osp
    
    def get_path_to_library_path(path_in_lib):
        for path_chunk in sys.path:
            searched_path = osp.join(path_chunk, path_in_lib)
            if osp.exists(searched_path):
                return searched_path
        raise Exception("Path %s not found in imported libraries" % path_in_lib)
    
    path = get_path_to_library_path("pytorch-cifar/main.py")
    print("Path:", path)
    
    current_python_path = sys.executable
    print("Current Python Path:", current_python_path)
    
    # Specify the command to execute
    command = [current_python_path, path, '--resume', '--lr=0.01']
    
    # Execute the command
    try:
        result = subprocess.run(command, check=True, text=True)
        sys.exit(0)
    except subprocess.CalledProcessError as e:
        print(f"Subprocess execution failed with exit code {e.returncode}.")
        print("Output:")
        print(e.stdout)
        print("Error:")
        print(e.stderr)
        sys.exit(1)
    
    # Properly exit the script
    


    Hope that helps!

Setup Info
    Tags
      Help me…