Kickstart python script execution from Dataiku

Options
NiteshK
NiteshK Registered Posts: 3 ✭✭✭✭

Can we use Dataiku to execute and kickstart python scripts which are placed in some other server [i.e. python scripts aren't there on the same server where Dataiku is installed].

I tried creating some python recipes in one of the dataiku projects but looks like those recipes are intended mainly for reading some datasets , doing some actions and creating a output dataset.

We have a data pipeline code built of few python scripts running one after the other and currently we are looking what goodies we can provide to GUI savvy folks from dataiku in the pipeline execution.

Answers

  • dima_naboka
    dima_naboka Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 28 Dataiker
    edited July 17
    Options
    Hi Nitesh,
    You have an option to remotely execute a shell command as part of DSS job by utilizing Shell recipe. For example, you can log in to a remote server and run Python code like this:
    ssh -t -i ~/.ssh/my_key centos@ec2-address.compute.amazonaws.com "python test_print.py”
    On the other hand, to benefit from executing your Python code locally as part of a DSS project you can import it using Git and Project Libraries. Then import and use it in Python recipe or Jupyter notebook
  • Ignacio_Toledo
    Ignacio_Toledo Dataiku DSS Core Designer, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 411 Neuron
    Options

    Hi @NiteshK
    . This is an interesting topic. We had a similar use case here, where there was a need to trigger a script execution in a different machine, and then wait for the results and import them into dataiku.

    Our take in the end was not to support that kind of use case, because it looked more like an automation of data acquisition (which can be solved with other more focused tools) than a data science project that could fit into the Dataiku model.

    I'm interested in knowing if with the solution @dima_naboka
    suggests you will be going ahead on this use case, and if succeed what was your final implementation.

Setup Info
    Tags
      Help me…