Run Custom Python Code Step in Scenarios Leveraging Containerization

kathyqingyuxu

Currently in a scenario an end user is able to create a custom python code step, however, this step is running locally and not in containers.

It would be helpful to allow end users to leverage containers in the scenario step to help offload the job from the local server. this in particular is super helpful if the script requires data input that is fairly large, and/or requires a lot of complex calculations.

Although, a work around is to leverage a python recipe in the flow and containerize the recipe, we sometimes leverage the scenario step in order to conduct checks on multiple tables on the flow. Through a recipe this would require adding multiple input datasets, which can be time consuming.

By adding the capability to containerize the step through the scenario this provides additional flexibility for the end user.

AshleyW

@kathyqingyuxu
,

You'll be pleased to hear this idea is in our backlog, and we are determining the next steps for development. We can't provide a timeline at this point, but be sure to check back for updates!

For everyone else, kudos the original post to signal that you're interested in Dataiku developing and releasing this feature! Feel free to leave a comment in the discussion about how this capability would help you or your team.

Take care,
Ashley