Discover this year's submissions to the Dataiku Frontrunner Awards and give kudos to your favorite use cases and success stories!READ MORE

running databricks notebook from dataiku

vijaynchakole
Level 1
running databricks notebook from dataiku

I have created jupyter notebook on databricks which runs NLP models in databricks . I want to run that notebook from dataiku and get results of that NLP model in Dataiku.

Can you please guide me , how we can run databricks notebook from dataiku platform?

0 Kudos
1 Reply
shashank
Dataiker
Dataiker

Can you please elaborate on your use case? I want to understand the requirement behind running a Databricks notebook from Dataiku, and also what's your storage layer (S3, Snowflake, etc.).

I can share some options assuming your use case pattern:

1. Option 1: Run the Notebook on EKS (w Spark): Setup a Compute using EKS cluster with Spark in Dataiku. Create a Spark Recipe (copy-paste the code) and run it into EKS (no code change required). This option is recommended as you can containerize your code as well as the associated libraries, which simplify your Ops process

2. Option 2: Keep it in Databricks: Convert your Notebook into a Databricks Job. A Databricks Job has a REST endpoint. Here you have 2 options to integrate into your Dataiku Project. Create a Code recipe to hit that endpoint or use Dataiku Scenarios to add a step(custom python) for triggering the NLP Job in Databricks.

Option 1 is Recommended but the solution may change based on your actual use case.