running databricks notebook from dataiku

vijaynchakole Registered Posts: 4 ✭✭✭

I have created jupyter notebook on databricks which runs NLP models in databricks . I want to run that notebook from dataiku and get results of that NLP model in Dataiku.

Can you please guide me , how we can run databricks notebook from dataiku platform?


  • shashank
    shashank Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 27 Dataiker

    Can you please elaborate on your use case? I want to understand the requirement behind running a Databricks notebook from Dataiku, and also what's your storage layer (S3, Snowflake, etc.).

    I can share some options assuming your use case pattern:

    1. Option 1: Run the Notebook on EKS (w Spark): Setup a Compute using EKS cluster with Spark in Dataiku. Create a Spark Recipe (copy-paste the code) and run it into EKS (no code change required). This option is recommended as you can containerize your code as well as the associated libraries, which simplify your Ops process

    2. Option 2: Keep it in Databricks: Convert your Notebook into a Databricks Job. A Databricks Job has a REST endpoint. Here you have 2 options to integrate into your Dataiku Project. Create a Code recipe to hit that endpoint or use Dataiku Scenarios to add a step(custom python) for triggering the NLP Job in Databricks.

    Option 1 is Recommended but the solution may change based on your actual use case.

Setup Info
      Help me…