Connect databrick's data to dataiku

mm
mm Registered Posts: 2 ✭✭✭

I am trying to connect databrick's data using SCP/SFTP connection however I am getting following error:

- HTTP Code:500

- Error type: java.io.IOException.

Please advice how I can add databrick's tables to dataiku.

Answers

  • Shashank
    Shashank Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 27 Dataiker

    If it's a Table Object in Databricks, use the JDBC URL of the Databricks Cluster and use that to create a connection in Dataiku. Once you created the connection you would be able to pull that as a dataset.

    If you want to load the underlying file of the table, then it depends on what storage you are using. Let's assume you are using AWS S3, for that you need to build an S3 connection and load the files.

    If this doesn't help, please share more details about your use case and how you are connecting.

  • mm
    mm Registered Posts: 2 ✭✭✭

    Its a table object in databricks, and I am using JDBC URL of the databricks cluster.

    1. I created "SCP/SFTP connection" in "DSS Settings" in "Connections". Is this the right way or shall I create some other type of connection?
    2. When I try to import data using "New SFTP Dataset ", it gives following error:

      Additional technical details

      • HTTP code: 500
      • Error type:java.io.IOException

    Please let me know if I am doing some step in a wrong way?

  • Shashank
    Shashank Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 27 Dataiker

    You need to use "Other SQL Databases" under the SQL Databases connection category and provide the JDBC URL connection details.

    SCP/SFTP is for File systems not for Databases

  • jbader
    jbader Registered Posts: 1

    Thanks for the direction on this one. Would you have a link to an example that does this? I am trying to get my team moving in this direction but they lack experience in this method. Thanks!

Setup Info
    Tags
      Help me…