Connect databrick's data to dataiku

mm
Level 1
Connect databrick's data to dataiku

I am trying to connect databrick's data using SCP/SFTP connection however I am getting following error:

- HTTP Code:500

- Error type: java.io.IOException.

Please advice how I can add databrick's tables to dataiku. 

0 Kudos
4 Replies
shashank
Dataiker

If it's a Table Object in Databricks, use the JDBC URL of the Databricks Cluster and use that to create a connection in Dataiku. Once you created the connection you would be able to pull that as a dataset.

If you want to load the underlying file of the table, then it depends on what storage you are using. Let's assume you are using AWS S3, for that you need to build an S3 connection and load the files.

If this doesn't help, please share more details about your use case and how you are connecting.

0 Kudos
jbader
Level 1

Thanks for the direction on this one.  Would you have a link to an example that does this?  I am trying to get my team moving in this direction but they lack experience in this method.   Thanks!

0 Kudos
mm
Level 1
Author

Its a table object in databricks, and I am using JDBC URL of the databricks cluster. 

  1. I created "SCP/SFTP connection" in "DSS Settings" in "Connections". Is this the right way or shall I create some other type of connection?
  2. When I try to import data using "New SFTP Dataset ", it gives following error:

    Additional technical details

    • HTTP code: 500
    • Error type:java.io.IOException

Please let me know if I am doing some step in a wrong way? 

 

 

0 Kudos
shashank
Dataiker

You need to use "Other SQL Databases" under the SQL Databases connection category and provide the JDBC URL connection details.

SCP/SFTP is for File systems not for  Databases

0 Kudos

Labels

?

Setup info

?
A banner prompting to get Dataiku