Has anyone connected to an external database using the API Connect Plugin using API Keys?

JT1
JT1 Dataiku DSS Core Designer, Registered Posts: 3

I'm trying to connect to an external database using an API Connect Plug in but keep getting an error: Test failed: Failed to read from python dataset : <class 'rest_api_client.RestAPIClientError'> : Error 401: b'{\n "message" : "Missing authentication header.",\n "status" : 401\n}'


Operating system used: Mac

Best Answer

  • JT1
    JT1 Dataiku DSS Core Designer, Registered Posts: 3
    Answer ✓

    Hi, someone from Dataiku was able to help me, I had to change the header parameters to match the key and value pairing. Thank you!!

Answers

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron

    @JT1

    Welcome to the Dataiku community. We are so glad to have you join us here.

    I connect to data sources via the API connect plugin multiple times an hour.

    It does take a bit of work to figure out how to use. At some point a tutorial should exist on using this tool.

    That said a few tips that may help.

    1. I use a Rest API tool like postman to make sure that I know exactly how the REST API I am working with actually works. If I can’t make the API work with Postman, I don’t know enough yet to get it to work from API Connect
    2. Under setting for the project there is a place to store your API key at the project level. I also tend to store some constant variable on that setting screen. Things like the Max Page size and starting skip value for paginated APIs.
    3. Make sure that you are using the latest version of the plugin. Over the past few years there have been a number of good reliability and performance fixes to the API Connect plugin. (Thanks Alex)

    Hope one of these may help a bit.

  • JT1
    JT1 Dataiku DSS Core Designer, Registered Posts: 3

    Hi Tom @tgb417
    ! Thank you for your reply! On Postman, it looks like everything is working, however, I keep getting errors. Do you know if the API Connect can only take JSON files or is CSV ok? Also I was provided with an API Key and Secret number, would I choose the API Key as the login type or Basic Auth?

  • tgb417
    tgb417 Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Registered, Dataiku Frontrunner Awards 2021 Finalist, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant, Neuron 2023 Posts: 1,598 Neuron
    edited July 17

    This question is not clear to me.

    Do you know if the API Connect can only take 
    JSON files or is CSV ok?

    You are saying "take JSON files". The API Connect plugin is designed to take as input only dataiku datasets. They may be in any to the typical forms supported by Dataiku. If by you "take JSON files" you mean that the API sending data to Dataiku would some how send a CSV file formatted result. I have not used the API connect to ingest a CSV file from a web server URL. I've only received JSON formated data which is very rich but definitly a bit harder to process. On the other hand I have used the standard HTTP Network connection to get .CSV datafiles. However, in those cases I have not had to use authentication.

    There is also an ODATA plugin that may be helpful for you depending on the details of your use case. There is also the ability to get data over SFTP that can be helpful to get HTML data. Finally if you have to you can of course write a bit of Python to do what you are trying to do.

    Although I use API Connect every day, I do not use all of it's features. It is my understanding that Basic Auth is for a typical User Name and Password. I have never had to use the API Key and Secret number type of authentication with the API connect plugin.

    You may want to reach out to your organizations Dataiku Customer Success folks, or the author of the plugin regarding your particular use case.

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,112 Neuron

    What database are you trying to connect to? What type of API does the database provide? What kind of authentication does it support? Can you provide the API documentation showing how to connect to it?

Setup Info
    Tags
      Help me…