http dataset from a rest api with basic or OAuth L2 authentication
I need to fetch a dataset from a rest api which is protected by basic auth or in some cases OAuth L2/L3. The http dataset does not have that option and only unsecure rest endpoints can be configured.
So I am thinkin of writing a small python requests/flask based client to read such a dataset. Question is how to protect the creds I might need to add.
Or is there another way available in dataiku
Best Answer
-
Hi,
The easiest option is probably to use a Python recipe to fetch the data (using the library requests for example) and write it in an output dataset. (note: a Python recipe does not necessary need an input dataset) Your code will eventually have to handle the refresh of the OAuth access token.
You may want to store the tokens in project variables (example of code here) or user secrets (as suggested by lpkronek).
Then, if you want a better way to package this, you may want to create a plugin, in particular a custom dataset component. As examples, you have the Salesforce (source code) and the Google Sheet (source code) plugins that fetch data from a REST API with OAuth and create Dataiku datasets.
Answers
-
DSS has a notion of user secrets that allows you to remove credentials from your code.
Credentials are then retrieved from the user profile at execution time. This solution requires that each user who wants to run the recipe to fill appropriate credentials in their profile.
