Want to Stop Rebuilding "Expensive" Parts of your Flow? Explicit Builds are the Answer!READ MORE

api connect plugin

tgb417
Neuron
Neuron
api connect plugin

Anyone out there using the API Connect Plugin?

What sorts of successes and challenges have you had with this tool?

In particular, I'm interested in how bulk data can be received from various APIs.  Key issues are both performance and reliability.

 

 

 

--Tom
0 Kudos
32 Replies
ikovacs
Level 2

Thanks Alex,  it works for me now!

0 Kudos
tgb417
Neuron
Neuron
Author

@AlexB 

I'm looking to figure out how to use Offset pagination.  The documentation does not give a clue how the limit and offset are incremented. For so-called offset pagination.

The API I'm working with would need iterative calls like:

https://hostname.com/apis/thing/stuff?limit=10&skip=0

https://hostname.com/apis/thing/stuff?limit=10&skip=10

https://hostname.com/apis/thing/stuff?limit=10&skip=20

To pull the "first" up to 30 records.  

Anyone out there gotten this working.  I need to paginate through 10s of thousands of records at times.

 

--Tom
0 Kudos
AlexB
Dataiker
Dataiker

Hi Tom !

For the Offset pagination mode, the offset key is incremented by the value of the limit key sent by the server. If you need to speed up the transfer, you first need to check that the server accept the limit value to be change. Try https://hostname.com/apis/thing/stuff?limit=100 and see if it returns 100 elements. If that's the case, you can add  a larger limit with the QueryParams key/value pair interface.

Screenshot 2022-08-10 at 09.43.42.png

Alex 

tgb417
Neuron
Neuron
Author

@AlexB ,

Unfortunately still no joy here.  I seem to have control of the Total Limit with the Query Params limit = 5 But the Skip value is not being updated I think.  Also is there a way to track the queries being sent by the plugin to see what is actually happening.

Showing Parameters for the  API Connect Plugin Data Object.  Query Params limit set to 5 and key limiting element per page limit, key for element offset skipShowing Parameters for the API Connect Plugin Data Object. Query Params limit set to 5 and key limiting element per page limit, key for element offset skip

--Tom
0 Kudos
tgb417
Neuron
Neuron
Author

@AlexB 

Maybe I'm doing the wrong type of pagination.

I note in the returned header the following values. Returned API request header with  total, results, limit, skip listed out.Returned API request header with total, results, limit, skip listed out. 

 

--Tom
0 Kudos
AlexB
Dataiker
Dataiker

Is this API public by any chance ? Or its documentation ?

If not, could you send us the json returned in the browser with the equivalent of https://hostname.com/apis/thing/stuff?limit=2 ?  (feel free to anonymize or remove any confidential data from it, I would just need to look at the overall json's structure...)

 

0 Kudos
tgb417
Neuron
Neuron
Author

@AlexB 

I sent you a private chat here on the community with some further details.  

--Tom
0 Kudos
maxmeu
Level 1

Hey,

 

I am working on a different problem with the API Connect Plugin. What is the syntax for the request's body using the raw format? I have copied the syntax format from an API documentation for a POST request, but I am unable to figure out how to include the variables from my dataset as key/value pair. Any ideas? Or do I need to tick a box somewhere else?

 

Best

Max

0 Kudos
AlexB
Dataiker
Dataiker

Hi,

Say you have a dataset containing the data that you want to use as variables:

variable_dataset.png

You need  to start by clicking on that dataset, select API connect on the right side plugin panel, followed by API Connect recipe.

flow.png

 In the recipe configuration panel, you then need to select all the columns that you will use as variables in you API calls. This is done in the Columns to use as variable section.

Then write the body. On this picture there is a example of a standard JSON body. You can see that the variable is called simply by using the {{variable_name}} syntax.

recipe_settings.png

There could be subtleties, such as the need to add brackets before and after the variable in case if strings. I would recommend testing your settings by using the https://httpbin.org/anything API, which will just send you back your request defined as a JSON structure. This way you can check in the "data" section that the JSON format is valid and your variable was used as expected.

apis_response.png

 Hope this helps,

Alex

0 Kudos
alec_peterson
Level 1

Hi Alex,

I am having an issue with the specific API I'm working with.

1) Before I can make API resource requests, I have to first make a POST request for an authorization token, and then use that token in the headers for subsequent API calls.

That part is successful and it's saving as a dataset with column name "api_response"

2) I am trying to use that token string in subsequent API calls i.e. using that dataset as an input to an API Connect recipe.

I've experimented with setting the  "Columns to use as variables" as "api_response", then putting the headers Key as "Authorization" (per my API's documentation) and the Value as "Bearer {{api_response}}" but it's giving an error that the token is invalid. I've verified the token string from the input dataset is correct.

I've tried reformatting the raw text per your example, as well as different versions of adding brackets and curly braces outside (per your comment about handling for strings) but it still gives the error.

 

Do you have any advice on how to refer to the token string stored in the "api_response" column of my input dataset?

0 Kudos
alec_peterson
Level 1

I was able to address this. In experimenting with the https://httpbin.org/anything

,the string being input from my first API call actually had quotes at the start and end of the token characters.

Instead of inputting the authorization token dataset (from the first API call), I put a Prepare recipe in between that got rid of the quotes on the ends and added "Bearer " to the start, giving a final output of "Bearer xxxxxxxxxx" (not including the quotes, and xxxxxxxxxx standing for the token string).

Using this new as a variable, and then inputting that variable via {{column_name}} form into the Value part of the headers field worked!

CoreyS
Community Manager
Community Manager

Thank you for sharing your solution with the rest of this community @alec_peterson!

Looking for more resources to help you use Dataiku effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as ‘Accepted Solution’ to help others like you!
0 Kudos