api connect plugin

tgb417
api connect plugin

Anyone out there using the API Connect Plugin?

What sorts of successes and challenges have you had with this tool?

In particular, I'm interested in how bulk data can be received from various APIs.  Key issues are both performance and reliability.

 

 

 

--Tom
0 Kudos
34 Replies
ikovacs
Level 2

Thanks Alex,  it works for me now!

0 Kudos
tgb417
Author

@AlexB 

I'm looking to figure out how to use Offset pagination.  The documentation does not give a clue how the limit and offset are incremented. For so-called offset pagination.

The API I'm working with would need iterative calls like:

https://hostname.com/apis/thing/stuff?limit=10&skip=0

https://hostname.com/apis/thing/stuff?limit=10&skip=10

https://hostname.com/apis/thing/stuff?limit=10&skip=20

To pull the "first" up to 30 records.  

Anyone out there gotten this working.  I need to paginate through 10s of thousands of records at times.

 

--Tom
0 Kudos
AlexB
Dataiker

Hi Tom !

For the Offset pagination mode, the offset key is incremented by the value of the limit key sent by the server. If you need to speed up the transfer, you first need to check that the server accept the limit value to be change. Try https://hostname.com/apis/thing/stuff?limit=100 and see if it returns 100 elements. If that's the case, you can add  a larger limit with the QueryParams key/value pair interface.

Screenshot 2022-08-10 at 09.43.42.png

Alexโ€ƒ

tgb417
Author

@AlexB ,

Unfortunately still no joy here.  I seem to have control of the Total Limit with the Query Params limit = 5 But the Skip value is not being updated I think.  Also is there a way to track the queries being sent by the plugin to see what is actually happening.

Showing Parameters for the  API Connect Plugin Data Object.  Query Params limit set to 5 and key limiting element per page limit, key for element offset skipShowing Parameters for the API Connect Plugin Data Object. Query Params limit set to 5 and key limiting element per page limit, key for element offset skip

--Tom
0 Kudos
tgb417
Author

@AlexB 

Maybe I'm doing the wrong type of pagination.

I note in the returned header the following values. Returned API request header with  total, results, limit, skip listed out.Returned API request header with total, results, limit, skip listed out. 

 

--Tom
0 Kudos
AlexB
Dataiker

Is this API public by any chance ? Or its documentation ?

If not, could you send us the json returned in the browser with the equivalent of https://hostname.com/apis/thing/stuff?limit=2 ?  (feel free to anonymize or remove any confidential data from it, I would just need to look at the overall json's structure...)

 

0 Kudos
tgb417
Author

@AlexB 

I sent you a private chat here on the community with some further details.  

--Tom
0 Kudos
maxmeu
Level 1

Hey,

 

I am working on a different problem with the API Connect Plugin. What is the syntax for the request's body using the raw format? I have copied the syntax format from an API documentation for a POST request, but I am unable to figure out how to include the variables from my dataset as key/value pair. Any ideas? Or do I need to tick a box somewhere else?

 

Best

Max

0 Kudos
AlexB
Dataiker

Hi,

Say you have a dataset containing the data that you want to use as variables:

variable_dataset.png

You needโ€ƒ to start by clicking on that dataset, select API connect on the right side plugin panel, followed by API Connect recipe.

flow.png

โ€ƒIn the recipe configuration panel, you then need to select all the columns that you will use as variables in you API calls. This is done in the Columns to use as variable section.

Then write the body. On this picture there is a example of a standard JSON body. You can see that the variable is called simply by using the {{variable_name}} syntax.

recipe_settings.png

There could be subtleties, such as the need to add brackets before and after the variable in case if strings. I would recommend testing your settings by using the https://httpbin.org/anything API, which will just send you back your request defined as a JSON structure. This way you can check in the "data" section that the JSON format is valid and your variable was used as expected.

apis_response.png

โ€ƒHope this helps,

Alex

0 Kudos
alec_peterson
Level 1

Hi Alex,

I am having an issue with the specific API I'm working with.

1) Before I can make API resource requests, I have to first make a POST request for an authorization token, and then use that token in the headers for subsequent API calls.

That part is successful and it's saving as a dataset with column name "api_response"

2) I am trying to use that token string in subsequent API calls i.e. using that dataset as an input to an API Connect recipe.

I've experimented with setting the  "Columns to use as variables" as "api_response", then putting the headers Key as "Authorization" (per my API's documentation) and the Value as "Bearer {{api_response}}" but it's giving an error that the token is invalid. I've verified the token string from the input dataset is correct.

I've tried reformatting the raw text per your example, as well as different versions of adding brackets and curly braces outside (per your comment about handling for strings) but it still gives the error.

 

Do you have any advice on how to refer to the token string stored in the "api_response" column of my input dataset?

0 Kudos
alec_peterson
Level 1

I was able to address this. In experimenting with the https://httpbin.org/anything

,the string being input from my first API call actually had quotes at the start and end of the token characters.

Instead of inputting the authorization token dataset (from the first API call), I put a Prepare recipe in between that got rid of the quotes on the ends and added "Bearer " to the start, giving a final output of "Bearer xxxxxxxxxx" (not including the quotes, and xxxxxxxxxx standing for the token string).

Using this new as a variable, and then inputting that variable via {{column_name}} form into the Value part of the headers field worked!

CoreyS
Dataiker Alumni

Thank you for sharing your solution with the rest of this community @alec_peterson!

Looking for more resources to help you use Dataiku effectively and upskill your knowledge? Check out these great resources: Dataiku Academy | Documentation | Knowledge Base

A reply answered your question? Mark as โ€˜Accepted Solutionโ€™ to help others like you!
0 Kudos
michael_belval
Level 1

Hi,

I am facing the same problem regarding pagination.

Let me explain, I have an API endpoint that I query to get results. My query is "https://domain.freshdesk.com/api/v2/contacts?page=1".

I want to enable pagination to retrieve all results from each page in my dataset.

But impossible to make it work correctly.

Can you help me?

THANKS

0 Kudos
tgb417
Author

@michael_belval 

First of all welcome to the Dataiku Community.

Second, I do not have access to the Fresh Desk API.  So the following are just guesses and suggestions.  If anyone else here is connecting into Fresh Desk.  Please jump in here and lend a hand.  That said

  • I would start my experiments with the fresh desk and the API connect plugin working with a non-paginated / preferably non authenticated GET request, and actually get data into Dataiku and Parse it.
  • Once you have figured that out.  I would get a tool like Postman and figure out by hand what the sequence of URLs is that are need to walk through the pages you are trying to gather.
  • From the url you have provided it appears that your โ€œKey for element offsetโ€ is the query parameter Pages.
  • You have not set pages up as a query parameter in the setup for API connect
  • I donโ€™t see from what you have provided what query parameter you need to provide to tell Fresh Desk how many records to skip between pages.  The API Freshdesk documentation should help you figure that out.
  • Finally it also appears that you are not setup with the latest API connect plugin on your server.  The current version is 1.2.0 and is faster and handles error messages better than earlier versions.  I can tell because the screen shot seems to be missing a few options.

Good luck with your use of API connect. It is definitly a bit hard to setup the first time.  

--Tom
0 Kudos