api connect plugin

[tgb417]

Anyone out there using the API Connect Plugin?

What sorts of successes and challenges have you had with this tool?

In particular, I'm interested in how bulk data can be received from various APIs.  Key issues are both performance and reliability.

[ikovacs]

Thanks Alex,  it works for me now!

[tgb417]

@AlexB 

I'm looking to figure out how to use Offset pagination.  The documentation does not give a clue how the limit and offset are incremented. For so-called offset pagination.

The API I'm working with would need iterative calls like:

https://hostname.com/apis/thing/stuff?limit=10&skip=0

https://hostname.com/apis/thing/stuff?limit=10&skip=10

https://hostname.com/apis/thing/stuff?limit=10&skip=20

To pull the "first" up to 30 records.  

Anyone out there gotten this working.  I need to paginate through 10s of thousands of records at times.

[AlexB]

Hi Tom !

For the Offset pagination mode, the offset key is incremented by the value of the limit key sent by the server. If you need to speed up the transfer, you first need to check that the server accept the limit value to be change. Try https://hostname.com/apis/thing/stuff?limit=100 and see if it returns 100 elements. If that's the case, you can add  a larger limit with the QueryParams key/value pair interface.

Alex 

[tgb417]

@AlexB ,

Unfortunately still no joy here.  I seem to have control of the Total Limit with the Query Params limit = 5 But the Skip value is not being updated I think.  Also is there a way to track the queries being sent by the plugin to see what is actually happening.

Showing Parameters for the API Connect Plugin Data Object. Query Params limit set to 5 and key limiting element per page limit, key for element offset skip

[tgb417]

@AlexB 

Maybe I'm doing the wrong type of pagination.

I note in the returned header the following values. Returned API request header with total, results, limit, skip listed out. 

[AlexB]

Is this API public by any chance ? Or its documentation ?

If not, could you send us the json returned in the browser with the equivalent of https://hostname.com/apis/thing/stuff?limit=2 ?  (feel free to anonymize or remove any confidential data from it, I would just need to look at the overall json's structure...)

[tgb417]

@AlexB 

I sent you a private chat here on the community with some further details.  

[maxmeu]

Hey,

I am working on a different problem with the API Connect Plugin. What is the syntax for the request's body using the raw format? I have copied the syntax format from an API documentation for a POST request, but I am unable to figure out how to include the variables from my dataset as key/value pair. Any ideas? Or do I need to tick a box somewhere else?

Best

Max

[AlexB]

Hi,

Say you have a dataset containing the data that you want to use as variables:

You need  to start by clicking on that dataset, select API connect on the right side plugin panel, followed by API Connect recipe.

 In the recipe configuration panel, you then need to select all the columns that you will use as variables in you API calls. This is done in the Columns to use as variable section.

Then write the body. On this picture there is a example of a standard JSON body. You can see that the variable is called simply by using the {{variable_name}} syntax.

There could be subtleties, such as the need to add brackets before and after the variable in case if strings. I would recommend testing your settings by using the https://httpbin.org/anything API, which will just send you back your request defined as a JSON structure. This way you can check in the "data" section that the JSON format is valid and your variable was used as expected.

 Hope this helps,

Alex

[alec_peterson]

Hi Alex,

I am having an issue with the specific API I'm working with.

1) Before I can make API resource requests, I have to first make a POST request for an authorization token, and then use that token in the headers for subsequent API calls.

That part is successful and it's saving as a dataset with column name "api_response"

2) I am trying to use that token string in subsequent API calls i.e. using that dataset as an input to an API Connect recipe.

I've experimented with setting the  "Columns to use as variables" as "api_response", then putting the headers Key as "Authorization" (per my API's documentation) and the Value as "Bearer {{api_response}}" but it's giving an error that the token is invalid. I've verified the token string from the input dataset is correct.

I've tried reformatting the raw text per your example, as well as different versions of adding brackets and curly braces outside (per your comment about handling for strings) but it still gives the error.

Do you have any advice on how to refer to the token string stored in the "api_response" column of my input dataset?

[alec_peterson]

I was able to address this. In experimenting with the https://httpbin.org/anything

,the string being input from my first API call actually had quotes at the start and end of the token characters.

Instead of inputting the authorization token dataset (from the first API call), I put a Prepare recipe in between that got rid of the quotes on the ends and added "Bearer " to the start, giving a final output of "Bearer xxxxxxxxxx" (not including the quotes, and xxxxxxxxxx standing for the token string).

Using this new as a variable, and then inputting that variable via {{column_name}} form into the Value part of the headers field worked!

[CoreyS]

Thank you for sharing your solution with the rest of this community @alec_peterson!