Using the next page url provided in the API Plugin

Emiel_Veersma Registered, Frontrunner 2022 Finalist, Dataiku Frontrunner Awards 2021 Participant, Frontrunner 2022 Participant Posts: 20 ✭✭✭✭✭

Hi all!

I'm trying to access the list applications endpoint of the Greenhouse API with the API plugin. The endpoint uses pagination and a next page is provided in the header. I've tried different setups with the API plugin, but I can't get it to work (while it works with a python script). The error I get is:

Failed to read from python dataset : <class 'rest_api_client.RestAPIClientError'> : The api-connect plugin is stuck in a loop. Please check the pagination parameters.

I'm wondering how I can debug this and get it to work. Help is much appreciated!


Best Answer

  • AlexB
    AlexB Dataiker Posts: 67 Dataiker
    Answer ✓

    Hi !

    This type of pagination is not yet supported by the API plugin. For now you can only retrieve the first page, or feed the page numbers using the api-connect recipe and a "page" query parameter for instance.


  • PhilipMannering
    PhilipMannering Dataiku DSS Core Designer, Dataiku DSS Adv Designer, Registered Posts: 1 ✭✭✭

    Is this now supported? I cannot get it to work.

  • AlexB
    AlexB Dataiker Posts: 67 Dataiker

    first check that the API you are trying to access is using RFC-5988 type pagination.

    - You will need to have a recent version of the plugin (v1.1.2 or later).

    - Then set the pagination mechanism to "Next page URL provided" and keep the "Key to next request URL" empty.

    - You might still have to set the "Next page URL is relative" on or off according to the API's specifications.

Setup Info
      Help me…