API Endpoint: Ingest Webhook Payload Data

[LucOBrien]

Hello, 

We have a partner company that is offering to deliver json data via a webhook and I am having trouble figuring out a way to take the data and store it back into a dataset in my project. 

I originally tried a SQL query endpoint and passed the parameters to a landing DB table via INSERT - this worked when using the "run test query" function. However, my IT department has squashed the use of the API Node to write directly. 

My next attempt has been to try to use the DSS public API to access a local dataset and write to it but I am getting the "No URL or API Key" error despite having them both in the code. 

Is it even possible for the API Node Endpoint to receive a POST request and use the data passed within DataIKU?

Thanks!

[Turribeach]

API endpoints as not really meant to use as a way to ingest data. I would also be wary of having a pattern where my API Node has an API endpoint which then writes back to the Designer Node. That looks like an anti-pattern to me. Having said that it's technically possible to have your API Node write back to a dataset into another node, although I would argue it should be an Automation node. You are probably not using the right API calls so post your code snippet in a code block if you want us to have a look at it. 

But even in that case you should be aware that most APIs tend to be stateless and that having a write back process breaks that premise. What happens if your webhook calls your API endpoint multiple times at the same time? Either one of them will fail or they will lock each other. For that reason a better approach will be to turn on query logging in the API node and then use those logs to load the data back. You can either use an NFS mount or some other network storage technology to share the log files with the Automation node.