Sign up to take part
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Registered users can ask their own questions, contribute to discussions, and be part of the Community!
Hi,
I have a service that continuously emits data. You start receiving data once you have connected opening a TCP connection and never stops until you terminate the connection.
I'd like to develop a custom plugin to be able to process that data on Dataiku how I can do that as data never ends?
Will "build" log overload the server?
Thanks
UPDATE:
We are loading data from a flight's metasearch service. They expose a data stream we consume polling from a TCP connection (https://github.com/gbrian/Flightmate-Stream). We plan to use Dataiku to parse, sanitize, ... data and the drop into Hadoop apart from applying the corresponding analysis and lab ๐
@alexander Hope this helps
Hi Swarna,
I am not a specialist on the topic but here is the link within our documentation regarding Kafka. https://doc.dataiku.com/dss/latest/streaming/kafka.html#connection-setup