Spark delta lake write support and structured streaming to delta tables
nod
Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 2 ✭✭✭
Delta lake is an open format thats already supported on open source spark, we would like to be able to write delta tables directly from dataiku with spark recipes.
This would enable more ETL use cases, and with spark structured streaming with delta tables better management of streaming data.