partitioning
If I have a flow where I have partitioned files, and output file is not partitioned since its small
Is that ok,
when date changes on partitioned tables that are being consumed, is that going to have any impact on output
Answers
-
Hi Venu,
When you use partitioning feature, in all recipes you need to configure (in input/output tab of the recipe) the partition dependencies.
If your input is partitioned by day, the output is not partitioned, the partition dependency is going to be set to "All Available": it means it's going to take all your partitions from the input to compute the output.You can watch this video training to understand better partitioning in DSS:
https://community.dataiku.com/t5/Online-Events/Partitioning-in-Dataiku-DSS-Watch-on-Demand/ba-p/6148
