Partition building : Ability to specify the order of built partition
Hello,
As counter-intuittive as it is, when you build partitions of a dataset, it doesn't follow an order but runs parallel to a certain extent. We can avoid the parellism in the settings of the recipe but it doesn't seem to respect the partition order.
In my current project, I have to deal with recursivity (let's say my partition n needs the partition n-1) and the order of buildings does matter.
I would like a setting on the dataset (or even on the recipe) to specify the build order with 3 possibilities (none-default, ascending, descending).
edit : the whole idea here is to have something simple and natural, not a complex python turnaround.
Best regards,
Simon
Comments
-
Elie Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered, Product Ideas Manager Posts: 33 DataikerHello @simon_aubert ,
Thank you for your submission.
Partitions are executed as independent "activities" within a job. Dataiku computes partition dependencies first (working backward from the target partition through the flow), then executes the resulting activities — but that dependency graph doesn't carry any notion of "partition N depends on partition N-1 having already been built."
It's built to answer "what inputs does this partition need," not "in what order should partitions run." To build a requested partition, Dataiku starts from the last dataset and works backwards to compute the partition dependencies before the job actually executes.
The Dataiku team has acknowledged that your idea meets the requested format. We will now work to ensure that your idea is reviewed by the appropriate internal stakeholder.
Thanks,
Have a nice day!