Spark pipeline merge rules

baobo
baobo Registered Posts: 8 ✭✭✭✭
What kinds of visual receipts can be merged together during the job executions?
Tagged:

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer, Registered Posts: 753 Dataiker
    Hi,

    All visual recipes except split can be merged, provided that they all use Spark engine.

    For each recipe, there are specific conditions that can prevent it from merging. In that case, you'll find the reason for non-mergeability in the logs.
  • baobo
    baobo Registered Posts: 8 ✭✭✭✭
    Thanks Clément.
    I notice there are some logs for non-mergeablity. But I cannot figure of the reason for the non-mergeablity.

    Does spark pipeline in dataiku as same as the pipeline concept in Spark ML ?
    if not, what is the difference between them?

    Furthermore, can you please explain "pre-merge prune"? How can it optimize the job?
Setup Info
    Tags
      Help me…