This website uses cookies. By clicking OK, you consent to the use of cookies.
Click Here
to learn more about how we use cookies.
OK
Community
All community
This category
This board
Knowledge base
Users
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
Search instead for
Did you mean:
Browse
Discussions
Setup & Configuration
Using Dataiku DSS
Plugins & Extending Dataiku DSS
General Discussion
Job Board
Community Resources
Product Ideas
Knowledge
Getting Started
Knowledge Base
Documentation
Academy
Quick Start Programs
Learning Paths
Certifications
Course Catalog
Academy Discussions
Community Programs
Dataiku Neurons
User Groups
User Groups Resources
Online Events
Upcoming Events
Past Events
Community Conundrums
Banana Data Podcast
Community Feedback
What's New
Sign In
New to Dataiku DSS? Try out our NEW Quick Start Programs today and get onboarded on the product in just one hour!
Let's go
Community
»
Discussions
»
Using Dataiku DSS
»
Options
Subscribe to RSS Feed
Mark Topic as New
Mark Topic as Read
Float this Topic for Current User
Bookmark
Subscribe
Mute
Printer Friendly Page
Spark pipeline merge rules
baobo
Level 1
12-02-2019
05:03 PM
Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Email to a Friend
Report Inappropriate Content
Spark pipeline merge rules
What kinds of visual receipts can be merged together during the job executions?
0
Kudos
Reply
All discussion topics
Previous Topic
Next Topic
2 Replies
Clément_Stenac
Dataiker
12-02-2019
05:07 PM
Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Email to a Friend
Report Inappropriate Content
Hi,
All visual recipes except split can be merged, provided that they all use Spark engine.
For each recipe, there are specific conditions that can prevent it from merging. In that case, you'll find the reason for non-mergeability in the logs.
0
Kudos
Reply
baobo
Level 1
Author
In response to
Clément_Stenac
12-06-2019
12:21 PM
Mark as New
Bookmark
Subscribe
Mute
Subscribe to RSS Feed
Permalink
Print
Email to a Friend
Report Inappropriate Content
Thanks Clément.
I notice there are some logs for non-mergeablity. But I cannot figure of the reason for the non-mergeablity.
Does spark pipeline in dataiku as same as the pipeline concept in Spark ML ?
if not, what is the difference between them?
Furthermore, can you please explain "pre-merge prune"? How can it optimize the job?
0
Kudos
Reply
Post Reply
Labels
(2)
Labels
Labels:
Flow
Spark