Spark Performance warning: Input dataset is read in a non-distributed way
Alwaleed
Registered Posts: 8 ✭
I Setting up the Spark using K8s but I'm facing the Performance warning: Input dataset is read in a non-distributed way and this make the queries taking so much time
attached a screenshot need your support to avoid the problem 
Answers
-
Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 DataikerHi @Alwaleed
,
The warning simply means that Spark can't read the SQL dataset directly. So it will need to use DSS to read the data which will be slower because it's not distributed.
If you want to enable direct spark reads for compatible database you need to enable the option at the connection level :
Then you will need to enable on the dataset level
