Pyspark Code to Spark Dataframe

[verleger]

I am new to Dataiku.  I have been going up and down all over the Dataiku document  and trying to make sense of this documentation specially looking into the  Dataiku  Pyspark code recipe  for my project, but I can not find anything useful !.  Examples and syntax  as simple as how to convert a Dataiku format to spark dataframe ! how to connect to multiple snowflake tables while working in the python notebook in spark recipe  ?  Details of the the methods of the  Dataiku library and what is available , what they do and how to use them ? looks like this class is not even explained well with details on the methods it offers as if  interfering  unnecessarily with a standard spark libraries which makes the coding harder rather than easier ! if anyone can point out a better link to look at to get more info about all above that will be appreciated. 

Verleger 

(Topic title edited by moderator to be more descriptive.)

[CoreyS]

Hi @verleger and welcome to the Dataiku Community. The best way to receive a helpful response is to ensure that you post in the right area. In this case, your comment was moved as a question to the Using Dataiku discussion board. For more best practices, be sure to follow these guidelines: How do I ask a good question? 

In terms of your question, while you wait for a more complete response here are some resources you may find helpful if you haven't checked them out already:

• PySpark recipes (Documentation)
• DSS and Spark (Documentation)
• Python API interaction with PySpark (Documentation)
• Using PySpark in Dataiku (Knowledge Base)

I hope this helps!

[AlexT]

Hi @verleger ,

Just to add a bit to the documentation Corey shared there is a specific example of how to use pyspark in DSS here:

https://doc.dataiku.com/dss/latest/code_recipes/pyspark.html#anatomy-of-a-basic-pyspark-recipe

To ensure best performance please make sure you enable spark native integration on your snowflake connection: 

https://doc.dataiku.com/dss/latest/connecting/sql/snowflake.html#spark-native-integration

To read a dataset you would use dkuspark all available methods are documented here :
For example here we would load 2 snowflake datasets 

# Import Dataiku APIs, including the PySpark layer
import dataiku
from dataiku import spark as dkuspark
# Import Spark APIs, both the base SparkContext and higher level SQLContext
from pyspark import SparkContext
from pyspark.sql import SQLContext

sc = SparkContext()
sqlContext = SQLContext(sc)

dataset1 = dataiku.Dataset("name_of_the_dataset")
df1 = dkuspark.get_dataframe(sqlContext, dataset1)

dataset2 = dataiku.Dataset("name_of_the_dataset")
df2 = dkuspark.get_dataframe(sqlContext, dataset2)

Additionally, you may want to consider snowpark integration:  the https://doc.dataiku.com/dss/latest/connecting/sql/snowflake.html#snowpark-integration