Loading a Spark Dataframe to snowflake , python3.8 upgrade

verleger
Level 1
Loading a Spark Dataframe to snowflake , python3.8 upgrade

 How do I write a spark Dataframe to Snowflake table  from Jupyter notebook in Dataiku ? 

so much document, very little examples !

Thanks

0 Kudos
1 Reply
Clรฉment_Stenac

Hi,

Assuming that "myoutput" is the name of the Snowflake dataset in which you want to write, and that "df" is the Spark dataframe, you would write:

import dataiku
from dataiku import spark as dkuspark

myoutput = dataiku.Dataset("myoutput")
dkuspark.write_with_schema(out, df)

 

https://doc.dataiku.com/dss/latest/code_recipes/pyspark.html

Best,

0 Kudos