Survey banner
The Dataiku Community is moving to a new home! New posts are now disabled and the community will shortly be in temporary read only mode: LEARN MORE

How to run Pyspark dataframe in Scenarios?

Level 2
How to run Pyspark dataframe in Scenarios?

I'm encountering a issue while attempting to run Spark Dataframe in specific scenario to retrive data,However Im facing error in this process.

0 Kudos
1 Reply

Hi @Ramya ,
When you say running spark dataframe in a scenario are you running a pyspark recipe by building an output dataset of that recipe?

Or are you trying to run PySpark code inside a Python Scenario step( this would not be supported)

You can run Spark code in PySpark Notebook or PySpark recipe you can only run Python code in Python scenario steps.

0 Kudos