How to run Pyspark dataframe in Scenarios?

Options
Ramya
Ramya Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 8

I'm encountering a issue while attempting to run Spark Dataframe in specific scenario to retrive data,However Im facing error in this process.

Tagged:

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    Hi @Ramya
    ,
    When you say running spark dataframe in a scenario are you running a pyspark recipe by building an output dataset of that recipe?

    Or are you trying to run PySpark code inside a Python Scenario step( this would not be supported)

    You can run Spark code in PySpark Notebook or PySpark recipe you can only run Python code in Python scenario steps.
    Thanks,

Setup Info
    Tags
      Help me…