How to show Spark progress within Jupyter Notebook?

tall_one
tall_one Registered Posts: 1

I'm used to working in Jupyter in standard AWS EC2 instances and via WSL. In both of these, PySpark displays progress while performing queries / transformations. Is there a way to get the behaviour in Dataiku's Jupyter implementation?

As always, I have set "spark.ui.showConsoleProgress" to "true"; however, it does not appear to have any effect.


Operating system used: Windows

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 Dataiker

    Hi @tall_one
    ,

    You can't currently use showConsoleProgress in Notebook.
    In spark recipes you can look at spark-history-server when running jobs instead of notebooks. By click on "Spark Event log"
    Screenshot 2023-11-14 at 1.03.26 PM.png

    Thanks

Setup Info
    Tags
      Help me…