How to show Spark progress within Jupyter Notebook?

tall_one

I'm used to working in Jupyter in standard AWS EC2 instances and via WSL. In both of these, PySpark displays progress while performing queries / transformations. Is there a way to get the behaviour in Dataiku's Jupyter implementation?

As always, I have set "spark.ui.showConsoleProgress" to "true"; however, it does not appear to have any effect.

---

Operating system used: Windows

Alexandru

Hi @tall_one
,

You can't currently use showConsoleProgress in Notebook.
In spark recipes you can look at spark-history-server when running jobs instead of notebooks. By click on "Spark Event log"


Thanks