Announcing the winners & finalists of the Dataiku Frontrunner Awards 2021! Read their inspiring stories

Cannot use SparkContext.getOrCreate()

pillsy
Level 1
Level 1
Cannot use SparkContext.getOrCreate()

I have been using the default PySpark notebook on a Dataiku instance which does have Spark set up, and cannot get past the very first step after the imports (which are successful modulo some apparently harmless DeprecationWarnings about docstrings).

 

Evaluating this cell in a Jupyter notebook gets stuck with an asterisk for an indefinite period of time:

# Load PySpark
sc = pyspark.SparkContext.getOrCreate()

Any ideas how to better diagnose or even resolve this issue?

Thanks!

0 Kudos
1 Reply
AlexT
Dataiker
Dataiker

Hi,

You could try to run the same code or the sample PySpark as a recipe instead of a Notebook and look at the job log for any errors.

If you are unable to find anything that could explain the issue I would suggest you open a support ticket with the job diagnostics. 

Kind Regards,

 

0 Kudos
A banner prompting to get Dataiku DSS