Python recipe parallelism & Distributed

NagarajuVarkala
NagarajuVarkala Registered Posts: 10 ✭✭✭

Hi,

How to achieve distributed & parallelism for python recipe in Dataiku DSS ?

Best Answer

  • arnaudde
    arnaudde Dataiker Posts: 52 Dataiker
    Answer ✓

    Hello,

    Note that if you use Dask you'll still need to provide the cluster (Kubernetes for example) that Dask will leverage. Our support covers DSS but neither the distribution and parallelization code with the Dask API nor maintenance of clusters.

    Most of our users use Spark for parallelization and distribution. If you enable Spark with DSS, you will be able to use Pyspark recipes with dedicated DSS integration https://doc.dataiku.com/dss/latest/code_recipes/pyspark.html


    Best,
    Arnaud

Answers

Setup Info
    Tags
      Help me…