Survey banner
The Dataiku Community is moving to a new home! Some short term disruption starting next week: LEARN MORE

Write to artifacts to a folder from a pyspark job - Access issue

skandagn
Level 2
Write to artifacts to a folder from a pyspark job - Access issue

Hi, I am trying to write a pySpark.ml model (Pipeline model) from a pyspark recipe to a folder located in Server's filesystem storage and I am getting access denied error. How can I write to a folder from a pyspark recipe. 

pipeline = Pipeline(stages=[<listofencoders>])

# Fit and transform the data using the pipeline
pipeline_model = pipeline.fit(data)

pipeline_model.save(dataiku.Folder("model").get_path() + "/model")


Operating system used: Linux

0 Kudos
1 Reply
Turribeach

Hi, you can't write to Dataiku folders directly like that. You need to use the upload_stream() method:

https://knowledge.dataiku.com/latest/code/managed-folders/concept-managed-folders.html

 

0 Kudos