set_definition_and_payload() error on sync recipe

Meirkhan · October 2021

Hi Dataikers,

The same code works for SQL recipe. But I am having a hard time applying it to the sync recipe

builder_gcs = None
builder_gcs = SyncRecipeCreator(f"compute_{dataset_name}_gcs", project=dataiku_project)
builder_gcs = builder_gcs.with_input(dataset_name, dataiku_project_name)
builder_gcs = builder_gcs.with_new_output(f"{dataset_name}_gcs", GCS_CONNECTION)
recipe_gcs = builder_gcs.build()

recipe_def = recipe_gcs.get_definition_and_payload()
recipe_payload = recipe_def.get_payload()
# Modifications on definition - but does not work without modifs too
recipe_def.set_payload(recipe_payload)
recipe_gcs.set_definition_and_payload(recipe_def)

Error: DataikuException: java.lang.IllegalArgumentException: Cannot set unparseable payload on recipe recipe_name of type sync

Even though I did not do any changes on payload (and payload is empty string for sync anyway).

Any ideas?

Alexandru · October 2021

Hi,

Can you please confirm your exact DSS version? Since get_definition_and_payload() was deprecated.

You can instead try using get_settings()

Regardless. on DSS 9.0.5 . I wasn't able to replicate the issue with similar code :

Screenshot 2021-10-28 at 15.23.24.png

As you can see the Update recipe worked without any exception only warnings are related to the deprecations.

Meirkhan · October 2021

I have DSS version 7.0.2

I am starting to think that it is because DSS cannot find the recipe.

Cause among the errors I have:

HTTPError: 400 Client Error: Bad Request for url: http://127.0.0.1:10001/dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs

I tested by giving complete random name of recipe, which for sure does not exists and I am receiving the same error.

But it is the case only for the SYNC recipe and code works fine with other recipe types.

In this case another question raises - what can be the reason for DSS for not finding sync recipes?

Alexandru · October 2021

Hi,

Does the following work for an existing Sync recipe?

recipe = project.get_recipe("myrecipe")
status = recipe.get_status()
print(status)

Thanks,

Meirkhan · October 2021

Attached screen

Meirkhan · October 2021

I have a test instance of DSS v 9.0.4

I confirm that it works there too

Alexandru · October 2021

Can you try adding :

recipe_gcs = dataiku_project.get_recipe(f"compute_{dataset_name}_gcs")

Before :

recipe_def = recipe_gcs.get_definition_and_payload()
recipe_payload = recipe_def.get_payload()
recipe_def.set_payload(recipe_payload)
recipe_gcs.set_definition_and_payload(recipe_def)

In your original code and see if this still fails on DSS7 does your original code fail on DSS 9.0.4 as well?

Thanks

Meirkhan · October 2021

Yes I did.

Whataver option of accessing sync recipe I try, it works on DSS 9.0.4 and fails on DSS 7.0.2

Full error if it helps:

2021-10-28 17:56:14 http://127.0.0.1:10001 "GET /dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs HTTP/1.1" 200 1500
2021-10-28 17:56:14 http://127.0.0.1:10001 "PUT /dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs HTTP/1.1" 400 None

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
/opt/dataiku-dss-7.0.2/python/dataikuapi/dssclient.py in _perform_http(self, method, path, params, body, stream, files, raw_body)
    903                     stream = stream)
--> 904             http_res.raise_for_status()
    905             return http_res

/data/DSS/5.1/code-envs/python/Python36_2/lib/python3.6/site-packages/requests/models.py in raise_for_status(self)
    939         if http_error_msg:
--> 940             raise HTTPError(http_error_msg, response=self)
    941 

HTTPError: 400 Client Error: Bad Request for url: http://127.0.0.1:10001/dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs

During handling of the above exception, another exception occurred:

DataikuException                          Traceback (most recent call last)
<ipython-input-70-d85988868863> in <module>
      3 recipe_payload = recipe_def.get_payload()
      4 recipe_def.set_payload(recipe_payload)
----> 5 recipe_gcs.set_definition_and_payload(recipe_def)

/opt/dataiku-dss-7.0.2/python/dataikuapi/dss/recipe.py in set_definition_and_payload(self, definition)
     47         return self.client._perform_json(
     48                 "PUT", "/projects/%s/recipes/%s" % (self.project_key, self.recipe_name),
---> 49                 body=definition.data)     50 
     51     ########################################################

/opt/dataiku-dss-7.0.2/python/dataikuapi/dssclient.py in _perform_json(self, method, path, params, body, files, raw_body)
    918 
    919     def _perform_json(self, method, path, params=None, body=None,files=None, raw_body=None):
--> 920         return self._perform_http(method, path,  params=params, body=body, files=files, stream=False, raw_body=raw_body).json()
    921 
    922     def _perform_raw(self, method, path, params=None, body=None,files=None, raw_body=None):

/opt/dataiku-dss-7.0.2/python/dataikuapi/dssclient.py in _perform_http(self, method, path, params, body, stream, files, raw_body)
    909             except ValueError:
    910                 ex = {"message": http_res.text}
--> 911             raise DataikuException("%s: %s" % (ex.get("errorType", "Unknown error"), ex.get("message", "No message")))
    912 
    913     def _perform_empty(self, method, path, params=None, body=None, files = None, raw_body=None):

DataikuException: java.lang.IllegalArgumentException: Cannot set unparseable payload on recipe MEIR_SANDBOX.compute_parcel_minus_building_refacto_gcs of type sync

set_definition_and_payload() error on sync recipe

Answers

Categories

Setup Info

Tags