set_definition_and_payload() error on sync recipe

Options
Meirkhan
Meirkhan Registered Posts: 20 ✭✭✭✭

Hi Dataikers,

The same code works for SQL recipe. But I am having a hard time applying it to the sync recipe

builder_gcs = Nonebuilder_gcs = SyncRecipeCreator(f"compute_{dataset_name}_gcs", project=dataiku_project)builder_gcs = builder_gcs.with_input(dataset_name, dataiku_project_name)builder_gcs = builder_gcs.with_new_output(f"{dataset_name}_gcs", GCS_CONNECTION)recipe_gcs = builder_gcs.build()recipe_def = recipe_gcs.get_definition_and_payload()recipe_payload = recipe_def.get_payload()# Modifications on definition - but does not work without modifs toorecipe_def.set_payload(recipe_payload)recipe_gcs.set_definition_and_payload(recipe_def)Error: DataikuException: java.lang.IllegalArgumentException: Cannot set unparseable payload on recipe recipe_name of type sync

Even though I did not do any changes on payload (and payload is empty string for sync anyway).

Any ideas?

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    Hi,

    Can you please confirm your exact DSS version? Since get_definition_and_payload() was deprecated.

    You can instead try using get_settings()

    Regardless. on DSS 9.0.5 . I wasn't able to replicate the issue with similar code :

    Screenshot 2021-10-28 at 15.23.24.png

    As you can see the Update recipe worked without any exception only warnings are related to the deprecations.

  • Meirkhan
    Meirkhan Registered Posts: 20 ✭✭✭✭
    Options

    I have DSS version 7.0.2

    I am starting to think that it is because DSS cannot find the recipe.

    Cause among the errors I have:

    HTTPError: 400 Client Error: Bad Request for url: http://127.0.0.1:10001/dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs

    I tested by giving complete random name of recipe, which for sure does not exists and I am receiving the same error.

    But it is the case only for the SYNC recipe and code works fine with other recipe types.

    In this case another question raises - what can be the reason for DSS for not finding sync recipes?

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    Hi,

    Does the following work for an existing Sync recipe?

    recipe = project.get_recipe("myrecipe")status = recipe.get_status()print(status)

    Thanks,

  • Meirkhan
    Meirkhan Registered Posts: 20 ✭✭✭✭
    Options
  • Meirkhan
    Meirkhan Registered Posts: 20 ✭✭✭✭
    Options

    I have a test instance of DSS v 9.0.4

    I confirm that it works there too

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,209 Dataiker
    Options

    Can you try adding :

    recipe_gcs = dataiku_project.get_recipe(f"compute_{dataset_name}_gcs")

    Before :

    recipe_def = recipe_gcs.get_definition_and_payload()
    recipe_payload = recipe_def.get_payload()
    recipe_def.set_payload(recipe_payload)
    recipe_gcs.set_definition_and_payload(recipe_def)

    In your original code and see if this still fails on DSS7 does your original code fail on DSS 9.0.4 as well?

    Thanks

  • Meirkhan
    Meirkhan Registered Posts: 20 ✭✭✭✭
    Options

    Yes I did.

    Whataver option of accessing sync recipe I try, it works on DSS 9.0.4 and fails on DSS 7.0.2

    Full error if it helps:

    2021-10-28 17:56:14 http://127.0.0.1:10001 "GET /dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs HTTP/1.1" 200 15002021-10-28 17:56:14 http://127.0.0.1:10001 "PUT /dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcs HTTP/1.1" 400 None
    ---------------------------------------------------------------------------HTTPError                                 Traceback (most recent call last)/opt/dataiku-dss-7.0.2/python/dataikuapi/dssclient.py in _perform_http(self, method, path, params, body, stream, files, raw_body)    903                     stream = stream)--> 904             http_res.raise_for_status()    905             return http_res/data/DSS/5.1/code-envs/python/Python36_2/lib/python3.6/site-packages/requests/models.py in raise_for_status(self)    939         if http_error_msg:--> 940             raise HTTPError(http_error_msg, response=self)    941HTTPError: 400 Client Error: Bad Request for url: http://127.0.0.1:10001/dip/publicapi/projects/MEIR_SANDBOX/recipes/compute_parcel_minus_building_refacto_gcsDuring handling of the above exception, another exception occurred:DataikuException                          Traceback (most recent call last)<ipython-input-70-d85988868863> in <module>      3 recipe_payload = recipe_def.get_payload()      4 recipe_def.set_payload(recipe_payload)----> 5 recipe_gcs.set_definition_and_payload(recipe_def)/opt/dataiku-dss-7.0.2/python/dataikuapi/dss/recipe.py in set_definition_and_payload(self, definition)     47         return self.client._perform_json(     48                 "PUT", "/projects/%s/recipes/%s" % (self.project_key, self.recipe_name),---> 49                 body=definition.data)     50     51     ########################################################/opt/dataiku-dss-7.0.2/python/dataikuapi/dssclient.py in _perform_json(self, method, path, params, body, files, raw_body)    918    919     def _perform_json(self, method, path, params=None, body=None,files=None, raw_body=None):--> 920         return self._perform_http(method, path,  params=params, body=body, files=files, stream=False, raw_body=raw_body).json()    921    922     def _perform_raw(self, method, path, params=None, body=None,files=None, raw_body=None):/opt/dataiku-dss-7.0.2/python/dataikuapi/dssclient.py in _perform_http(self, method, path, params, body, stream, files, raw_body)    909             except ValueError:    910                 ex = {"message": http_res.text}--> 911             raise DataikuException("%s: %s" % (ex.get("errorType", "Unknown error"), ex.get("message", "No message")))    912    913     def _perform_empty(self, method, path, params=None, body=None, files = None, raw_body=None):DataikuException: java.lang.IllegalArgumentException: Cannot set unparseable payload on recipe MEIR_SANDBOX.compute_parcel_minus_building_refacto_gcs of type sync
Setup Info
    Tags
      Help me…