RestException: INTERNAL_ERROR: Ticket not given or unrecognized while using MLFlow commands

Chaitra
Chaitra Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 5 ✭✭✭

Hi
I am working on logging fine tuned hugging face model to MLflow from Dataiku notebook.
I am not sure why I am facing Internal error, it works fine at times but again starts giving this error, at the simplest step creating experiment

Not sure if the error is in code or notebook kernel or authentication


RestException Traceback (most recent call last)
<ipython-input-8-8fd008361684> in <cell line: 1>()
13 print("Experiment created with name : {}".format(EXPERIMENT_NAME))
14
---> 15 experiment = mlflow.get_experiment_by_name(EXPERIMENT_NAME)
16 experiment_id = experiment.experiment_id
17 print("Experiment name is: {} and ID is: {} ".format(EXPERIMENT_NAME,experiment_id))

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/tracking/fluent.py in get_experiment_by_name(name)
1628 Creation timestamp: 1662004217511
1629 """
-> 1630 return MlflowClient().get_experiment_by_name(name)
1631
1632

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/tracking/client.py in get_experiment_by_name(self, name)
1254 Lifecycle_stage: active
1255 """
-> 1256 return self._tracking_client.get_experiment_by_name(name)
1257
1258 def create_experiment(

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/tracking/_tracking_service/client.py in get_experiment_by_name(self, name)
499 :py:class:mlflow.entities.Experiment
500 """
--> 501 return self.store.get_experiment_by_name(name)
502
503 def create_experiment(self, name, artifact_location=None, tags=None):

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py in get_experiment_by_name(self, experiment_name)
517 try:
518 req_body = message_to_json(GetExperimentByName(experiment_name=experiment_name))
--> 519 response_proto = self._call_endpoint(GetExperimentByName, req_body)
520 return Experiment.from_proto(response_proto.experiment)
521 except MlflowException as e:

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/store/tracking/rest_store.py in _call_endpoint(self, api, json_body, endpoint)
80 endpoint, method = _METHOD_TO_INFO[api]
81 response_proto = api.Response()
---> 82 return call_endpoint(self.get_host_creds(), endpoint, method, json_body, response_proto)
83
84 def search_experiments(

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/utils/rest_utils.py in call_endpoint(host_creds, endpoint, method, json_body, response_proto, extra_headers)
368 response = http_request(**call_kwargs)
369
--> 370 response = verify_rest_response(response, endpoint)
371 response_to_parse = response.text
372 js_dict = json.loads(response_to_parse)

/opt/dataiku/code-env/lib/python3.9/site-packages/mlflow/utils/rest_utils.py in verify_rest_response(response, endpoint)
238 if response.status_code != 200:
239 if _can_parse_as_json_object(response.text):
--> 240 raise RestException(json.loads(response.text))
241 else:
242 base_msg = (

RestException: INTERNAL_ERROR: Ticket not given or unrecognized

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,225 Dataiker

    Hi,
    Could please try running this code in test recipe instead of notebook, grab the failed job diagnostics + instance diagnostics and send this to dataiku support via a ticket :

    https://doc.dataiku.com/dss/latest/troubleshooting/obtaining-support.html

    It seems like there is an issue with the mlflow tracking server that should be started after → start_run()

    https://doc.dataiku.com/dss/latest/mlops/experiment-tracking/tracking.html#quick-start-sample

    Thanks

  • Chaitra
    Chaitra Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Registered Posts: 5 ✭✭✭

    Hi,
    Thanks, Yes I tried in recipe as well, still it failed
    I have created a ticket and share the diagnostic report


    Thanks & Regards

Setup Info
    Tags
      Help me…