Write mlflow logs to a managed folder

Skanda Gurunathan
Skanda Gurunathan Registered Posts: 8 ✭✭✭

I am trying to use mlflow with dataiku and trying to write all the experiment tracking logs and metrics to a specific folder.

I can see the folder in the flow, but the logs and metrics dont appear there but it appears in the dataiku's ExperimentTracking UI. What's wrong in the code doing this?

project = dataiku.api_client().get_default_project()
managed_folder = project.get_managed_folder('YvNMItjP')

mlflow_extension = project.get_mlflow_extension()

with project.setup_mlflow(managed_folder=managed_folder) as mlflow_handle:
    print(mlflow_handle.get_tracking_uri())
  
    experiment_id = str(int(time.time()))
    mlflow_handle.create_experiment(experiment_id)
    mlflow_handle.set_experiment(experiment_id)

    with mlflow_handle.start_run(run_name="21-12-2023") as run:
        print(run.info.run_id)
        mlflow_handle.tracking.MlflowClient().set_experiment_tag(
            experiment_id, "library", "Keras")

        mlflow_handle.tracking.MlflowClient().set_experiment_tag(
            experiment_id, "predictionType", "BINARY_CLASSIFICATION")


Operating system used: Linux

Answers

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 2,024 Neuron

    Why do you want the logs and metrics written to the folder? What are you trying to achieve?

  • Skanda Gurunathan
    Skanda Gurunathan Registered Posts: 8 ✭✭✭

    I want to view the files and logs in that folder / kinda data governance thing. We dont want to log to a folder that is invisible to users in the flow. So basically instead of logging to default mlflow folder, I want to log it to a specific managed folder in the flow

Setup Info
    Tags
      Help me…