Get job memory details [Python API]

Options
RajeshRana84
RajeshRana84 Registered Posts: 5 ✭✭✭✭

Good Day All,

I am looking to get the following things for any job :

1. Total EKS Memory (ex. 20 GB)

2. Total EKS CPU (ex. 10 CPU)

3. Spark Driver RAM (ex. 4 GB)

4. Spark Driver CPU (ex. 2 CPU)

5. Is it a Local job or not ?

6. Is it using SPARK or not ?

I know how to get Job_logs for any job but will I find all the above details in the logs or Is there any other way to get all the above details for a particular job ?

Thanks,
Rajesh

Answers

  • Clément_Stenac
    Clément_Stenac Dataiker, Dataiku DSS Core Designer Posts: 753 Dataiker
    Options

    Hi,

    Dataiku has the ability to retrieve all of that information, but that requires some setup. Could you please get in touch with your Dataiku Customer Success Manager, who can help set up a session to go over this?

Setup Info
    Tags
      Help me…