Accessing audit log with API

Options
MRvLuijpen
MRvLuijpen Partner, L2 Admin, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 107 Neuron

Hello Community,

We were wondering if it would be possible to access the information inside the audit.log files with the use of Dataiku API. We are not able (security/DevOps doesn't allow this) to make a file connection to the run/audit files.

We have done some analysis on the audit log files (by downloading/uploading 2,1 GB logfiles) and we found several very useful information in these files.

We would really be helped with some sample code.

Thanks in advance,

Marc Robert

Best Answer

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker
    Answer ✓
    Options

    Hello,

    Thank you for the confirmation. Unfortunately, it is currently not possible to read the audit logs through the API but our engineers are aware of this request.

    Audit logs are rotated at 100MB and up to 20 files are saved before rotated. However, you can modify the storage of the log location and the rotation described, here, to a location where you are able to read the files either through DSS or through python. Once this is done, you will not have to worry about DSS removing files, and you will be able to control how often these files are updated / accessed.

    An alternative situation that you may want to look into is Event Server described, here.

    Sincerely,

    Roy

Answers

  • RoyE
    RoyE Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 31 Dataiker
    Options

    Hello Marc,

    Just for clarification, do you mean your security team does not want you to create an DSS internal managed dataset that is pointing to your <DATA_DIR>/run/audit folder? If so, do you happen to know the reason for this?

    If you are able to do the above, please follow the steps below.

    First, you will need to set up a connection that is pointing your data directory.

    Screen Shot 2021-10-09 at 9.22.06.png

    Once set up, you can create a filesystem that points to your /run/audit/ folder path to create a dataset based of your audit logs.

    Screen Shot 2021-10-09 at 9.23.13.pngScreen Shot 2021-10-09 at 9.24.21.png

    Unfortunately, we do not have have any APIs that are currently able to access the audit logs.

    Roy

  • MRvLuijpen
    MRvLuijpen Partner, L2 Admin, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 107 Neuron
    Options

    Hi Roy,

    Thanks for you reply. And you are right.

    "The actual reason for the security team to not wanting to create an DSS internal managed dataset from the <DATA-DIR>/run/audit folder is that we/they do not know how, what and when Dataiku will lock these logging. And for the security process it is more important to have a complete audit log instead of read accessing the audit logs. "

    With kind regards

  • MRvLuijpen
    MRvLuijpen Partner, L2 Admin, L2 Designer, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS ML Practitioner, Dataiku DSS Core Concepts, Neuron 2020, Neuron, Dataiku DSS Adv Designer, Registered, Dataiku DSS Developer, Neuron 2021, Neuron 2022, Frontrunner 2022 Finalist, Frontrunner 2022 Winner, Frontrunner 2022 Participant, Neuron 2023 Posts: 107 Neuron
    Options
    Thank you for this update
Setup Info
    Tags
      Help me…