No Logging "DEBUG INFO Message " in the log file at project

NerGuRy
NerGuRy Registered Posts: 3

hello.
I would like to inquire about log settings.
“Jobs” and “Scenarios” are being performed every minute in my dataiku project.
I am keeping one week's worth of logs, but there is "[02:09:41] [INFO] [dku.utils] - [2023-11-20 02:09:40,211] [73/MainThread] [DEBUG] " in the log
Because there are a lot of debug INFO logs, 3M files are created every minute of execution, which means a lot of storage usage.
I would like to reduce storage usage by loading only error logging without saving the debug INFO log.
Please tell me where I can set this.
thank you.


Operating system used: linux


Operating system used: linux

Tagged:

Answers

  • Alexandru
    Alexandru Dataiker, Dataiku DSS Core Designer, Dataiku DSS ML Practitioner, Dataiku DSS Adv Designer, Registered Posts: 1,212 Dataiker

    Hi @NerGuRy
    ,

    Reducing the verbosity of logs can affect your ability to support you if you have job failures or issues.
    So we would recommend staying on the default log levels. Moving to ERROR only would hinder the ability to support you.

    The output.log is not the only file created whenever a job is run. A snapshot of the project configuration is also performed so it can be referenced during an investigation of a potential failure.

    If you have any particular very large output logs files created please share those job diagnostics over a support to investigate.

    Kind Regards,

  • Turribeach
    Turribeach Dataiku DSS Core Designer, Neuron, Dataiku DSS Adv Designer, Registered, Neuron 2023 Posts: 1,981 Neuron

    I agree with Alex. Rather than modifying the verbosity of the logs I would suggest you increase the purging of old logs for those projects that run high frequency of jobs. Have a look at the project macros that can help you with you with this. You can schedule project macros in a scenario as well.

  • NerGuRy
    NerGuRy Registered Posts: 3

    Thanks for yor help.

    I understood what you mean, and I added JOB which process deleting log file.

    Have a Good Day^^

  • NerGuRy
    NerGuRy Registered Posts: 3

    Thanks for yor help.

    I will check Macro Func.

    Have a Nice Day ^^

Setup Info
    Tags
      Help me…