Session consuming more space in yarn queue

Srkanoje Registered Posts: 32 ✭✭✭✭

We are using jupyter notebooks in dataiku project and doing some pypspark stuff. Sometimes, unfortunately we miss closing the session which is leading to holding the resources in yarn queue. Is there anyway that it can get auto-closed once we come out of the notebook/terminate the session after a while if idle.

Best Answer

  • fchataigner2
    fchataigner2 Dataiker Posts: 355 Dataiker
    Answer ✓

    (correction: hiveserver2 connection are cleaned up every 10 min, not 5 min)

    There is no control on the closing of these sessions. You can close them manually in yarn (ie yarn application -kill ...) . Another option is to touch the Hive settings, for example by changing a property of the hive connection in Administration > Settings > Hive, which will evict idle connections from the cache, and kill them.


Setup Info
      Help me…