Dataiku not starting after reboot

Solved!
sandeep
Level 2
Dataiku not starting after reboot

Hi,

I ran the script that allowed Dataiku installer script to run automatically upon system boot.

sudo -i "/home/dataiku/dataiku-dss-8.0.4/scripts/install/install-boot.sh" "/home/dataiku/DATA_DIR" dataiku

However, after reboot, Dataiku doesn't turn on (see below code snippet). It seems that there is some issue with running nginx.

-bash-4.2$ DATA_DIR/bin/dss status
backend                          RUNNING   pid 7677, uptime 0:01:55
ipython                          RUNNING   pid 7675, uptime 0:01:55
nginx                            FATAL     Exited too quickly (process log may have details)

 Based on https://community.dataiku.com/t5/Setup-Configuration/dss-does-not-start-anymore-nginx-backoff/m-p/25... I examined the nginx logs in DATA_DIR/run/nginx.log based on  and , I found the following:

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/robots.txt" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /robots.txt HTTP/1.1", host: "10.85.235.54:11000"
2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/static/dataiku/img/glyphicons-halflings.png" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /static/dataiku/img/glyphicons-halflings.png HTTP/1.1", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/static/dataiku/css/style.css"
2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/static/dataiku/img/glyphicons-halflings-white.png" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /static/dataiku/img/glyphicons-halflings-white.png HTTP/1.1", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/static/dataiku/css/style.css"
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:01 [emerg] 9059#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:02 [emerg] 9759#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:05 [emerg] 10156#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:08 [emerg] 11377#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:08 [emerg] 7680#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:10 [emerg] 7779#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:12 [emerg] 7814#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:15 [emerg] 7849#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/get-processors-library/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/get-processors-library/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/list-custom-formulas-functions HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/list-custom-formulas-functions", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/get-expression-syntax HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/get-expression-syntax", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-types/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-types/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-common-charsets/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-common-charsets/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-format-types/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-format-types/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/flow/recipes/get-types-descriptors HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/flow/recipes/get-types-descriptors", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/get-configuration HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/get-configuration", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:19 [emerg] 7673#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:21 [emerg] 7780#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:23 [emerg] 7817#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:27 [emerg] 7848#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:34 [emerg] 7676#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:35 [emerg] 7783#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:38 [emerg] 7821#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:42 [emerg] 7853#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)

Can you help me to interpret this log file or help me resolve this issue to enable the nginx process to start properly after reboot?

0 Kudos
1 Solution
sergeyd
Dataiker

Hi @sandeep 

You have SELinux enabled on the host. Set it to permissive instead of enforcing. 

That's one of the requirements: https://doc.dataiku.com/dss/8.0/installation/requirements.html

 

View solution in original post

3 Replies
sergeyd
Dataiker

Hi @sandeep 

You have SELinux enabled on the host. Set it to permissive instead of enforcing. 

That's one of the requirements: https://doc.dataiku.com/dss/8.0/installation/requirements.html

 

sandeep
Level 2
Author

Thanks Sergey - this worked perfectly! Just wanted to check with you on something.

While SELinux was enabled, I was able to manually start nginx by running:

DATA_DIR/bin/dss start

However, after SELinus is disabled, nginx started automatically on reboot. Can you explain why this is the case?

0 Kudos
sergeyd
Dataiker

Yes, that's correct behaviour. The difference is that during reboot systemctl starts DSS. You should have added a boot-up script for this. That falls under SELinux security while starting/restarting DSS manually via dss binary has fewer limitations.