Dataiku not starting after reboot
Hi,
I ran the script that allowed Dataiku installer script to run automatically upon system boot.
sudo -i "/home/dataiku/dataiku-dss-8.0.4/scripts/install/install-boot.sh" "/home/dataiku/DATA_DIR" dataiku
However, after reboot, Dataiku doesn't turn on (see below code snippet). It seems that there is some issue with running nginx.
-bash-4.2$ DATA_DIR/bin/dss status backend RUNNING pid 7677, uptime 0:01:55 ipython RUNNING pid 7675, uptime 0:01:55 nginx FATAL Exited too quickly (process log may have details)
Based on https://community.dataiku.com/t5/Setup-Configuration/dss-does-not-start-anymore-nginx-backoff/m-p/2596, I examined the nginx logs in DATA_DIR/run/nginx.log based on and , I found the following:
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/robots.txt" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /robots.txt HTTP/1.1", host: "10.85.235.54:11000" 2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/static/dataiku/img/glyphicons-halflings.png" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /static/dataiku/img/glyphicons-halflings.png HTTP/1.1", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/static/dataiku/css/style.css" 2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/static/dataiku/img/glyphicons-halflings-white.png" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /static/dataiku/img/glyphicons-halflings-white.png HTTP/1.1", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/static/dataiku/css/style.css" nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 15:53:01 [emerg] 9059#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 15:53:02 [emerg] 9759#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 15:53:05 [emerg] 10156#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 15:53:08 [emerg] 11377#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 17:41:08 [emerg] 7680#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 17:41:10 [emerg] 7779#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 17:41:12 [emerg] 7814#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 17:41:15 [emerg] 7849#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/get-processors-library/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/get-processors-library/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/list-custom-formulas-functions HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/list-custom-formulas-functions", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/get-expression-syntax HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/get-expression-syntax", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-types/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-types/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-common-charsets/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-common-charsets/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-format-types/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-format-types/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/flow/recipes/get-types-descriptors HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/flow/recipes/get-types-descriptors", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" 2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/get-configuration HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/get-configuration", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/" nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:05:19 [emerg] 7673#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:05:21 [emerg] 7780#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:05:23 [emerg] 7817#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:05:27 [emerg] 7848#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:19:34 [emerg] 7676#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:19:35 [emerg] 7783#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:19:38 [emerg] 7821#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied) nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied) 2021/03/05 18:19:42 [emerg] 7853#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
Can you help me to interpret this log file or help me resolve this issue to enable the nginx process to start properly after reboot?
Best Answer
-
Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
Hi @sandeep
You have SELinux enabled on the host. Set it to permissive instead of enforcing.
That's one of the requirements: https://doc.dataiku.com/dss/8.0/installation/requirements.html
Answers
-
Thanks Sergey - this worked perfectly! Just wanted to check with you on something.
While SELinux was enabled, I was able to manually start nginx by running:
DATA_DIR/bin/dss start
However, after SELinus is disabled, nginx started automatically on reboot. Can you explain why this is the case?
-
Sergey Dataiker, Dataiku DSS Core Designer, Dataiku DSS & SQL, Dataiku DSS Core Concepts Posts: 365 Dataiker
Yes, that's correct behaviour. The difference is that during reboot systemctl starts DSS. You should have added a boot-up script for this. That falls under SELinux security while starting/restarting DSS manually via dss binary has fewer limitations.