Dataiku not starting after reboot

sandeep

Hi,

I ran the script that allowed Dataiku installer script to run automatically upon system boot.

sudo -i "/home/dataiku/dataiku-dss-8.0.4/scripts/install/install-boot.sh" "/home/dataiku/DATA_DIR" dataiku

However, after reboot, Dataiku doesn't turn on (see below code snippet). It seems that there is some issue with running nginx.

-bash-4.2$ DATA_DIR/bin/dss status
backend                          RUNNING   pid 7677, uptime 0:01:55
ipython                          RUNNING   pid 7675, uptime 0:01:55
nginx                            FATAL     Exited too quickly (process log may have details)

Based on https://community.dataiku.com/t5/Setup-Configuration/dss-does-not-start-anymore-nginx-backoff/m-p/2596, I examined the nginx logs in DATA_DIR/run/nginx.log based on and , I found the following:

nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/robots.txt" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /robots.txt HTTP/1.1", host: "10.85.235.54:11000"
2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/static/dataiku/img/glyphicons-halflings.png" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /static/dataiku/img/glyphicons-halflings.png HTTP/1.1", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/static/dataiku/css/style.css"
2021/01/08 15:13:46 [error] 17560#0: *1 open() "/home/dataiku/dataiku-dss-8.0.4/frontend/static/dataiku/img/glyphicons-halflings-white.png" failed (2: No such file or directory), client: 10.85.235.54, server: , request: "GET /static/dataiku/img/glyphicons-halflings-white.png HTTP/1.1", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/static/dataiku/css/style.css"
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:01 [emerg] 9059#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:02 [emerg] 9759#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:05 [emerg] 10156#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 15:53:08 [emerg] 11377#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:08 [emerg] 7680#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:10 [emerg] 7779#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:12 [emerg] 7814#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:41:15 [emerg] 7849#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/get-processors-library/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/get-processors-library/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/list-custom-formulas-functions HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/list-custom-formulas-functions", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/shaker/get-expression-syntax HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/shaker/get-expression-syntax", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-types/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-types/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-common-charsets/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-common-charsets/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *2 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/datasets/get-format-types/ HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/datasets/get-format-types/", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/flow/recipes/get-types-descriptors HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/flow/recipes/get-types-descriptors", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
2021/03/05 17:57:35 [error] 8723#0: *3 connect() failed (111: Connection refused) while connecting to upstream, client: 10.85.239.3, server: , request: "GET /dip/api/get-configuration HTTP/1.1", upstream: "http://127.0.0.1:11001/dip/api/get-configuration", host: "10.85.235.54:11000", referrer: "http://10.85.235.54:11000/home/"
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:19 [emerg] 7673#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:21 [emerg] 7780#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:23 [emerg] 7817#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:05:27 [emerg] 7848#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:34 [emerg] 7676#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:35 [emerg] 7783#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:38 [emerg] 7821#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)
nginx: [alert] could not open error log file: open() "/var/log/nginx/error.log" failed (13: Permission denied)
2021/03/05 18:19:42 [emerg] 7853#0: open() "/home/dataiku/DATA_DIR/install-support/nginx.conf" failed (13: Permission denied)

Can you help me to interpret this log file or help me resolve this issue to enable the nginx process to start properly after reboot?

Sergey

Hi @sandeep

You have SELinux enabled on the host. Set it to permissive instead of enforcing.

That's one of the requirements: https://doc.dataiku.com/dss/8.0/installation/requirements.html

sandeep

Thanks Sergey - this worked perfectly! Just wanted to check with you on something.

While SELinux was enabled, I was able to manually start nginx by running:

DATA_DIR/bin/dss start

However, after SELinus is disabled, nginx started automatically on reboot. Can you explain why this is the case?

Sergey

Yes, that's correct behaviour. The difference is that during reboot systemctl starts DSS. You should have added a boot-up script for this. That falls under SELinux security while starting/restarting DSS manually via dss binary has fewer limitations.