bizarre nginx issues caused by incorrect file permissions

nginx_issues.md

issues experienced

nginx error.log shows permission denied

nginx log in /var/log/nginx/error.log shows intereseting Error 13: Permission denied:

2017/11/14 00:33:30 [crit] 5117#5117: *224 open() "/var/lib/nginx/proxy/3/02/0000000023" failed (13: Permission denied) while reading upstream, client: 93.136.115.2, server: git.domain.tld, request: "GET /css/semantic-2.2.10.min.css HTTP/1.1", upstream: "http://127.0.0.1:3000/css/semantic-2.2.10.min.css", host: "git.domain.tld"

random images/css styles not loading

Browser reports broken link. curl reports 200 ok. (!!!?) after 10 force reloads of web page, maybe everything will load correctly. expirienced via ssl/non-ssl/http1 or http2.

upstream git server ssl issues on older libssl (ubuntu/windows)

rpc failed error 56

$ git clone https://git.domain.tld/seven/test.git .
Cloning into '.'...
Username for 'https://git.domain.tld': seven
Password for 'https://[email protected]':
remote: Counting objects: 1359, done.
remote: Compressing objects: 100% (1332/1332), done.
error: RPC failed; result=56, HTTP code = 200
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed

exit status 141 - {error: git-upload-pack died of signal 13

2017/11/13 21:02:58 http: multiple response.WriteHeader calls
2017/11/13 21:02:58 [ERROR] [.../routes/repo/http.go:284 serviceRPC()] HTTP.serviceRPC: fail to serve RPC 'upload-pack': exit status 141 - {error: git-upload-pack died of signal 13
 %!s(int=0)  %!s(bytes.readOp=0)}
[Macaron] 2017-11-13 21:02:58: Completed POST /seven/test.git/git-upload-pack 500 Internal Server Error in 126.261691ms

Stack overflow did not help much, since for this error - there is 100 possible reasons listed: https://stackoverflow.com/questions/15240815/git-fatal-the-remote-end-hung-up-unexpectedly

"error: git-upload-pack died of signal 13" https://confluence.atlassian.com/bitbucketserverkb/error-git-upload-pack-died-of-signal-13-779171770.html The error above does not represent an issue with Server. It means that a Git client has closed its connection prematurely.

Solution

1. Find out the owner of nginx process:

ps aux | grep "nginx: worker process"
nginx     5117  0.0  0.1 125644  6676 ?        S    00:30   0:00 nginx: worker process

2. Check if owner of process is the same as owner of folders/files:

sudo ls -l /var/lib/nginx/proxy/
total 40
drwx------  4 nginx root 4096 Nov 14 00:51 0
drwx------  5 nginx root 4096 Jan 19  2015 1
drwx------  5 nginx root 4096 Jan 19  2015 2
drwx------  5 nginx root 4096 Jan 19  2015 3
drwx------  5 nginx root 4096 Nov 15 10:33 4
drwx------  5 nginx root 4096 Nov 15 10:33 5
drwx------  5 nginx root 4096 Nov 15 12:19 6
drwx------  5 nginx root 4096 Nov 15 12:19 7
drwx------  5 nginx root 4096 Nov 15 12:19 8
drwx------ 49 nginx root 4096 Nov 13 20:09 9

3. Depending of step 1, reset file permissions:

sudo chown -R nginx:root /var/lib/nginx/proxy/

or:

sudo chown -R www-data:root /var/lib/nginx/proxy/

and restart service:

sudo systemctl restart nginx

Cause

nginx log in /var/log/nginx/error.log shows intereseting Error 13: Permission denied:

2017/11/14 00:33:30 [crit] 5117#5117: *224 open() "/var/lib/nginx/proxy/3/02/0000000023" failed (13: Permission denied) while reading upstream, client: 93.136.115.2, server: git.domain.tld, request: "GET /css/semantic-2.2.10.min.css HTTP/1.1", upstream: "http://127.0.0.1:3000/css/semantic-2.2.10.min.css", host: "git.domain.tld"

While upgrading from Ubuntu 14.04 LTS to 16.04, nginx migration did not reset file permissions correctly in nginx temp folder /var/lib/nginx/proxy/. This cause some content to be available, and some not - all extremely hard to debug.

Notice folder 5 with owner root:root:

sudo ls -l /var/lib/nginx/proxy/
total 40
drwx------  4 nginx root 4096 Nov 14 00:51 0
drwx------  5 nginx root 4096 Jan 19  2015 1
drwx------  5 nginx root 4096 Jan 19  2015 2
drwx------  5 nginx root 4096 Jan 19  2015 3
drwx------  5 nginx root 4096 Nov 15 10:33 4
drwx------  5 root root 4096 Nov 15 10:33 5
drwx------  5 nginx root 4096 Nov 15 12:19 6
drwx------  5 nginx root 4096 Nov 15 12:19 7
drwx------  5 nginx root 4096 Nov 15 12:19 8
drwx------ 49 nginx root 4096 Nov 13 20:09 9

debug git

Run once to get full debug:

export GIT_TRACE_PACKET=1
export GIT_TRACE=1
export GIT_CURL_VERBOSE=1

Add GIT_CURL_VERBOSE=1 before your git command:

GIT_CURL_VERBOSE=1 git clone https://git.domain.tkd/seven/test.git .

I had the same issue and your solution fixed it 😃 Thanks for the root cause explanation

Very helpful. Thanks a lot.