Gitlab Nginx Reverse Proxy Configuration (with Let's Encrypt SSL)

gitlab.rb

# These are fragments of the configuration that will need to be updated

# ---

## GitLab URL
##! URL on which GitLab will be reachable.
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'https://gitlab.your-fqdn.com'
#external_url 'http://gitlab.internal.lan'

# ---

### Trusted proxies
###! Customize if you have GitLab behind a reverse proxy which is running on a
###! different machine.
###! **Add the IP address for your reverse proxy to the list, otherwise users
###!   will appear signed in from that address.**
gitlab_rails['trusted_proxies'] = ['10.100.0.0/15']

# ---

################################################################################
## GitLab NGINX
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html
################################################################################

# nginx['enable'] = true
# nginx['client_max_body_size'] = '250m'
# nginx['redirect_http_to_https'] = false
# nginx['redirect_http_to_https_port'] = 80

##! Most root CA's are included by default
# nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/ca.crt"

##! enable/disable 2-way SSL client authentication
# nginx['ssl_verify_client'] = "off"

##! if ssl_verify_client on, verification depth in the client certificates chain
# nginx['ssl_verify_depth'] = "1"

# nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
# nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"
# nginx['ssl_ciphers'] = "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256"
# nginx['ssl_prefer_server_ciphers'] = "on"

##! **Recommended by: https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html
##!                   https://cipherli.st/**
# nginx['ssl_protocols'] = "TLSv1.1 TLSv1.2"

##! **Recommended in: https://nginx.org/en/docs/http/ngx_http_ssl_module.html**
# nginx['ssl_session_cache'] = "builtin:1000  shared:SSL:10m"

##! **Default according to https://nginx.org/en/docs/http/ngx_http_ssl_module.html**
# nginx['ssl_session_timeout'] = "5m"

# nginx['ssl_dhparam'] = nil # Path to dhparams.pem, eg. /etc/gitlab/ssl/dhparams.pem
# nginx['listen_addresses'] = ['*', '[::]']

##! **Defaults to forcing web browsers to always communicate using only HTTPS**
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html#setting-http-strict-transport-security
# nginx['hsts_max_age'] = 31536000
# nginx['hsts_include_subdomains'] = false

##! **Docs: http://nginx.org/en/docs/http/ngx_http_gzip_module.html**
# nginx['gzip_enabled'] = true

##! **Override only if you use a reverse proxy**
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html#setting-the-nginx-listen-port
nginx['listen_port'] = 80

##! **Override only if your reverse proxy internally communicates over HTTP**
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl
nginx['listen_https'] = false

# nginx['custom_gitlab_server_config'] = "location ^~ /foo-namespace/bar-project/raw/ {\n deny all;\n}\n"
# nginx['custom_nginx_config'] = "include /etc/nginx/conf.d/example.conf;"
# nginx['proxy_read_timeout'] = 3600
# nginx['proxy_connect_timeout'] = 300
nginx['proxy_set_headers'] = {
 "Host" => "$http_host_with_default",
 #"X-Real-IP" => "$remote_addr",
 "X-Forwarded-For" => "$proxy_add_x_forwarded_for",
 "X-Forwarded-Proto" => "https",
 "X-Forwarded-Ssl" => "on",
 #"Upgrade" => "$http_upgrade",
 #"Connection" => "$connection_upgrade"
}
# nginx['proxy_cache_path'] = 'proxy_cache keys_zone=gitlab:10m max_size=1g levels=1:2'
# nginx['proxy_cache'] = 'gitlab'
# nginx['http2_enabled'] = true
nginx['real_ip_trusted_addresses'] = ['10.100.0.0/15']
nginx['real_ip_header'] = 'X-Real-IP'
nginx['real_ip_recursive'] = 'on'
# nginx['custom_error_pages'] = {
#   '404' => {
#     'title' => 'Example title',
#     'header' => 'Example header',
#     'message' => 'Example message'
#   }
# }

### Advanced settings
# nginx['dir'] = "/var/opt/gitlab/nginx"
# nginx['log_directory'] = "/var/log/gitlab/nginx"
# nginx['worker_processes'] = 4
# nginx['worker_connections'] = 10240
# nginx['log_format'] = '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
# nginx['sendfile'] = 'on'
# nginx['tcp_nopush'] = 'on'
# nginx['tcp_nodelay'] = 'on'
# nginx['gzip'] = "on"
# nginx['gzip_http_version'] = "1.0"
# nginx['gzip_comp_level'] = "2"
# nginx['gzip_proxied'] = "any"
# nginx['gzip_types'] = [ "text/plain", "text/css", "application/x-javascript", "text/xml", "application/xml", "application/xml+rss", "text/javascript", "application/json" ]
# nginx['keepalive_timeout'] = 65
# nginx['cache_max_size'] = '5000m'
# nginx['server_names_hash_bucket_size'] = 64

### Nginx status
# nginx['status'] = {
#  "enable" => true,
#  "listen_addresses" => ["127.0.0.1"],
#  "fqdn" => "dev.example.com",
#  "port" => 9999,
#  "options" => {
#    "stub_status" => "on", # Turn on stats
#    "server_tokens" => "off", # Don't show the version of NGINX
#    "access_log" => "off", # Disable logs for stats
#    "allow" => "127.0.0.1", # Only allow access from localhost
#    "deny" => "all" # Deny access to anyone else
#  }
# }

nginx.conf

# See the following links for getting Gitlab up and running with this configuration:
# https://gitlab.com/gitlab-org/gitlab-ce/issues/32937 Gitlab should not require X-Forwarded-Ssl: on if behind the HTTPS enabled reverse proxy when X-Forwarded-Proto: https is set
# https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/nginx.md#supporting-proxied-ssl
# https://docs.gitlab.com/omnibus/settings/nginx.html#change-the-default-proxy-headers
# https://gitlab.com/gitlab-org/gitlab-ce/issues/3538 Search for 'trusted_proxies' in the gitlab configuration file

server {
    server_name gitlab.your-fqdn.com;
    server_tokens off;

    # In global config (not needed)
    set $allowed 0;

    if ($us_country = yes) {
        set $allowed 1;
    }

    if ($local_redirect = yes) {
        set $allowed 1;
    }

    if ($allowed = 0) {
        return 444;
    }

    location / {
        client_max_body_size 0;
        gzip off;

        ## https://github.com/gitlabhq/gitlabhq/issues/694
        ## Some requests take more than 30 seconds.
        proxy_read_timeout 300;
        proxy_connect_timeout 300;
        proxy_redirect off;

        # Internal host name/FQDN
        proxy_pass http://gitlab.internal.lan;

        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-Ssl on;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_buffering off;
        proxy_http_version 1.1;
    }

    access_log /home/webmaster-data/www/gitlab/logs/access.log;
    error_log /home/webmaster-data/www/gitlab/logs/error.log;
    
    # Following configuration is maintained by Let's Encrypt/Certbot
    
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/gitlab.your-fqdn.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/gitlab.your-fqdn.com-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    
    if ($host = gitlab.your-fqdn.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name gitlab.your-fqdn.com;
    listen 80;
    return 404; # managed by Certbot

}

ty helped with my config issues.

Is the above nginx.conf the configuration of the gitlab (vm/container) nginx server or for the external internet facing nginx reverse proxy?

@JamieJ98 You're welcome.

@BrianDee67 They're two separate files that are for the Gitlab and Nginx (reverse proxy) instances, respectively. The gitlab configuration is what is needed to accept the forwarded headers, etc., and process them as if it was exposed to the public network. The nginx configuration is the other side that is exposed to the public network to make all that happen. I use certbot to enable SSL. So anything I'm hosting for public access goes through my Nginx reverse proxy server and then I configure the connection(s) to establish the SSL connection. Then the traffic that is forwarded to the internal server is configured as regular traffic, since it's not needed as all that traffic is happening in an isolated network ID in my environment with strong firewall rules put in-place to make sure traffic doesn't make its way through that isn't desired.

So, there's a bit of leg work that is involved to make sure it's as secure as you can reasonably make it. That part is up to the implementer/admin.

Thanks for your prompt reply. I implemented the configuration you provided and I did obtain a valid letsencrypt certificate however, still could not get the Gitlab sign-in page to load.
The url to the nginx reverse proxy server was secure (showing the locked symbol) but the page only returned a 503 Bad Gateway or 504 Timeout instead of loading the Gitlab site.

@BrianDee67 If you run nginx -t does it say the configuration is good? I'm not really sure how to troubleshoot further without error logs, etc.

Thanks for your prompt reply. I implemented the configuration you provided and I did obtain a valid letsencrypt certificate however, still could not get the Gitlab sign-in page to load. The url to the nginx reverse proxy server was secure (showing the locked symbol) but the page only returned a 503 Bad Gateway or 504 Timeout instead of loading the Gitlab site.

You can change: proxy_pass http://gitlab.internal.lan; to proxy_pass https://[IP LAN]:443/; it working for me :)

Thanks for your prompt reply. I implemented the configuration you provided and I did obtain a valid letsencrypt certificate however, still could not get the Gitlab sign-in page to load. The url to the nginx reverse proxy server was secure (showing the locked symbol) but the page only returned a 503 Bad Gateway or 504 Timeout instead of loading the Gitlab site.

You can change: proxy_pass http://gitlab.internal.lan; to proxy_pass https://[IP LAN]:443/; it working for me :)

You could do that, but I'm not sure why you wouldn't configure your Gitlab instance to listen to traffic on port 80, internally. I'd suspect that opens up possibilities of troubleshooting woes if something changes with future updates.

I also can't speak to how that might affect the forwarding of headers when logging activity from clients, correctly.

@Erutan409

I use haproxy with similar setup. It works. But fot git clone works only over https (after adding access token ex: https://gitlab-ci-token:@gitlab.example.com/repo/project). But it does not work over ssh. There is a method to forward port 22 from haproxy to gitlab server (https://www.haproxy.com/blog/route-ssh-connections-with-haproxy/)

So can it be done that way ? (I also didn't check, email url, email reply, upload attachment from clipboard to wiki.)

@khurshid-alam Yeah, I struggled with this, too. I ended up using a different port (766) for all SSH related connections to Gitlab. I have this port directly forward to that server, since you can't do similar reverse proxying with SSH. It doesn't support SNI like HTTP requests do; which is unfortunate. Anyway, this is what I did and it works.

I don't have a write-up on this as it's a little out-of-scope of what I meant for the gist to serve. Be careful and read up on how to properly set this up. It's mostly straight-forward, but just do your due diligence to make sure it's as secure as you can possibly make it. Don't expose the default SSH to the DMZ. You should be able to allow your server to respond to one SSH user on that alternate port, which is specifically used by Gitlab for using that protocol for Git.

Thanks for sharing, it helped me to solve my configuration issue.
Have you tested this configuration with gitlab pages?

@KazaiMazai No, I haven't. I didn't even know that was a thing until you asked. I'd imagine it would work, without requiring much deviation; if at all.