Keystone 5: Secure Cookies and Reverse Proxies

Can't sign in, eh?

TL;DR

When...

Keystone sessions are being used (eg. for authentication)
secureCookies Keystone config is true (the default when NODE_ENV is 'production')
The app server is behind a reverse proxy (like nginx)

You should ensure the following conditions are met:

Connections between the browser and the proxy are secure (ie. over HTTPS)
The proxy is configured to add a X-Forwarded-Proto header to requests
Keystone's Express server is configured to trust the proxy (ie. trust proxy is set to 1)

Otherwise, you won't be able to authenticate.

The Details

A lot of people have trouble getting cookies to work when they first deploy Keystone to a production environment with a reverse proxy. This usually manifests and an inability to sign into the Admin UI -- users enter a correct credentials but, when submitted, they are bounced back to an empty sign in form. Inspection of network requests shows no Set-Cookie header being received from the server despite the GraphQL mutation returning the correct user data.

Several factors combine to cause this issue:

`NODE_ENV`

The NODE_ENV environment variable is a de-facto standard used by Node.js applications to distinguish between development and production environments. Keystone also provides a CLI for common production operations such as building assets and starting the node process. As such, it's typical for a Keystone application to be run in production with commands similar to this:

NODE_ENV=production keystone build
NODE_ENV=production keystone start

Secure Cookies

When Keystone is used in an environment with NODE_ENV set to 'production', it automatically defaults it's secureCookies config to true.

As though the Keystone object was created with:

const keystone = new Keystone({
  // ...
  secureCookies: process.env.NODE_ENV === 'production', // Default to true in production
});

This secureCookies value makes it's way to the cookie.secure config for the express-session session middleware. There, the option instructs the resultant middleware to set the Secure attribute on Set-Cookie headers returned from the app.

Technically, the Secure attribute is intended for the client -- it instructs browsers to only send the cookie to the server if the request is made over HTTPS. By default however, the express-session package goes further. When the Secure attribute is set on new cookie, express-session will not send the Set-Cookie headers to the client unless the connection is secure. This is slightly more secure and a fairly sensible choice on the part of Express. (Chrome and Firefox create a similar effect by not storing Secure cookies unless they were received over HTTPS.)

Reverse Proxies

Reverse proxies are, by their nature, deployed "close" to the application servers, usually on the same private network. Although secure communication (HTTPS) is needed over the public Internet (ie. between the browser and the reverse proxy) it's often deemed unnecessary for traffic over a private network (ie. between the reverse proxy and the app server). As such, TLS (SSL) is often terminated at the proxy, with requests between the proxy and app being performed over plain HTTP.

As described above, the behaviour of express-session is to only send Secure cookies if the request is received over HTTPS. If you have a reverse proxy that terminates TLS, this is not longer the case. For this behaviour to be adjusted, two changes must be made:

`X-Forwarded-Proto` Header

X-Forwarded-... headers are the de-facto standard method proxies use to pass information about the incoming request upstream to the app server. We're specifically interested in the X-Forwarded-Proto header, used to indicate the protocol (http or https) used by the request received by the proxy.

Strictly speaking, only the X-Forwarded-Proto header is required to resolve the secure cookie problem. In practice, you'll probably want to the some other X-Forwarded-... headers; they're often required for other reasons and are usually a good idea.

If you're using nginx, the location block that contains your proxy_pass directive might include these proxy_set_header directives:

# Set additional headers on the upstream request
proxy_set_header   X-Real-IP           $remote_addr;
proxy_set_header   X-Forwarded-For     $proxy_add_x_forwarded_for;
proxy_set_header   X-Forwarded-Proto   $scheme;
proxy_set_header   X-Forwarded-Host    $host;
proxy_set_header   X-Forwarded-Port    $server_port;

Express `trust proxy` Setting

Including the X-Forwarded-Proto header on upstream request is not enough. Express will ignore these headers unless we instruct it not to using the trust proxy setting.

Like other Express settings, trust proxy is configured using app.set(). A number of values will work. For our purposes either the value true, the Number 1 or the IP address of your nginx server should suffice.

Accessing the Express app object in a Keystone project isn't entirely obvious but the Custom Servers guide tells us how -- we can export a configureExpress function from the project's entrypoint (usually index.js).

The resultant code may look something like this:

// ...

module.exports = {
  keystone,
  apps: [
    new GraphQLApp(),
    new AdminUIApp({ authStrategy, enableDefaultRoute: true }),
  ],
  configureExpress: app => {
    app.set('trust proxy', true);
  },
};

Deployment

You've likely now mad changes to both code and nginx config. Depending on how your production environment is configured this may complicate deployment. Ensure both the app code is deployed and nginx service is reloaded.

Troubleshooting

Still having problems, eh? That sucks. Here are some approaches that might help.

Simple Repo Case

As always, make sure you have a simple reproduction case. You can test the auth mutation directly and check the returned headers using curl. Something like this will suffice:

curl 'https://ks5proxytest.local/admin/api' \
--silent --dump-header - \
--data-binary '{"query":"mutation { authenticate: authenticateUserWithPassword(email: \"[email protected]\", password: \"qwerty\") { item { id } } }"}'

Here we have:

https://ks5proxytest.local/admin/api -- The URL of the GraphQL endpoint
--silent -- Don't show progress bar
--dump-header - -- Write the response headers to stdout
--data-binary '...' -- The GraphQL payload

If you're debugging with a self-signed certificate you'll also need --insecure.

Debugging Cookies

express-session doesn't add Set-Cookie as a normal header, it's generated using the on-headers package just before the response is sent. This makes it difficult to determine whether the header isn't being set or whether it's being dropped somewhere else, before it reaches the browser.

You need to use the on-headers package yourself to add a listener that can access the value. This can be done through Keystone's configureExpress() function with code similar to:

const onHeaders = require('on-headers');
const configureExpress = app => {
  // Add middleware to add a listener that can access the cookie header before the response is sent
  app.use((req, res, next) => {
    onHeaders(res, () => {
      // Should be an array; let's join it together
      const headerValue = Array.isArray(res.getHeader('set-cookie')) ? res.getHeader('set-cookie').join(' ') : '';
      console.log('Set-Cookie response header being set as...\nSet-Cookie: ', headerValue);
    });
    next();
  });

  app.set('trust proxy', true);
};

Reproduce in Dev

It's a bit of effort but you can reproduce this entire scenario in dev by:

Forcing the secureCookies Keystone config to true
Installing nginx
Configuring a server block that..
- Proxies to localhost:3000
- Uses a self-signed certificate

VinayaSathyanarayana/Keystone 5 - Secure Cookies and Reverse Proxies.md