Skip to content

Instantly share code, notes, and snippets.

@jasonmimick

jasonmimick/setting-up-tls-for-mongodb-on-pcf.md

Last active January 11, 2019 19:00
Show Gist options
  • Save jasonmimick/f77c2da78c2b6a9a7bf53070cadffd46 to your computer and use it in GitHub Desktop.
Save jasonmimick/f77c2da78c2b6a9a7bf53070cadffd46 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
setting-up-tls-for-mongodb-on-pcf.md

title: "Setting up SSL for the MongoDB Enterprise PCF Tile" author: [Jason Mimick] date: "2019-01-11" keywords: [Databases, MongoDB, Cloud Foundry, Pivotal] ...

Setting up SSL for the MongoDB Enterprise PCF Tile

This note describes the procedure for enabling TLS connections for MongoDB clusters deployed on PCF through the MongoDB Enterprise PCF Tile. Currently, the MongoDB PCF Tile does not support automatic configuration of TLS/SSL settings directly. Until this functionality is added to the tile, the recommended approach for configuring TLS is outlined in this note.

The general procedure is as follows:

  • Provision your root CA and db-server certificates.
  • Use cf create-service to create your MongoDB cluster.
  • Configure MongoDB Ops Manager Project TLS/Security settings.
  • Manually place certificates on MongoDB vms
  • Update MongoDB Ops Manager db settings with server cert.

Provision certificates

The first step is to get your certificates. You may already have been provided some root certificate authority, or, more likly need to create your own. To create your own CA, create a private key (.key), certificate (.crt), and .pem:

openssl genrsa -out rootCA.key 2048
openssl req -new -x509 -days 1826 -key rootCA.key -out rootCA.crt
cat rootCA.key rootCA.crt > rootCA.pem

Next, we will create a wildcard certificate for our MongoDB compute nodes. By default, the MongoDB VMs will get deployed with hostnames which match *.mongodb.internal. So we'll create a key, certificate signing request, and private key file as follows:

openssl genrsa -out pcf-mongodb.key 2048
# Use the '*.mongodb.internal' for "Common Name (eg, fully qualified host name)"
openssl req -new -key pcf-mongodb.key -out pcf-mongodb.csr
openssl x509 -req -in pcf-mongodb.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out pcf-mongodb.crt -days 500 -sha256

Create MongoDB deployment

For this use the Cloud Floundry cf tool. For example:

cf create-service mongodb-odb standalone_small opendb -c '{ "orgId" : "5bf47de7612ed81fa7963e26", "projectName" : "foo" }'

SSH into the PCF Ops Manager director and check the tasks.

ssh -i pcf.test.pcf-test.com-ssl-key.pem [email protected]
bosh -e azure --ca-cert /var/tempest/workspaces/default/root_ca_certificate --client ops_manager --client-secret zfP-a8mEjb4MbrxIJakfdSmK6tMsBph- tasks

Place certificates on MongoDB VMs

First, copy certificates to the PCF Ops Manager director vm:

scp -i pcf.test.pcf-test.com-ssl-key.pem ./rootCA.pem [email protected]:~
scp -i pcf.test.pcf-test.com-ssl-key.pem ./pcf-mongodb.pem [email protected]:~

Find the VMs for your service's deployment, eg:

bosh -e azure --ca-cert /var/tempest/workspaces/default/root_ca_certificate --client ops_manager --client-secret zfP-a8mEjb4MbrxIJakfdSmK6tMsBph- vms

Place the certs on each MongoDB vm in /var/vcap/store/mongodb-data and make sure the certs are owned by vcap:vcap.

#Copy rootCA.pem to MongoDB VM
bosh -e azure --ca-cert /var/tempest/workspaces/default/root_ca_certificate --client ops_manager --client-secret zfP-a8mEjb4MbrxIJakfdSmK6tMsBph- -d service-instance_5db662f0-57d0-4b14-b761-af95eeb6f2be scp ./rootCA.pem mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239:/var/vcap/store/mongodb-data

# Copy pcf-mongodb.pem to MongoDB VM
bosh -e azure --ca-cert /var/tempest/workspaces/default/root_ca_certificate --client ops_manager --client-secret zfP-a8mEjb4MbrxIJakfdSmK6tMsBph- -d service-instance_5db662f0-57d0-4b14-b761-af95eeb6f2be scp ./pcf-mongodb.pem mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239:/var/vcap/store/mongodb-data

# Fix permissions on pem files
bosh -e azure --ca-cert /var/tempest/workspaces/default/root_ca_certificate --client ops_manager --client-secret zfP-a8mEjb4MbrxIJakfdSmK6tMsBph- -d service-instance_5db662f0-57d0-4b14-b761-af95eeb6f2be ssh -c 'sudo chown vcap:vcap /var/vcap/store/mongodb-data/*.pem' mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239

Finally, we can list all pem file in the direcotry to validate permissions:

ubuntu@test-ops-manager:~$ bosh -e azure --ca-cert /var/tempest/workspaces/default/root_ca_certificate --client ops_manager --client-secret zfP-a8mEjb4MbrxIJakfdSmK6tMsBph- -d service-instance_5db662f0-57d0-4b14-b761-af95eeb6f2be ssh -c 'ls -l /var/vcap/store/mongodb-data/*.pem' mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239
Using environment '10.0.8.6' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Using deployment 'service-instance_5db662f0-57d0-4b14-b761-af95eeb6f2be'

Task 3488. Done
mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239: stderr | Unauthorized use is strictly prohibited. All access and activity
mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239: stderr | is subject to logging and monitoring.
mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239: stdout | -rw-r--r-- 1 vcap vcap 2956 Jan 11 17:13 /var/vcap/store/mongodb-data/pcf-mongodb.pem
mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239: stdout | -rw-rw-r-- 1 vcap vcap 2956 Jan 11 17:12 /var/vcap/store/mongodb-data/rootCA.pem
mongod_node/60201164-4473-4eeb-8e2b-4f208e8b6239: stderr | Connection to 10.0.4.19 closed.

You need to repeat this process for all the MongoDB nodes in your deployment. DO NOT modify any settings in MongoDB Ops Manager until you've successfully validated all certificates have been placed onto all MongoDB VM nodes.

Enable SSL for MongoDB Ops Manager Project

  • Navigate to the "Security" settings in MongoDB Ops Manager for your project.
  • Click on "Authentication & TLS/SSL", then the "EDIT SETTINGS" button.
  • Click "Next", and enable TSL/SSL.
  • Set the "TSL/SSL CA File Path" to /var/vcap/store/mongodb-data/rootCA.pem
  • Click "Next"
  • Enter the same file path for each Agents "PEM Key File"
  • Click "Save"
  • Back on the project page click "REVIEW & DEPLOY" and deploy those changes.
  • (Optional) ssh into one of your MongoDB VMs and run tail -f /var/vcap/sys/log/mongod_node/automation-agent-verbose.log in order to watch the automation agent logs and ensure there are no errors when pushing out this configuration.

Enable SSL for MongoDB deployment

Once the TSL settings for the project have successfully been deployed, there is one more step.

  • Find your deployment in MongoDB Ops Manager. (Click processes.) Then select the "MODIFY" button.
  • Scroll down to the "ADVANCED OPTIONS" click the "+Add Option", and add the following parameters:
    • sslPEMKeyFile : /var/vcap/store/mongodb-data/pcf-mongodb.pem
    • 'sslMode' : allowSSL
  • Click "Apply" and then "REVIEW & DEPLOY" your changes.
  • (Optional) Do the same ssh to tail the automation agent logs.

The last step is to, for each MongoDB node in your deployment, edit the ADVANCED OPTION bindIp to be 127.0.0.1,<your-vm-id>.mongodb.internal. Do this, and "REVIEW & DEPLOY" your changes.

Test your connection with the MongoDB shell

Finally, you're ready to test. Here we can ssh into a MongoDB VM and use the Mongo shell already installed to test our setup.

/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-3.6.9/bin/mongo --ssl --sslPEMKeyFile /var/vcap/store/mongodb-data/pcf-mongodb.pem --sslCAFile /var/vcap/store/mongodb-data/rootCA.pem --host $(hostname -f) --port 28000

Note

  • The default port for PCF Tile deployment is 28000.
  • Running the shell without the --host flag night result in this error:
/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-3.6.9/bin/mongo --ssl --sslPEMKeyFile /var/vcap/store/mongodb-data/pcf-mongodb.pem --sslCAFile /var/vcap/store/mongodb-data/rootCA.pem --port 28000
MongoDB shell version v3.6.9
connecting to: mongodb://127.0.0.1:28000/
2019-01-11T17:54:48.667+0000 E NETWORK  [thread1] The server certificate does not match the host name. Hostname: 127.0.0.1 does not match CN: *.mongodb.internal
2019-01-11T17:54:48.667+0000 E QUERY    [thread1] Error: socket exception [CONNECT_ERROR] for The server certificate does not match the host name. Hostname: 127.0.0.1 does not match CN: *.mongodb.internal :
connect@src/mongo/shell/mongo.js:257:13
@(connect):1:6
exception: connect failed

Adding --allowInvalidCertificates will work with a warning:

/var/lib/mongodb-mms-automation/mongodb-linux-x86_64-3.6.9/bin/mongo --ssl --sslPEMKeyFile /var/vcap/store/mongodb-data/pcf-mongodb.pem --sslCAFile /var/vcap/store/mongodb-data/rootCA.pem --port 28000 --sslAllowInvalidCertificates
MongoDB shell version v3.6.9
connecting to: mongodb://127.0.0.1:28000/
2019-01-11T17:59:58.455+0000 W NETWORK  [thread1] The server certificate does not match the host name. Hostname: 127.0.0.1 does not match CN: *.mongodb.internal
Implicit session: session { "id" : UUID("063dad2f-71d4-473f-bcfc-ce65f9f7a0a3") }
MongoDB server version: 3.6.9
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
	http://docs.mongodb.org/
Questions? Try the support group
	http://groups.google.com/group/mongodb-user
>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment