(Sample) Project devops file: Server Maintenance & Troubleshooting

sample-devops-file.md

(Sample) Project devops file: Server Maintenance & Troubleshooting

Project topology

• lucyJS: node.js server that handles HTTP requests for the web and iPhone clients
• kinderDB: mongoDB database that stores user-generated content + metrics
• nginx: reverse-proxy that maps our subdomains and ports to the right services, caches static files, and transforms HTTPS requests into HTTP

Domain/port mappings

• lucy.com -> bonfire:8080 (production server, with HTTPS support)
• dev.lucy.com -> bonfire:1097 (staging / pre-production server, with HTTPS support)
• test.lucy.com -> bonfire:1098 (staging / pre-production server, with HTTPS support)

Current deployments

• Box "bonfire" @ OVH (IP: XXX.XXX.XXX.XXX)
  • exposed services:
    • port 80: nginx proxy
    • port 8000: production "lucyJS" app server
    • port 1097: staging / pre-production "lucyJS" app server
    • port 1098: test server for staging / pre-production "lucyJS" app server
  • directories:
    • /home/lucyApp/.forever: logs of "forever"
    • /home/lucyApp/lucyJS: (production) clone of master branch from git + UGC uploads
    • /home/lucyApp/lucyStaging: staging + test instance, clone of staging branch from git
  • maintenance:
    • status page: http://travaux.ovh.net/
    • credentials: ssh://hello:coco@bonfire.ovh.cloud.com
• Box "namaste" @ DigitalOcean
  • exposed services:
    • port 27017: production mongodb "kinderDB" database server
    • port 27018: staging / pre-production mongodb "kinderDB" database server
  • directories:
    • /home/kinderDB/db: data directory of MongoDB database
    • /home/kinderDB/dumps: database dumps, regularly added by crontab

"Lucy is down?" Here is what to do:

Run the following procedure in order to know what has gone wrong, and how to fix it in the most efficient manner:

• Enter a (new) random query on google.com, if it does not work, check your internet connexion
• Check (again) accessing http://lucy.com (home page in HTTP mode) and refresh the page:
  • If you see an ngnix error, read the "lucyJS is down" procedure below
  • If nothing displays and the browser's loading animation keeps turning until timeout is reached...
    • Try accessing http://lucy.com/js/main.js (a static file), if not working: read the "nginx is down" procedure below
    • Check that "noms de domaines" and "serveurs dédiés" are green on that page: http://travaux.ovh.net/
    • ...or call OVH / Kimsufi, or reach them on Twitter: https://twitter.com/ovh_support_fr
  • If you're unable to connect to our server using SSH, check the "OVH is Down" section below

lucyJS is down

Re-try accessing the service after entering each line (one at a time) on the production server:

• lucyJS/start
• lucyJS/restart
• forever stop 0
• lucyJS/start

kinderdDB is down

• kinderDB/start_mongo
• kinderDB/stop_mongo

nginx is down

• Make sure that nginx is running (need to be root)
• NB: Nginx reboots automatically after a hardware reboot.

OVH is down

• Ask for a hard reset on OVH admin page:
  • Page Admin OVH -> Serveurs dédiés -> Services -> Rebooter
  • See credentials above

Monitoring logs

• /lucyJS/appd.out contains Lucy's HTTP logs (including errors), since its last start.
• /lucyJS/appd.prev contains Lucy's previous HTTP logs (including errors), since its previous start.
• /lucyJS/access.log logs HTTP requests that took more than 1 second to process (useful to troubleshoot/optimise DB queries, and app controllers in general)

TODO:

• how to start/restart production and staging servers
• process to push a new version in production
• software stack update procedures
• data backup/restore procedures
• search index (re-)population
• logs retrieval
• ...