Skip to content

Instantly share code, notes, and snippets.

@phpdave
Last active April 25, 2024 22:23
Show Gist options
  • Save phpdave/8abde81ef0ae04bf064e797303ba035d to your computer and use it in GitHub Desktop.
Save phpdave/8abde81ef0ae04bf064e797303ba035d to your computer and use it in GitHub Desktop.
How to Automate your deployments and move towards Continuous Integration and Continuous Deployment
# Deploy script for non-source tracked (sandbox) development work flow
# orgs.
#
# It will only deploy if a change from git has been made anywhere
# in the `force-app` directory.
# This must be ran from inside of an sfdx project
import argparse
from subprocess import Popen, PIPE, call, check_output
from string import Template
PROD_URL = 'https://login.salesforce.com'
TEST_URL = 'https://test.salesforce.com'
AUTH_CMD = Template('sfdx force:auth:jwt:grant -i $clientid -r $instanceurl '
'-f server.key -u $username --setdefaultusername')
GPG_DECRYPT = Template('gpg --batch --passphrase $password '
'-d config/server.key.gpg')
RM_RF_BUILD = ['rm', '-rf', 'build']
SFDX_DEPLOY = Template('sfdx force:mdapi:deploy --deploydir build '
'--testlevel RunLocalTests --wait $wait')
GIT_LOG = ['git', 'log', '--merges', '-n', '1', '--pretty=format:%P']
GIT_DIFF_TREE = Template('git diff-tree --no-commit-id --name-only -r $commit')
def main():
args = get_parser().parse_args()
if is_force_app_change():
print('force-app changes found.')
call(RM_RF_BUILD)
call(['mkdir', 'build'])
with open('server.key', 'wb') as key:
cmd = (GPG_DECRYPT
.substitute(password=args.encryptionpassword)
.split(' '))
key.write(check_output(cmd))
instanceurl = TEST_URL
if args.prod:
instanceurl = PROD_URL
# for some reason on windows it can't find
# sfdx even though its on the path. so we need
# to ask it execute via the shell
call('sfdx force:source:convert -d build', shell=True)
call(AUTH_CMD.substitute(clientid=args.clientid,
instanceurl=instanceurl,
username=args.username),
shell=True)
call(SFDX_DEPLOY.substitute(wait=args.wait), shell=True)
call(RM_RF_BUILD)
call(['rm', '-f', 'server.key'])
call('sfdx force:auth:logout --all --noprompt', shell=True)
else:
print('No force-app changes found.')
def is_force_app_change():
"""
Checks if the last git commit on the given branch modified
any files under the `force-app` directory.
"""
print('Checking merge for force-app changes.')
ret = False
for commit in get_commits_from_merge():
for file in get_changed_files_from_commit(commit):
print(file.rstrip())
if 'force-app' in file:
ret = True
return ret
def get_commits_from_merge():
"""
Gets a list of commits from the last merge
"""
commits = []
with Popen(GIT_LOG, stdout=PIPE) as proc:
while True:
line = proc.stdout.readline().decode()
if not line:
break
if line != '':
for commit in line.split(' '):
commits.append(commit)
return commits
def get_changed_files_from_commit(commit):
"""
Gets a list of files that were changed in the given commit
"""
files = []
cmd = GIT_DIFF_TREE.substitute(commit=commit).split(' ')
with Popen(cmd, stdout=PIPE) as proc:
while True:
line = proc.stdout.readline().decode()
if not line:
break
if line != '':
files.append(line)
return files
def get_parser():
"""
Gets the parser for command line arguments
"""
parser = argparse.ArgumentParser(description=('Deploys a non-source '
'tracked project'))
parser.add_argument('-c',
'--clientid',
help='connected app client id',
type=str,
required=True)
parser.add_argument('-e',
'--encryptionpassword',
help='gpg encryption passphrase for server.key.gpg',
type=str,
required=True)
parser.add_argument('--prod',
help=('sets the instance url to '
'https://login.salesforce.com'),
action='store_true')
parser.add_argument('-u',
'--username',
help='username to authenticate with',
type=str,
required=True)
parser.add_argument('-w',
'--wait',
help='wait time for sfdx force:mdapi:deploy',
type=int,
default=0)
return parser
if __name__ == '__main__':
main()

I. Prepare and Understand

Set up Public Keys between your Deployment Machine and the Remote Machines to deploy to

  1. On your local machine generate a public key that will be exchanged with the remote servers you'll be deploying to. This will authorize your computer to connect and not require a password
ssh-keygen -t rsa -b 4096 -f ~/.ssh/id_rsa_abccompany -C "Comment to remember whos key"

also setup your ~/.ssh/config with this info to tie your key to which host

Host 192.168.1.5
    Port 22
	HostName 192.168.1.5
	User myuser
	AddKeysToAgent yes
	IdentityFile ~/.ssh/id_rsa_abccompany
	IdentitiesOnly yes
  1. Copy the public key to the remote system's user directory ssh authorized keys file. Create .ssh dir if its not already created
ssh [email protected] 'mkdir .ssh'
scp "/path/to/your/key/.ssh/id_rsa.pub" [email protected]:~/.ssh/TransferedKey.pub
ssh [email protected] < SetSSHDirPermissions.sh

SetSSHDirPermissions.sh

cd ~/.ssh
cp authorized_keys authorized_keys_Backup
cat TransferedKey.pub >> authorized_keys
rm TransferedKey.pub
chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Special care is needed for symlinks and mounts inside your document root. You'll have to find our symlinks take note of these and figure out how you'll re-create them after deployment

  1. Find symlinks
ls -lR /var/www/html | grep ^l

Document root

Your site's document root will always be a symbolic link to a folder in the release directory which allows for clean switching to the new version of your site and quick rollback by switching the symlink to the previous version. Additionally it'll have a date-stamp and source control revision #

/var/www/html/site1.example.com ->  /var/www/html/releases/site1.example.com/20170315_Rev2579
/var/www/html/site2.example.com ->  /var/www/html/releases/site2.example.com/20170310_Rev2569

II. Promote Document Root/Application Code

  1. Configuration and verify connection works
Project/App/Website Name: site1.example.com
Source Control: svn.example.com or git.example.com
Source Control branch to deploy: master
Source Control revision to deploy: head
Server: MyWebServer.example.com
IP: 64.64.64.201
SSH Connection Authentication: Pub Key
Project Path:/var/www/html/site1.example.com
Release Path:/var/www/html/releases/site1.example.com/
  1. Merge development branch to Main Branch and emit post-commit hooks
git checkout master
git merge devbranch
# Trigger Build and running of tests on CI Server
  1. Run SQL promotion script

  2. Auto-Run build scripts on CI server and generate source files (minify & combine JS and CSS, etc... )

java -jar yuicompressor-x.y.z.jar jsdir/*.js -o alljs-min.js --charset utf-8
java -jar yuicompressor-x.y.z.jar cssdir/*.css -o allcss-min.css --charset utf-8
  1. Auto-Run selenium tests/phpunit tests
java -jar selenium-server.jar -htmlSuite "*firefox" "http://example.com" "testsuite\testsuite.html" "testsuite\results.html"
grep testsuite\results.html "failed"
#if "failed" found then don't deploy
  1. Deploy if everything is successful. Generate tarball (.tar.gz) of generated source files with a filename of {git revision}-{build number}.tar.gz. Transfer tarball and uncompressed to a temporary location on deployment target system. Copy from temp directory a release directory (that is not currently live) via Rsync. Change the current symbolic link to the new release. Document root of Apache is /var/www/example.com/current, which is a symbolic link to /var/www/example.com/releases/A or /var/www/example.com/releases/B. Note: this also helps PHP Opcache not have to start from scratch if you did timestamps
#package the directory you want to promote
tar -zcvf example.com.2280-1.tar.gz example.com

#Send file over to machine
scp example.com.2280-1.tar.gz [email protected]:/example.com.2280-1.tar.gz

#Extract the tarball
tar -zxvf example.com.2280-1.tar.gz -C /tmp

#Copy from temp to release directories
rsync --recursive --checksum --perms --links --delete \  /tmp/example.com.2280-1/ \  /var/www/example.com/releases/B
  1. Switch the symbolic in an atomic way
ln -s /var/www/example.com/releases/B /var/www/example.com/releases/current.new
mv -T /var/www/example.com/releases/current.new /var/www/example.com/releases/current
  1. If requested rollback A. DX make it easy to see history of deployments and which ones you can revert too rollback site1.example.com 2500 B. Switch the symlink to the previous version C. Revert any DB changes

  2. Run other tests A) Can be reached internally B) Can be reached externally etc...

III. Database Migrations (Deploying Database Changes)

  1. Manually create 2 sql files up{git revision}.sql and down{git revision}.sql
  2. Run up{git revision}.sql when promoting or down{git revision}.sql if you need to rollback
  3. Copy the sql code over
scp -r "/sqlsource/migrations" [email protected]:/sqlsource/migrations
  1. Create 2 local shell scripts that we will send over remotely to be executed

PromoteSQL.sh

system -i "RUNSQLSTM SRCSTMF('/path/to/migrations/sql/up1.sql') COMMIT(*NONE) NAMING(*SQL)"

RollbackSQL.sh

system -i "RUNSQLSTM SRCSTMF('/path/to/migrations/sql/down1.sql') COMMIT(*NONE) NAMING(*SQL)"
  1. Promote the SQL by sending the PromoteSQL.sh script to the remote system to be ran.
ssh [email protected] < PromoteSQL.sh
  1. (If needed) Rollback the SQL by sending the RollbackSQL.sh script to the remote system to be ran.
ssh [email protected] < RollbackSQL.sh

Alternataive methods to deploy SQL

  1. PHP file with IBM_DB2 connection, PDO or ODBC connection and Run up{git revision}.sql through that connection
  2. JDBC connection w/ JTOpen/JT400 driver through JDBC and Run up{git revision}.sql through that connection

IV. Post Install clean-up

  1. Clear unused files, db objects

Optionals

  1. Tag a git version to signify when you went live with a release
  2. Verify the deployment server meets the application requirements (pre-deployment)
  3. Update your vendor folder (composer install --no-dev --optimize-autoloader)
  4. Add/edit cronjobs
  5. Deploy/Verify external files/services are working (CDNs)
  6. emit webhooks events for the different phases of deployment

#Settings

  1. Enable mod_realdoc so that when you switch deployments old requests continue to use the old release.
<IfModule mod_realdoc.c>
    RealpathEvery 2
</IfModule>

if you configure Apache’s DOCUMENT_ROOT for your site to be /var/www/site/htdocs and then you have /var/www/site be a symlink to alternatingly /var/www/A and /var/www/B you need to check your code for any hardcoded instances of /var/www/site/htdocs. This includes your PHP include_path setting. One way of doing this is to set your include_path as the very first thing you do if you have a front controller in your application. You can use something like this:

ini_set('include_path', $_SERVER['DOCUMENT_ROOT'].'/../include');

That means once mod_realdoc has resolved /var/www/site/htdocs to /var/www/A/htdocs your include_path will be /var/www/A/htdocs/../include for the remainder of this PHP request and even if the symlink is switched to /var/www/B halfway through the request it won’t be visible to this request.

References

  1. https://en.wikipedia.org/wiki/Continuous_delivery
  2. https://medium.com/salesforce-engineering/tarball-is-not-a-dirty-word-3bd0af99e8fd#.8roru9d3n
  3. https://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/
  4. https://docs.jelastic.com/php-zero-downtime-deploy
  5. https://github.com/etsy/mod_realdoc
  6. https://github.com/etsy/incpath
  7. http://stackoverflow.com/questions/305035/how-to-use-ssh-to-run-shell-script-on-a-remote-machine

CD Diagram

CD Diagram

  1. Consider using https://www.sitepoint.com/phpseclib-securely-communicating-with-remote-servers-via-php/ this to execute the commands
  2. Consider using: A: Raw Shell scripts B: https://www.phing.info/ https://github.com/phpdave/phing-tutorial C: https://www.ansible.com/ D: https://deployer.org/ E: https://envoyer.io/
  3. Automate updating depencies using A: PHP: Composer B: Javascript: Bower,Grunt, Gulp

ANT or PHING Dependency Managers - Composer or NPM JS Build tool - Bower,Grunt, Gulp - Choose one

Test Suite - #1 Build Tools Scripted Deployment

CI - Continous Integration Automated tests - regression detection Every day at 3 day they delivered

CD - Continous Delivery CI plus Continous Deployment Release to Prod is automatic

Continuous Delivery

  • CI - Continous Integration
    • Automated tests - regression detection
    • Every day at 3pm code is checked in (geared for larger shops)
  • CD - Continous Delivery
    • CI plus Continous Deployment
    • Promotion of code to QA and Prod is automatic via scripted deployment.
    • DDL code must be manually scripted for deployment and for rolling back.
  • Tools for CI/CD
    • PHP Dependency Managers - Composer
    • JS Dependency management - Bower
    • Build tool - Grunt, Gulp - Choose one
    • Testing tools - PHPUnit, Selenium
    • ANT Build tool to send a command to server (restart mysql)
  • DDL promotion
    • Create a copy of Table1 to a new Table2
    • Create a way to get the delta between the two
    • Point from Table1 to Table2 and discontinue use of Table1
  • Code Promotion
    • Create 3 directories for code,
    • Create 3 symbolic links: current, archive1, archive2
    • Go live is just a matter of switching the symbolic link between the 3 directories. You keep the archives in case you need to revert back.
  • Source Control
    • Use different branches for each project
    • Merge back into Master branch when ready to deploy code.
#!/bin/bash
#Checkout the file changes between revisions
#checkout.sh 200 201 ./changes 201
#checkout.sh 200 201 ./backup 20
#Get the files that have changed between REVISION_START and REVISION_STOP and deploy it in the folder DESTINATION_PATH and pull back only things that match the SEARCH_PATTERN
REVISION_START=${1}
REVISION_STOP=${2}
DESTINATION_PATH="${3}"
REVISION_TO_EXPORT="${4}"
SEARCH_PATTERN="projectname in path"
#Your svn repository url
REPOSITORY="svn://svn.example.com"
mkdir -p /Deployment
cd /Deployment
svn co --depth empty $REPOSITORY
FILES=$(svn diff --summarize -r$REVISION_START:$REVISION_STOP | grep $SEARCH_PATTERN)
mkdir -p $DESTINATION_PATH
for FILE in $FILES ; do
if [[ $FILE == *"$SEARCH_PATTERN"* ]]
then
#Convert backslashes to forward slashes on Windows
FILE="${FILE//\\//}"
FOLDER=$(dirname $FILE)
DIR=$DESTINATION_PATH/$FOLDER
mkdir -p ${DIR}
svn export --force -r $REVISION_TO_EXPORT $REPOSITORY/$FILE $DESTINATION_PATH/$FOLDER
fi
done
#!/bin/sh
#Bundle it, send it, extract it, Copy it view differences
tar -zcvf app1.example.com.1285.tar.gz app1.example.com
scp app1.example.com.1285.tar.gz [email protected]:/app1.example.com.1285.tar.gz
tar -zxvf app1.example.com.1285.tar.gz -C /tmp
cd /tmp
rsync -avun /tmp/app1.example.com/ /var/www/html/app1.example.com/
diff --brief -r /tmp/app1.example.com/ /var/www/html/app1.example.com/
#!/bin/bash
#ZZZPromoteCodeAndDB.sh dev 1205 1206
ENVIRONMENT_TYPE=${1}
REVISION_START=${2}
REVISION_STOP=${3}
#make sure we have all parameters
if [ "${1}" = "" ] || [ "${2}" = "" ] || [ "${3}" = "" ]
then
echo "Usage: promote.sh [dev|qa|prod] [starting-src-revision#] [ending-src-revision#]"
exit
fi
read -p "Do you want to checkout the changes between ${2}-${3} ? (y/n)?" CONT
echo # (optional) move to a new line
if [ "$CONT" = "y" ];
then
# Get changes and backup
checkout.sh $REVISION_START $REVISION_STOP ./${ENVIRONMENT_TYPE}/changes $REVISION_STOP
checkout.sh $REVISION_START $REVISION_STOP ./${ENVIRONMENT_TYPE}/backup $REVISION_START
fi
read -p "Do you want to upload the code thats been checked out to the ${ENVIRONMENT_TYPE} environment? (y/n)?" CONT
echo
if [ "$CONT" = "y" ];
then
if [ "${ENVIRONMENT_TYPE}" = "dev" ]
then
echo "Deploying to dev"
# Deploy changes to two servers
scp -r "/Deployment/${ENVIRONMENT_TYPE}/changes/app1.example.com" myuser@server${ENVIRONMENT_TYPE}.example.com:/var/www/html/app1.example.com
scp -r "/Deployment/${ENVIRONMENT_TYPE}/changes/app2.example.com" myuser@as400${ENVIRONMENT_TYPE}.example.com:/www/zendsvr/app2.example.com
# Deploy SQL Changes
scp -r "/Deployment/Migrations" [email protected]:/home/myuser/Migrations
ssh [email protected] < ZZZRunSQLOnRemoteIBMi.sh
elif [ "${ENVIRONMENT_TYPE}" = "qc" ]
then
echo "Deploying to Quality Control (qc)"
# Deploy changes to two servers
scp -r "/Deployment/${ENVIRONMENT_TYPE}/changes/app1.example.com" myuser@server${ENVIRONMENT_TYPE}.example.com:/var/www/html/app1.example.com
scp -r "/Deployment/${ENVIRONMENT_TYPE}/changes/app2.example.com" myuser@as400${ENVIRONMENT_TYPE}.example.com:/www/zendsvr/app2.example.com
# Deploy SQL Changes
scp -r "/Deployment/Migrations" [email protected]:/home/myuser/Migrations
ssh [email protected] < ZZZRunSQLOnRemoteIBMi.sh
elif [ "${ENVIRONMENT_TYPE}" = "production" ]
then
read -p "Are you sure you want to deploy to Revision ${REVISION_STOP} to ${ENVIRONMENT_TYPE}? " -n 1 -r
echo # (optional) move to a new line
if [[ $REPLY =~ ^[Yy]$ ]]
then
echo "Deploying to Production (prod)"
# Deploy changes to two servers
scp -r "/Deployment/${ENVIRONMENT_TYPE}/changes/app1.example.com" myuser@server${ENVIRONMENT_TYPE}.example.com:/var/www/html/app1.example.com
scp -r "/Deployment/${ENVIRONMENT_TYPE}/changes/app2.example.com" myuser@as400${ENVIRONMENT_TYPE}.example.com:/www/zendsvr/app2.example.com
# Deploy SQL Changes
scp -r "/Deployment/Migrations" [email protected]:/home/myuser/Migrations
ssh [email protected] < ZZZRunSQLOnRemoteIBMi.sh
fi
else
echo "Must pass parameter 1 (dev,qc,production) 2 Current checkout version 3 version to promote to (ZZZPromoteCodeAndDB.sh qc 201 205)"
fi
fi
system -i "RUNSQLSTM SRCSTMF('/home/MYUSER/1up.sql') COMMIT(*NONE) NAMING(*SQL)"
<?php
$outout = shell_exec( 'mydeploy_fromgithub.sh' );
error_log($output);
#!/usr/bin/bash
export PATH=/QOpenSys/usr/bin:$PATH
export HOME=/home/QTMHHTTP
cd /www/zendphp7/htdocs
git reset --hard HEAD
eval `ssh-agent`
ssh-add /home/QTMHHTTP/.ssh/id_rsa
ssh-add -l
git stash -u
git pull
/usr/local/phpci/composer.phar self-update
php /usr/local/phpci/composer.phar install --no-dev --no-interaction --no-progress
kill $SSH_AGENT_PID
unset SSH_AGENT_PID
unset SSH_AUTH_SOCK
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment