Skip to content

Instantly share code, notes, and snippets.

@exequielrafaela

exequielrafaela/DEVELOPMENT_WORKFLOW_A.md

Last active June 28, 2019 22:52
Show Gist options
  • Save exequielrafaela/908748a56569358891a8bf3a9e6e010a to your computer and use it in GitHub Desktop.
Save exequielrafaela/908748a56569358891a8bf3a9e6e010a to your computer and use it in GitHub Desktop.
Download ZIP
CI/CD Development Workflow w/ Spinnaker + Jenkins
Raw
DEVELOPMENT_WORKFLOW_A.md

Workflow implementation example A

Jenkins as 1ry executor with dev/stg/prod docker image tags prefix

DEV

  1. Scm-github / scm-bitbucket: git push / PR to develop branch.
  2. Jenkins webhook automatically/manual triggers job which will build docker image w/ unit tests (optional can be in a separate pre-dependency Jenkins / CircleCI job) adding dev-build# tag -> If OK.
  3. Push docker image to AWS ECR.
  4. New docker image with tag dev-* detected by Spinnaker hook that auto triggers a deployment pipeline with DB migrations + ENV vars secret mgmt (Hashicorp Vault or AWS SSM Parameter Store).
  5. K8s (Dev Env deployed).
  6. Notify success (Slack integration).

STG

  1. Scm-github / scm-bitbucket: git push / PR to master branch.
  2. Jenkins webhook automatically triggers job (or manual trigger) which will re-build docker image w/ unit tests (optional can be in a separate pre-dependency Jenkins / CircleCI job) with stg-build# tag (rebuild only if specifically needed - probably not) just directly promote already existing dev-build# docker image by adding stg-build# tag (more efficient and consistent cross env).
  3. Push docker image to AWS ECR.
  4. New docker image with tag stg-* detected by Spinnaker hook that auto triggers a deployment pipeline with DB migrations + ENV vars secret mgmt (Hashicorp Vault or AWS SSM Parameter Store).
  5. Integration Testing automatic execution.
  6. K8s (Stage Env deployed)
  7. Notify success (Slack integration).

PROD

  1. Jenkins manual job trigger to promote docker image tagged as stg-build# by adding prod-build# tag.
  2. Push docker image to AWS ECR.
  3. New docker image with tag prod-* detected by Spinnaker hook that auto triggers a deployment pipeline with DB migrations + ENV vars secret mgmt (Hashicorp Vault or AWS SSM Parameter Store).
  4. K8s (Prod Env deployed)
  5. Notify success (Slack integration).
Raw
DEVELOPMENT_WORKFLOW_B.md

Workflow implementation example B

Spinnaker as 1ry Executor with dev / release docker image tags prefix and auto git tag release

DEV

  1. Scm-github / scm-bitbucket: git push / PR to develop branch.
  2. Spinnaker webhook automatically triggers pipeline job (or manual trigger).
  3. Call Jenkins (integration) which will build the docker image w/ unit test (optional can be in a separate pre-dependency Jenkins / CircleCI job) adding dev-build#-commit# image tag -> If OK.
  4. Push docker image to AWS ECR.
  5. Ongoing Spinnaker deployment pipeline will use docker image tagged with dev-build#-commit# with DB migrations + ENV vars secret mgmt (Hashicorp Vault or AWS SSM Parameter Store).
  6. K8s (Dev Env deployed).
  7. Notify success (Slack integration).

STG

  1. Scm-github / scm-bitbucket: git push / PR to master branch.
  2. Spinnaker webhook automatically triggers pipeline job (or manual trigger).
  3. Call Jenkins (integration) which will build docker image w/ unit test (optional can be in a separate pre-dependency Jenkins / CircleCI job) with release-build#-commit# docker image tag (rebuild only if specifically needed - depends on the development team needs) / or just directly promote dev-build#-commit# adding new docker image tag release-build#-commit# (more efficient and consistent cross env).
  4. Push docker image to AWS ECR.
  5. Ongoing Spinnaker deployment pipeline will use docker image tagged with release-build#-commit# with DB migrations + ENV vars secret mgmt (Hashicorp Vault or AWS SSM Parameter Store).
  6. Integration Testing automatic execution.
  7. K8s (Stage Env deployed).
  8. Notify success (Slack integration).

PROD

  1. Spinnaker pipeline job manual trigger.
  2. Call Jenkins (integration) to create new git prod stable release tag.
$ git tag -a release_2019-06-19-09-32-29 24b93f5 -m 'User (Sun Jun 16 10:58:11 2019): [JIRA-3531] coming soon mvp (#518) 24b93f5'`
$ git push --tags (to [email protected]:Organization/repo.git)
$* [new tag] release_2019-06-19-09-32-29
  1. Ongoing Spinnaker deployment pipeline will use the Stage env tested docker image tagged as release-build#-commit# with DB migrations + ENV vars secret mgmt (Hashicorp Vault or AWS SSM Parameter Store).
  2. K8s (Prod Env deployed).
  3. Notify success (Slack integration).
Raw
spk-demo-app-py-django-beat-pipeline-metadata-dev.json
{
"keepWaitingPipelines" : false,
"lastModifiedBy" : "demo-user",
"roles" : [ "devops" ],
"index" : 0,
"serviceAccount" : "dev-|-demo-web-backend-|-deploy@managed-service-account",
"triggers" : [ {
"branch" : "develop",
"enabled" : true,
"project" : "demo-project",
"runAsUser" : "project-svc-account",
"secret" : "xxxxxxxxxxxxxxxx",
"slug" : "demo-web-backend",
"source" : "github",
"type" : "git"
} ],
"limitConcurrent" : true,
"application" : "demo-web",
"name" : "DEV | DemoWeb BackEnd | Deploy",
"stages" : [ {
"continuePipeline" : false,
"failPipeline" : true,
"job" : "app-demo-dev-build-image",
"master" : "aws-jenkins",
"name" : "Build Image",
"parameters" : { },
"propertyFile" : "build.properties",
"refId" : "1",
"requisiteStageRefIds" : [ ],
"type" : "jenkins"
}, {
"continuePipeline" : false,
"failPipeline" : true,
"job" : "app-demo-dev-refresh-secrets",
"master" : "aws-jenkins",
"name" : "Refresh Secrets",
"parameters" : {
"envName" : "dev",
"k8sCluster" : "cluster-1.k8s.devstg.demo-project.aws"
},
"refId" : "2",
"requisiteStageRefIds" : [ "1" ],
"type" : "jenkins"
}, {
"clusters" : [ {
"account" : "project-k8s-devstg",
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ ],
"command" : [ ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-dev-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}",
"uri" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : {
"cpu" : "300m",
"memory" : "400Mi"
},
"name" : "demo-web-backend",
"ports" : [ {
"containerPort" : 80,
"name" : "http",
"protocol" : "TCP"
} ],
"requests" : {
"cpu" : "150m",
"memory" : "300Mi"
},
"volumeMounts" : [ ]
} ],
"delayBeforeDisableSec" : 3,
"delayBeforeScaleDownSec" : 3,
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "api",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ "demo-web-dev-api-lb" ],
"maxRemainingAsgs" : 2,
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-web",
"replicaSetAnnotations" : { },
"scaleDown" : true,
"securityGroups" : [ ],
"stack" : "dev",
"strategy" : "redblack",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy API",
"overrideTimeout" : true,
"refId" : "3",
"requisiteStageRefIds" : [ "5" ],
"stageTimeoutMs" : 480000,
"type" : "deploy"
}, {
"clusters" : [ {
"account" : "project-k8s-devstg",
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ "-m", "celery", "worker", "-A", "demo-project", "-l", "info" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-dev-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}",
"uri" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : {
"cpu" : "150m",
"memory" : "350Mi"
},
"livenessProbe" : {
"failureThreshold" : 2,
"handler" : {
"execAction" : {
"commands" : [ "python", "kubernetes/scripts/liveness/logs-timestamp-check.py", "\"-s 300\"" ]
},
"httpGetAction" : {
"path" : "/",
"port" : 80,
"uriScheme" : "HTTP"
},
"tcpSocketAction" : {
"port" : 80
},
"type" : "EXEC"
},
"initialDelaySeconds" : 120,
"periodSeconds" : 30,
"successThreshold" : 1,
"timeoutSeconds" : 10
},
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : {
"cpu" : "50m",
"memory" : "250Mi"
},
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "worker",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ ],
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-web",
"replicaSetAnnotations" : { },
"securityGroups" : [ ],
"stack" : "dev",
"strategy" : "",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy New Worker",
"refId" : "4",
"requisiteStageRefIds" : [ "11" ],
"type" : "deploy"
}, {
"account" : "project-k8s-devstg",
"annotations" : { },
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"containers" : [ {
"args" : [ "manage.py", "migrate" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-dev-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"account" : "project-sr-ecr",
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : { },
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : { },
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"labels" : { },
"name" : "Run Migrations",
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"refId" : "5",
"requisiteStageRefIds" : [ "7" ],
"type" : "runJob"
}, {
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-web-dev-worker",
"completeOtherBranchesThenFail" : false,
"continuePipeline" : true,
"credentials" : "project-k8s-devstg",
"failPipeline" : false,
"interestingHealthProviderNames" : [ "KubernetesService" ],
"name" : "Destroy Prev Worker",
"namespaces" : [ "demo-web" ],
"refId" : "6",
"requisiteStageRefIds" : [ "5" ],
"target" : "current_asg_dynamic",
"type" : "destroyServerGroup"
}, {
"account" : "project-k8s-devstg",
"annotations" : { },
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"containers" : [ {
"args" : [ "manage.py", "collectstatic", "--noinput", "-v", "1" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-dev-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"account" : "project-sr-ecr",
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : { },
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : { },
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"labels" : { },
"name" : "Upload Static Assets",
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"refId" : "7",
"requisiteStageRefIds" : [ "2" ],
"type" : "runJob"
}, {
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-web-dev-beat",
"completeOtherBranchesThenFail" : false,
"continuePipeline" : true,
"credentials" : "project-k8s-devstg",
"failPipeline" : false,
"interestingHealthProviderNames" : [ "KubernetesService" ],
"name" : "Destroy Prev Beat",
"namespaces" : [ "demo-web" ],
"refId" : "8",
"requisiteStageRefIds" : [ "5" ],
"target" : "current_asg_dynamic",
"type" : "destroyServerGroup"
}, {
"clusters" : [ {
"account" : "project-k8s-devstg",
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ "-m", "celery", "beat", "-A", "demo-project", "-l", "info" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-dev-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}",
"uri" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : {
"cpu" : "100m",
"memory" : "300Mi"
},
"livenessProbe" : {
"failureThreshold" : 2,
"handler" : {
"execAction" : {
"commands" : [ "python", "kubernetes/scripts/liveness/logs-timestamp-check.py", "\"-s 600\"" ]
},
"httpGetAction" : {
"path" : "/",
"port" : 80,
"uriScheme" : "HTTP"
},
"tcpSocketAction" : {
"port" : 80
},
"type" : "EXEC"
},
"initialDelaySeconds" : 120,
"periodSeconds" : 30,
"successThreshold" : 1,
"timeoutSeconds" : 10
},
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : {
"cpu" : "50m",
"memory" : "200Mi"
},
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "beat",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ ],
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-web",
"replicaSetAnnotations" : { },
"securityGroups" : [ ],
"stack" : "dev",
"strategy" : "",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy New Beat",
"refId" : "9",
"requisiteStageRefIds" : [ "10" ],
"type" : "deploy"
}, {
"name" : "Wait",
"refId" : "10",
"requisiteStageRefIds" : [ "8" ],
"type" : "wait",
"waitTime" : 15
}, {
"name" : "Wait",
"refId" : "11",
"requisiteStageRefIds" : [ "6" ],
"type" : "wait",
"waitTime" : 15
}, {
"completeOtherBranchesThenFail" : false,
"continuePipeline" : true,
"failPipeline" : false,
"job" : "app-demo-dev-send-notifications",
"markUnstableAsSuccessful" : false,
"master" : "aws-jenkins",
"name" : "Notify",
"parameters" : {
"appEnv" : "dev",
"appName" : "app-demo",
"appRepo" : "[email protected]:organization/demo-web-backend.git",
"branchName" : "develop",
"currentCommitId" : "${CURRENT_COMMIT_ID}",
"notificationsChannel" : "product-release-be",
"previousCommitId" : "${PREVIOUS_COMMIT_ID}"
},
"refId" : "12",
"requisiteStageRefIds" : [ "3" ],
"type" : "jenkins"
} ],
"id" : "48b94a82-07be-4ee8-b0b7-1111111111111",
"updateTs" : "1559156385000",
"notifications" : [ {
"address" : "engineering-alerts",
"level" : "pipeline",
"type" : "slack",
"when" : [ "pipeline.starting", "pipeline.complete", "pipeline.failed" ]
} ]
}
Raw
spk-demo-app-py-django-beat-pipeline-metadata-prd.json
{
"keepWaitingPipelines": false,
"lastModifiedBy" : "demo-user",
"roles" : [ "devops" ],
"limitConcurrent": true,
"notifications": [
{
"address": "engineering-alerts",
"level": "pipeline",
"type": "slack",
"when": [
"pipeline.starting",
"pipeline.complete",
"pipeline.failed"
]
}
],
"parameterConfig": [
{
"description": "Image tag to use for this release",
"label": "Image Tag",
"name": "imageTag",
"required": true
},
{
"default": "Yes",
"description": "Whether to deploy the worker or not",
"hasOptions": true,
"label": "Deploy Worker",
"name": "deployWorker",
"options": [
{
"value": "Yes"
},
{
"value": "No"
}
],
"required": true
}
],
"stages": [
{
"continuePipeline": false,
"failPipeline": true,
"job": "demo-app-rel-tag-release",
"master": "aws-jenkins",
"name": "Tag Release",
"parameters": {
"imageTag": "${parameters.imageTag}"
},
"propertyFile": "build.properties",
"refId": "1",
"requisiteStageRefIds": [],
"type": "jenkins"
},
{
"continuePipeline": false,
"failPipeline": true,
"job": "demo-app-prd-refresh-secrets",
"master": "aws-jenkins",
"name": "Update Secrets",
"parameters": {
"envName": "prd",
"k8sCluster": "cluster-1.k8s.prd.demo-project.aws"
},
"refId": "6",
"requisiteStageRefIds": [
"1"
],
"type": "jenkins"
},
{
"account": "demo-project-k8s-prd",
"annotations": {},
"application": "demo-app",
"cloudProvider": "kubernetes",
"cloudProviderType": "kubernetes",
"containers": [
{
"args": [
"manage.py",
"migrate"
],
"command": [
"python"
],
"envFrom": [
{
"prefix": "",
"secretRef": {
"name": "demo-app-prd-config",
"optional": false
}
}
],
"envVars": [],
"imageDescription": {
"account": "demo-project-sr-ecr",
"imageId": "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-app-backend:${TAG}",
"registry": "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository": "demo-app-backend",
"tag": "${TAG}"
},
"imagePullPolicy": "IFNOTPRESENT",
"limits": {},
"name": "demo-app-backend",
"ports": [],
"requests": {},
"volumeMounts": []
}
],
"dnsPolicy": "ClusterFirst",
"labels": {},
"name": "Run Migrations",
"namespace": "demo-app",
"nodeSelector": {
"kops.k8s.io/instancegroup": "nodes"
},
"refId": "7",
"requisiteStageRefIds": [
"12"
],
"type": "runJob"
},
{
"clusters": [
{
"account": "demo-project-k8s-prd",
"application": "demo-app",
"cloudProvider": "kubernetes",
"containers": [
{
"args": [],
"command": [],
"envFrom": [
{
"prefix": "",
"secretRef": {
"name": "demo-app-prd-config",
"optional": false
}
}
],
"envVars": [],
"imageDescription": {
"account": "demo-project-sr-ecr",
"imageId": "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-app-backend:${TAG}",
"registry": "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository": "demo-app-backend",
"tag": "${TAG}"
},
"imagePullPolicy": "IFNOTPRESENT",
"limits": {
"cpu": "750m",
"memory": "750Mi"
},
"name": "demo-app-backend",
"ports": [
{
"containerPort": 80,
"name": "http",
"protocol": "TCP"
}
],
"requests": {
"cpu": "150m",
"memory": "300Mi"
},
"volumeMounts": []
}
],
"delayBeforeDisableSec": 3,
"delayBeforeScaleDownSec": 3,
"deployment": {
"deploymentStrategy": {
"rollingUpdate": {
"maxSurge": 1,
"maxUnavailable": 1
},
"type": "RollingUpdate"
},
"enabled": false,
"minReadySeconds": 0
},
"dnsPolicy": "ClusterFirst",
"freeFormDetails": "api",
"initContainers": [],
"interestingHealthProviderNames": [
"KubernetesContainer",
"KubernetesPod"
],
"loadBalancers": [
"demo-app-prd-api-lb"
],
"maxRemainingAsgs": 2,
"namespace": "demo-app",
"nodeSelector": {
"kops.k8s.io/instancegroup": "nodes"
},
"podAnnotations": {},
"provider": "kubernetes",
"region": "demo-app",
"replicaSetAnnotations": {},
"scaleDown": true,
"stack": "prd",
"strategy": "redblack",
"targetSize": 3,
"terminationGracePeriodSeconds": 30,
"useSourceCapacity": false,
"volumeSources": []
}
],
"name": "Deploy API",
"overrideTimeout": true,
"refId": "8",
"requisiteStageRefIds": [
"7"
],
"stageTimeoutMs": 480000,
"type": "deploy"
},
{
"cloudProvider": "kubernetes",
"cloudProviderType": "kubernetes",
"cluster": "demo-app-prd-worker",
"completeOtherBranchesThenFail": false,
"continuePipeline": true,
"credentials": "demo-project-k8s-prd",
"failPipeline": false,
"interestingHealthProviderNames": [
"KubernetesService"
],
"name": "Destroy Prev Worker",
"namespaces": [
"demo-app"
],
"refId": "9",
"requisiteStageRefIds": [
"7"
],
"stageEnabled": {
"expression": "${parameters.deployWorker == 'Yes'}",
"type": "expression"
},
"target": "current_asg_dynamic",
"type": "destroyServerGroup"
},
{
"clusters": [
{
"account": "demo-project-k8s-prd",
"application": "demo-app",
"cloudProvider": "kubernetes",
"containers": [
{
"args": [
"-m",
"celery",
"worker",
"-A",
"demo-project",
"-l",
"info"
],
"command": [
"python"
],
"envFrom": [
{
"prefix": "",
"secretRef": {
"name": "demo-app-prd-config",
"optional": false
}
}
],
"envVars": [],
"imageDescription": {
"account": "demo-project-sr-ecr",
"imageId": "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-app-backend:${TAG}",
"registry": "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository": "demo-app-backend",
"tag": "${TAG}"
},
"imagePullPolicy": "IFNOTPRESENT",
"limits": {
"cpu": "500m",
"memory": "700Mi"
},
"name": "demo-app-backend",
"ports": [],
"requests": {
"cpu": "100m",
"memory": "300Mi"
},
"volumeMounts": []
}
],
"deployment": {
"deploymentStrategy": {
"rollingUpdate": {
"maxSurge": 1,
"maxUnavailable": 1
},
"type": "RollingUpdate"
},
"enabled": false,
"minReadySeconds": 0
},
"dnsPolicy": "ClusterFirst",
"freeFormDetails": "worker",
"initContainers": [],
"interestingHealthProviderNames": [
"KubernetesContainer",
"KubernetesPod"
],
"namespace": "demo-app",
"nodeSelector": {
"kops.k8s.io/instancegroup": "nodes"
},
"podAnnotations": {},
"provider": "kubernetes",
"region": "demo-app",
"replicaSetAnnotations": {},
"stack": "prd",
"strategy": "",
"targetSize": 1,
"terminationGracePeriodSeconds": 30,
"useSourceCapacity": false,
"volumeSources": []
}
],
"name": "Deploy New Worker",
"refId": "10",
"requisiteStageRefIds": [
"9"
],
"stageEnabled": {
"expression": "${parameters.deployWorker == 'Yes'}",
"type": "expression"
},
"type": "deploy"
},
{
"account": "demo-project-k8s-prd",
"annotations": {},
"application": "demo-app",
"cloudProvider": "kubernetes",
"cloudProviderType": "kubernetes",
"containers": [
{
"args": [
"manage.py",
"collectstatic",
"--noinput",
"-v",
"0"
],
"command": [
"python"
],
"envFrom": [
{
"prefix": "",
"secretRef": {
"name": "demo-app-prd-config",
"optional": false
}
}
],
"envVars": [],
"imageDescription": {
"account": "demo-project-sr-ecr",
"imageId": "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-app-backend:${TAG}",
"registry": "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository": "demo-app-backend",
"tag": "${TAG}"
},
"imagePullPolicy": "IFNOTPRESENT",
"limits": {},
"name": "demo-app-backend",
"ports": [],
"requests": {},
"volumeMounts": []
}
],
"dnsPolicy": "ClusterFirst",
"labels": {},
"name": "Upload Static Assets",
"namespace": "demo-app",
"nodeSelector": {
"kops.k8s.io/instancegroup": "nodes"
},
"refId": "12",
"requisiteStageRefIds": [
"6"
],
"type": "runJob"
},
{
"cloudProvider": "kubernetes",
"cloudProviderType": "kubernetes",
"cluster": "demo-app-prd-beat",
"completeOtherBranchesThenFail": false,
"continuePipeline": true,
"credentials": "demo-project-k8s-prd",
"failPipeline": false,
"interestingHealthProviderNames": [
"KubernetesService"
],
"name": "Destroy Prev Beat",
"namespaces": [
"demo-app"
],
"refId": "13",
"requisiteStageRefIds": [
"7"
],
"target": "current_asg_dynamic",
"type": "destroyServerGroup"
},
{
"clusters": [
{
"account": "demo-project-k8s-prd",
"application": "demo-app",
"cloudProvider": "kubernetes",
"containers": [
{
"args": [
"-m",
"celery",
"beat",
"-A",
"demo-project",
"-l",
"info"
],
"command": [
"python"
],
"envFrom": [
{
"prefix": "",
"secretRef": {
"name": "demo-app-prd-config",
"optional": false
}
}
],
"envVars": [],
"imageDescription": {
"account": "demo-project-sr-ecr",
"imageId": "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-app-backend:${TAG}",
"registry": "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository": "demo-app-backend",
"tag": "${TAG}"
},
"imagePullPolicy": "IFNOTPRESENT",
"limits": {
"cpu": "500m",
"memory": "700Mi"
},
"name": "demo-app-backend",
"ports": [],
"requests": {
"cpu": "100m",
"memory": "300Mi"
},
"volumeMounts": []
}
],
"deployment": {
"deploymentStrategy": {
"rollingUpdate": {
"maxSurge": 1,
"maxUnavailable": 1
},
"type": "RollingUpdate"
},
"enabled": false,
"minReadySeconds": 0
},
"dnsPolicy": "ClusterFirst",
"freeFormDetails": "beat",
"initContainers": [],
"interestingHealthProviderNames": [
"KubernetesContainer",
"KubernetesPod"
],
"namespace": "demo-app",
"nodeSelector": {
"kops.k8s.io/instancegroup": "nodes"
},
"podAnnotations": {},
"provider": "kubernetes",
"region": "demo-app",
"replicaSetAnnotations": {},
"stack": "prd",
"strategy": "",
"targetSize": 1,
"terminationGracePeriodSeconds": 30,
"useSourceCapacity": false,
"volumeSources": []
}
],
"name": "Deploy New Beat",
"refId": "14",
"requisiteStageRefIds": [
"13"
],
"stageEnabled": {
"expression": "${parameters.deployWorker == 'Yes'}",
"type": "expression"
},
"type": "deploy"
},
{
"completeOtherBranchesThenFail": false,
"continuePipeline": true,
"failPipeline": false,
"job": "demo-app-prd-send-notifications",
"master": "aws-jenkins",
"name": "Notify",
"parameters": {
"appEnv": "prd",
"appName": "demo-app",
"appRepo": "[email protected]:Life-House/native-web-backend.git",
"branchName": "master",
"currentCommitId": "${CURRENT_COMMIT_ID}",
"notificationsChannel": "product-release-be",
"previousCommitId": "${PREVIOUS_COMMIT_ID}"
},
"refId": "15",
"requisiteStageRefIds": [
"8"
],
"type": "jenkins"
}
],
"triggers": [],
"updateTs": "1559160740000"
}
Raw
spk-demo-app-py-django-beat-pipeline-metadata-stg.json
{
"keepWaitingPipelines" : false,
"lastModifiedBy" : "demo-user",
"roles" : [ "devops" ],
"index" : 2,
"serviceAccount" : "stg-|-demo-web-backend-|-deploy@managed-service-account",
"triggers" : [ {
"branch" : "master",
"enabled" : true,
"project" : "demo-project",
"runAsUser" : "project-svc-account",
"secret" : "xxxxxxxxxxxxxxxx",
"slug" : "demo-web-backend",
"source" : "github",
"type" : "git"
} ],
"limitConcurrent" : true,
"application" : "demo-web",
"name" : "STG | DemoWeb BackEnd | Deploy",
"stages" : [ {
"continuePipeline" : false,
"failPipeline" : true,
"job" : "nwbe-rel-build-image",
"master" : "aws-jenkins",
"name" : "Build Image",
"parameters" : { },
"propertyFile" : "build.properties",
"refId" : "1",
"requisiteStageRefIds" : [ ],
"type" : "jenkins"
}, {
"continuePipeline" : false,
"failPipeline" : true,
"job" : "nwbe-stg-refresh-secrets",
"master" : "aws-jenkins",
"name" : "Refresh Secrets",
"parameters" : {
"envName" : "stg",
"k8sCluster" : "cluster-1.k8s.devstg.demo-project.aws"
},
"refId" : "2",
"requisiteStageRefIds" : [ "1" ],
"type" : "jenkins"
}, {
"clusters" : [ {
"account" : "project-k8s-devstg",
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ ],
"command" : [ ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-stg-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}",
"uri" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : {
"cpu" : "300m",
"memory" : "400Mi"
},
"name" : "demo-web-backend",
"ports" : [ {
"containerPort" : 80,
"name" : "http",
"protocol" : "TCP"
} ],
"requests" : {
"cpu" : "150m",
"memory" : "300Mi"
},
"volumeMounts" : [ ]
} ],
"delayBeforeDisableSec" : 3,
"delayBeforeScaleDownSec" : 3,
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "api",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ "demo-web-stg-api-lb" ],
"maxRemainingAsgs" : 2,
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-web",
"replicaSetAnnotations" : { },
"scaleDown" : true,
"securityGroups" : [ ],
"stack" : "stg",
"strategy" : "redblack",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy API",
"overrideTimeout" : true,
"refId" : "3",
"requisiteStageRefIds" : [ "5" ],
"stageTimeoutMs" : 480000,
"type" : "deploy"
}, {
"clusters" : [ {
"account" : "project-k8s-devstg",
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ "-m", "celery", "worker", "-A", "demo-project", "-l", "info" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-stg-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}",
"uri" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : {
"cpu" : "150m",
"memory" : "350Mi"
},
"livenessProbe" : {
"failureThreshold" : 2,
"handler" : {
"execAction" : {
"commands" : [ "python", "kubernetes/scripts/liveness/logs-timestamp-check.py", "\"-s 300\"" ]
},
"httpGetAction" : {
"path" : "/",
"port" : 80,
"uriScheme" : "HTTP"
},
"tcpSocketAction" : {
"port" : 80
},
"type" : "EXEC"
},
"initialDelaySeconds" : 120,
"periodSeconds" : 30,
"successThreshold" : 1,
"timeoutSeconds" : 10
},
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : {
"cpu" : "50m",
"memory" : "250Mi"
},
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "worker",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ ],
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-web",
"replicaSetAnnotations" : { },
"securityGroups" : [ ],
"stack" : "stg",
"strategy" : "",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy New Worker",
"refId" : "4",
"requisiteStageRefIds" : [ "6" ],
"type" : "deploy"
}, {
"account" : "project-k8s-devstg",
"annotations" : { },
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"containers" : [ {
"args" : [ "manage.py", "migrate" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-stg-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"account" : "project-sr-ecr",
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : { },
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : { },
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"labels" : { },
"name" : "Run Migrations",
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"refId" : "5",
"requisiteStageRefIds" : [ "7" ],
"type" : "runJob"
}, {
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-web-stg-worker",
"completeOtherBranchesThenFail" : false,
"continuePipeline" : true,
"credentials" : "project-k8s-devstg",
"failPipeline" : false,
"interestingHealthProviderNames" : [ "KubernetesService" ],
"name" : "Destroy Prev Worker",
"namespaces" : [ "demo-web" ],
"refId" : "6",
"requisiteStageRefIds" : [ "5" ],
"target" : "current_asg_dynamic",
"type" : "destroyServerGroup"
}, {
"account" : "project-k8s-devstg",
"annotations" : { },
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"containers" : [ {
"args" : [ "manage.py", "collectstatic", "--noinput", "-v", "0" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-stg-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"account" : "project-sr-ecr",
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : { },
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : { },
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"labels" : { },
"name" : "Upload Static Assets",
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"refId" : "7",
"requisiteStageRefIds" : [ "2" ],
"type" : "runJob"
}, {
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-web-stg-beat",
"completeOtherBranchesThenFail" : false,
"continuePipeline" : true,
"credentials" : "project-k8s-devstg",
"failPipeline" : false,
"interestingHealthProviderNames" : [ "KubernetesService" ],
"name" : "Destroy Prev Beat",
"namespaces" : [ "demo-web" ],
"refId" : "8",
"requisiteStageRefIds" : [ "5" ],
"target" : "current_asg_dynamic",
"type" : "destroyServerGroup"
}, {
"clusters" : [ {
"account" : "project-k8s-devstg",
"application" : "demo-web",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ "-m", "celery", "beat", "-A", "demo-project", "-l", "info" ],
"command" : [ "python" ],
"envFrom" : [ {
"prefix" : "",
"secretRef" : {
"name" : "app-demo-stg-config",
"optional" : false
}
} ],
"envVars" : [ ],
"imageDescription" : {
"imageId" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}",
"registry" : "111111111111.dkr.ecr.us-east-2.amazonaws.com",
"repository" : "demo-web-backend",
"tag" : "${TAG}",
"uri" : "111111111111.dkr.ecr.us-east-2.amazonaws.com/demo-web-backend:${TAG}"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : {
"cpu" : "100m",
"memory" : "300Mi"
},
"livenessProbe" : {
"failureThreshold" : 2,
"handler" : {
"execAction" : {
"commands" : [ "python", "kubernetes/scripts/liveness/logs-timestamp-check.py", "\"-s 600\"" ]
},
"httpGetAction" : {
"path" : "/",
"port" : 80,
"uriScheme" : "HTTP"
},
"tcpSocketAction" : {
"port" : 80
},
"type" : "EXEC"
},
"initialDelaySeconds" : 120,
"periodSeconds" : 30,
"successThreshold" : 1,
"timeoutSeconds" : 10
},
"name" : "demo-web-backend",
"ports" : [ ],
"requests" : {
"cpu" : "50m",
"memory" : "200Mi"
},
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "beat",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ ],
"namespace" : "demo-web",
"nodeSelector" : {
"kops.k8s.io/instancegroup" : "nodes"
},
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-web",
"replicaSetAnnotations" : { },
"securityGroups" : [ ],
"stack" : "stg",
"strategy" : "",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy New Beat",
"refId" : "9",
"requisiteStageRefIds" : [ "8" ],
"type" : "deploy"
}, {
"completeOtherBranchesThenFail" : false,
"continuePipeline" : true,
"failPipeline" : false,
"job" : "demo-app-stg-send-notifications",
"markUnstableAsSuccessful" : false,
"master" : "aws-jenkins",
"name" : "Notify",
"parameters" : {
"appEnv" : "stg",
"appName" : "demo-app",
"appRepo" : "[email protected]:organization/demo-web-backend.git",
"branchName" : "master",
"currentCommitId" : "${CURRENT_COMMIT_ID}",
"notificationsChannel" : "product-release-be",
"previousCommitId" : "${PREVIOUS_COMMIT_ID}"
},
"refId" : "10",
"requisiteStageRefIds" : [ "3" ],
"type" : "jenkins"
} ],
"disabled" : false,
"id" : "d9135563-2de5-4dde-9b60-1111111111111",
"updateTs" : "1559156396000",
"notifications" : [ {
"address" : "engineering-alerts",
"level" : "pipeline",
"type" : "slack",
"when" : [ "pipeline.starting", "pipeline.complete", "pipeline.failed" ]
} ]
}
Raw
spk-demo-app-uwsgi-nginx-py-flask-celery-flower-pipeline-metadata.json
{
"keepWaitingPipelines" : false,
"appConfig" : { },
"limitConcurrent" : true,
"application" : "demo-app",
"lastModifiedBy" : "demo-user",
"name" : "PROD - Deploy Project API",
"stages" : [ {
"clusters" : [ {
"account" : "demo-app-prod",
"application" : "demo-app",
"capacity" : {
"desired" : 2,
"max" : 3,
"min" : 1
},
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ ],
"command" : [ ],
"envFrom" : [ ],
"envVars" : [ {
"name" : "S3_ENDPOINT_URL",
"value" : "s3.amazonaws.com"
}, {
"name" : "BACKUP_BUCKET",
"value" : "project-demo-project-prod-backup"
}, {
"name" : "EXTERNAL_API_URL",
"value" : "https://api.domain.com.ar"
}, {
"name" : "ENVIRONMENT",
"value" : "prod"
}, {
"name" : "Project_FTP_SERVER",
"value" : "exchange.demo-app.com"
}, {
"name" : "CELERY_BROKER",
"value" : "pyamqp://guest@demo-app-prd-rmq"
},{
"name" : "TIMEZONE",
"value" : "America/Argentina/Buenos_Aires"
}, {
"name" : "DEBUG",
"value" : "false"
}, {
"name" : "REGION_NAME",
"value" : "us-east-1"
}, {
"envSource" : {
"secretSource" : {
"key" : "DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "DB_CONN_STRING"
},{
"envSource" : {
"secretSource" : {
"key" : "CELERY_DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "CELERY_DB_CONN_STRING"
}, {
"envSource" : {
"secretSource" : {
"key" : "FLASK_SECRET_KEY",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "FLASK_SECRET_KEY"
}, {
"envSource" : {
"secretSource" : {
"key" : "ROLLBAR_ACCESS_TOKEN",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "ROLLBAR_ACCESS_TOKEN"
}],
"imageDescription" : {
"account" : "project-shared-registry-1",
"fromTrigger" : true,
"imageId" : "111111111111.dkr.ecr.us-east-1.amazonaws.com/demo-project-uwsgi-nginx-flask:prod-.*",
"registry" : "111111111111.dkr.ecr.us-east-1.amazonaws.com",
"repository" : "demo-project-uwsgi-nginx-flask",
"tag" : "prod-.*"
},
"imagePullPolicy" : "IFNOTPRESENT",
"name" : "demo-project-uwsgi-nginx-flask",
"ports" : [ {
"containerPort" : 80,
"name" : "http",
"protocol" : "TCP"
} ],
"readinessProbe" : {
"failureThreshold" : 3,
"handler" : {
"execAction" : {
"commands" : [ ]
},
"httpGetAction" : {
"path" : "/admin/login",
"port" : 80,
"uriScheme" : "HTTP"
},
"tcpSocketAction" : {
"port" : 80
},
"type" : "HTTP"
},
"initialDelaySeconds" : 0,
"periodSeconds" : 10,
"successThreshold" : 1,
"timeoutSeconds" : 1
},
"volumeMounts" : [ ]
} ],
"delayBeforeDisableSec" : 10,
"deployment" : {
"enabled" : false
},
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "api",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ "demo-app-prd-elb" ],
"maxRemainingAsgs" : 5,
"namespace" : "demo-app-ns",
"nodeSelector" : { },
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "default",
"replicaSetAnnotations" : { },
"scaleDown" : true,
"scalingPolicy" : {
"cpuUtilization" : {
"target" : 40
}
},
"securityGroups" : [ ],
"stack" : "prd",
"strategy" : "redblack",
"targetSize" : 2,
"terminationGracePeriodSeconds" : 30,
"volumeSources" : [ ]
} ],
"name" : "Deploy API",
"refId" : "1",
"requisiteStageRefIds" : [ ],
"type" : "deploy"
}, {
"clusters" : [ {
"account" : "demo-app-prod",
"application" : "demo-app",
"capacity" : {
"max" : 6,
"min" : 2
},
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ "celery", "-A", "project.demo-project.tasks", "worker", "--loglevel=INFO" ],
"command" : [ ],
"envFrom" : [ ],
"envVars" : [ {
"name" : "S3_ENDPOINT_URL",
"value" : "s3.amazonaws.com"
}, {
"name" : "BACKUP_BUCKET",
"value" : "project-demo-project-prod-backup"
}, {
"name" : "EXTERNAL_API_URL",
"value" : "https://api.domain.com.ar"
}, {
"name" : "ENVIRONMENT",
"value" : "prod"
}, {
"name" : "Project_FTP_SERVER",
"value" : "exchange.demo-app.com"
}, {
"name" : "CELERY_BROKER",
"value" : "pyamqp://guest@demo-app-prd-rmq"
},{
"name" : "TIMEZONE",
"value" : "America/Argentina/Buenos_Aires"
}, {
"name" : "DEBUG",
"value" : "false"
}, {
"name" : "REGION_NAME",
"value" : "us-east-1"
}, {
"envSource" : {
"secretSource" : {
"key" : "DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "DB_CONN_STRING"
},{
"envSource" : {
"secretSource" : {
"key" : "CELERY_DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "CELERY_DB_CONN_STRING"
}, {
"envSource" : {
"secretSource" : {
"key" : "FLASK_SECRET_KEY",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "FLASK_SECRET_KEY"
}, {
"envSource" : {
"secretSource" : {
"key" : "ROLLBAR_ACCESS_TOKEN",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "ROLLBAR_ACCESS_TOKEN"
}],
"imageDescription" : {
"account" : "project-shared-registry-1",
"fromTrigger" : true,
"imageId" : "111111111111.dkr.ecr.us-east-1.amazonaws.com/demo-project-uwsgi-nginx-flask:prod-.*",
"registry" : "111111111111.dkr.ecr.us-east-1.amazonaws.com",
"repository" : "demo-project-uwsgi-nginx-flask",
"tag" : "prod-.*"
},
"imagePullPolicy" : "IFNOTPRESENT",
"name" : "demo-project-uwsgi-nginx-flask",
"ports" : [ {
"containerPort" : 80,
"name" : "http",
"protocol" : "TCP"
} ],
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "worker",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ ],
"namespace" : "demo-app-ns",
"nodeSelector" : { },
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "default",
"replicaSetAnnotations" : { },
"scalingPolicy" : {
"cpuUtilization" : {
"target" : 40
}
},
"securityGroups" : [ ],
"stack" : "prd",
"strategy" : "",
"targetSize" : 2,
"terminationGracePeriodSeconds" : 30,
"volumeSources" : [ ]
} ],
"name" : "Deploy worker",
"refId" : "2",
"requisiteStageRefIds" : [ ],
"type" : "deploy"
}, {
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-app-prd-worker",
"credentials" : "demo-app-prod",
"interestingHealthProviderNames" : [ "KubernetesService" ],
"name" : "Disable Server Group",
"namespaces" : [ "default" ],
"refId" : "3",
"requisiteStageRefIds" : [ "2" ],
"target" : "ancestor_asg_dynamic",
"type" : "disableServerGroup"
}, {
"action" : "scale_exact",
"capacity" : {
"desired" : 0
},
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-app-prd-worker",
"credentials" : "demo-app-prod",
"name" : "Resize Server Group",
"namespaces" : [ "default" ],
"refId" : "4",
"requisiteStageRefIds" : [ "3" ],
"resizeType" : "exact",
"target" : "ancestor_asg_dynamic",
"type" : "resizeServerGroup"
}, {
"cloudProvider" : "kubernetes",
"cloudProviderType" : "kubernetes",
"cluster" : "demo-app-prd-worker",
"credentials" : "demo-app-prod",
"interestingHealthProviderNames" : [ "KubernetesService" ],
"name" : "Destroy Server Group",
"namespaces" : [ "default" ],
"refId" : "5",
"requisiteStageRefIds" : [ "4" ],
"target" : "ancestor_asg_dynamic",
"type" : "destroyServerGroup"
}, {
"clusters" : [ {
"account" : "demo-app-prod",
"application" : "demo-app",
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ "flower", "-A", "project.demo-project.tasks", "--port=5555" ],
"command" : [ ],
"envFrom" : [ ],
"envVars" : [ {
"name" : "S3_ENDPOINT_URL",
"value" : "s3.amazonaws.com"
}, {
"name" : "BACKUP_BUCKET",
"value" : "project-demo-project-prod-backup"
}, {
"name" : "EXTERNAL_API_URL",
"value" : "https://api.domain.com.ar"
}, {
"name" : "ENVIRONMENT",
"value" : "prod"
}, {
"name" : "Project_FTP_SERVER",
"value" : "exchange.demo-app.com"
}, {
"name" : "CELERY_BROKER",
"value" : "pyamqp://guest@demo-app-prd-rmq"
},{
"name" : "TIMEZONE",
"value" : "America/Argentina/Buenos_Aires"
}, {
"name" : "DEBUG",
"value" : "false"
}, {
"name" : "REGION_NAME",
"value" : "us-east-1"
}, {
"envSource" : {
"secretSource" : {
"key" : "DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "DB_CONN_STRING"
},{
"envSource" : {
"secretSource" : {
"key" : "CELERY_DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "CELERY_DB_CONN_STRING"
}, {
"envSource" : {
"secretSource" : {
"key" : "FLASK_SECRET_KEY",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "FLASK_SECRET_KEY"
}, {
"envSource" : {
"secretSource" : {
"key" : "ROLLBAR_ACCESS_TOKEN",
"optional" : false,
"secretName" : "demo-app-api-prod"
}
},
"name" : "ROLLBAR_ACCESS_TOKEN"
}],
"imageDescription" : {
"account" : "project-shared-registry-1",
"fromTrigger" : true,
"imageId" : "111111111111.dkr.ecr.us-east-1.amazonaws.com/demo-project-uwsgi-nginx-flask:prod-.*",
"registry" : "111111111111.dkr.ecr.us-east-1.amazonaws.com",
"repository" : "demo-project-uwsgi-nginx-flask",
"tag" : "prod-.*"
},
"imagePullPolicy" : "IFNOTPRESENT",
"name" : "demo-project-uwsgi-nginx-flask",
"ports" : [ {
"containerPort" : 80,
"name" : "http",
"protocol" : "TCP"
} ],
"volumeMounts" : [ ]
} ],
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "flower",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ "demo-app-prd-flower" ],
"namespace" : "demo-app-ns",
"nodeSelector" : { },
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "default",
"replicaSetAnnotations" : { },
"securityGroups" : [ ],
"stack" : "prd",
"strategy" : "highlander",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"volumeSources" : [ ]
} ],
"name" : "Deploy flower",
"refId" : "6",
"requisiteStageRefIds" : [ ],
"type" : "deploy"
} ],
"index" : 1,
"id" : "bbcbab42-b67e-4201-90f5-889aebaefdb5",
"triggers" : [ {
"account" : "project-shared-registry-1",
"enabled" : true,
"registry" : "111111111111.dkr.ecr.us-east-1.amazonaws.com",
"repository" : "demo-project-uwsgi-nginx-flask",
"tag" : "prod-.*",
"type" : "docker"
} ],
"updateTs" : "1540816686000",
"notifications" : [ {
"address" : "tech-spinnaker",
"level" : "pipeline",
"type" : "slack",
"when" : [ "pipeline.starting", "pipeline.complete", "pipeline.failed" ]
} ]
}
Raw
spk-demo-app-uwsgi-nginx-py-flask-pipeline-metadata.json
{
"keepWaitingPipelines" : false,
"limitConcurrent" : true,
"application" : "demo_app",
"lastModifiedBy" : "demo_user",
"name" : "DEV - Deploy DEMO APP API",
"stages" : [ {
"name" : "Deploy API",
"refId" : "1",
"requisiteStageRefIds" : [ ],
"type" : "deploy"
}, {
"clusters" : [ {
"account" : "demo-project-dev",
"application" : "demo_app",
"capacity" : {
"desired" : 2,
"max" : 3,
"min" : 1
},
"cloudProvider" : "kubernetes",
"containers" : [ {
"args" : [ ],
"command" : [ ],
"envFrom" : [ ],
"envVars" : [ {
"name" : "PYTHONPATH",
"value" : "/app"
}, {
"name" : "TIMEZONE",
"value" : "America/Argentina/Buenos_Aires"
}, {
"name" : "DEBUG",
"value" : "true"
}, {
"name" : "ENVIRONMENT",
"value" : "dev"
}, {
"name" : "COGNITO_USERS_POOL_ID",
"value" : "us-east-1_xxxxxxxxx"
}, {
"name" : "COGNITO_REGION",
"value" : "us-east-1"
}, {
"envSource" : {
"secretSource" : {
"key" : "DB_CONN_STRING",
"optional" : false,
"secretName" : "demo-app-api-dev"
}
},
"name" : "DB_CONN_STRING"
},{
"envSource" : {
"secretSource" : {
"key" : "FLASK_SECRET_KEY",
"optional" : false,
"secretName" : "demo-app-api-dev"
}
},
"name" : "FLASK_SECRET_KEY"
}, {
"envSource" : {
"secretSource" : {
"key" : "ROLLBAR_ACCESS_TOKEN",
"optional" : false,
"secretName" : "demo-app-api-dev"
}
},
"name" : "ROLLBAR_ACCESS_TOKEN"
}],
"imageDescription" : {
"account" : "demo-shared-ecr-registry-1",
"fromTrigger" : true,
"imageId" : "111111111111.dkr.ecr.us-east-1.amazonaws.com/demo-project-app-uwsgi-nginx-flask:dev-.*",
"registry" : "111111111111.dkr.ecr.us-east-1.amazonaws.com",
"repository" : "demo-project-app-uwsgi-nginx-flask",
"tag" : "dev-.*"
},
"imagePullPolicy" : "IFNOTPRESENT",
"limits" : { },
"name" : "demo-project-app-uwsgi-nginx-flask",
"ports" : [ {
"containerPort" : 80,
"name" : "http",
"protocol" : "TCP"
} ],
"readinessProbe" : {
"failureThreshold" : 3,
"handler" : {
"execAction" : {
"commands" : [ ]
},
"httpGetAction" : {
"path" : "/status",
"port" : 80,
"uriScheme" : "HTTP"
},
"tcpSocketAction" : {
"port" : 80
},
"type" : "HTTP"
},
"initialDelaySeconds" : 0,
"periodSeconds" : 10,
"successThreshold" : 1,
"timeoutSeconds" : 1
},
"requests" : { },
"volumeMounts" : [ ]
} ],
"delayBeforeDisableSec" : 10,
"dnsPolicy" : "ClusterFirst",
"events" : [ ],
"freeFormDetails" : "api",
"initContainers" : [ ],
"interestingHealthProviderNames" : [ "KubernetesContainer", "KubernetesPod" ],
"loadBalancers" : [ "demo-app-dev-api-cip" ],
"maxRemainingAsgs" : 3,
"namespace" : "demo-project-app-api-dev",
"nodeSelector" : { },
"podAnnotations" : { },
"provider" : "kubernetes",
"region" : "demo-project-app-api-dev",
"replicaSetAnnotations" : { },
"rollback" : {
"onFailure" : true
},
"scaleDown" : true,
"securityGroups" : [ ],
"stack" : "dev",
"strategy" : "redblack",
"targetSize" : 1,
"terminationGracePeriodSeconds" : 30,
"tolerations" : [ ],
"volumeSources" : [ ]
} ],
"name" : "Deploy API",
"refId" : "1",
"requisiteStageRefIds" : [ ],
"type" : "deploy"
} ],
"index" : 0,
"id" : "1d3f9c7e-d213-40eb-bf54-xxxxxxxxxxxx",
"triggers" : [ {
"account" : "demo-shared-registry-1",
"enabled" : true,
"registry" : "111111111111.dkr.ecr.us-east-1.amazonaws.com",
"repository" : "demo-project-app-uwsgi-nginx-flask",
"tag" : "dev-.*",
"type" : "docker"
} ],
"updateTs" : "1539968277000",
"notifications" : [ {
"address" : "tech-spinnaker",
"level" : "pipeline",
"type" : "slack",
"when" : [ "pipeline.starting", "pipeline.complete", "pipeline.failed" ]
} ]
}
Raw
WORFLOW_CONSIDERATIONS.md

DEVELOPMENT WORKFLOW CONSIDERATIONS

NOTE1:

  • Spinnaker will allow to easily manage from the UI
  • Deployment Strategy (eg: Blue/Gren, Canary, Highlander and more ),
  • Deploy Multiple services in parallel (eg: API, Worker, Cache, etc),
  • Deployment type (Deployment or Replica Set),
  • K8s Service config & integration (ClusterIP, Load Balancer, NodePort, and more),
  • Autoscaling (K8s HPA -> Replicas#),
  • Maximum number of server groups (K8s RS / deployments) to leave,
  • Pod annotations,
  • Node Selector (Pod to Instance Group affinity),
  • Docker container cmds & arguments,
  • Docker container ENV Vars and Secrets,
  • Resource request and limits both CPU and Memory,
  • Ports,
  • Volumes,
  • Readiness Check and Probes
  • User and SELinux containers security context.
  • K8s Multi-cluster support

NOTE2:

Spinnaker natively supports replica sets / deployments (server groups) rollbacks from the UI -> but DB migration changes should have also rollback migrations, or probably simpler to be retrocompatible with at least 2 or 3 previous deployed releases to avoid rollback DB inconsistencies.

NOTE3:

The tags can be improved attaching the latest short git commit hash id if needed as show in workflow example B.

Tags eg for Flow A (Jenkins -> build docker image -> push to AWS ECR)

  • parsedImage: [name:11111111111.dkr.ecr.us-east-1.amazonaws.com/3pt-api-x, tag:dev-149]
  • parsedImage: [name:11111111111.dkr.ecr.us-east-1.amazonaws.com/3pt-api-x, tag:stg-91]
  • parsedImage: [name:11111111111.dkr.ecr.us-east-1.amazonaws.com/3pt-api-x, tag:prod-50]

Tags eg for Flow B (Spinnaker -> Jenkins -> git tag-> build docker image -> ush to AWS ECR)

  • parsedImage: [name:11111111111.dkr.ecr.us-east-1.amazonaws.com/3pt-api-x, tag:dev-550-b2cdc01]
  • parsedImage: [name:11111111111.dkr.ecr.us-east-1.amazonaws.com/3pt-api-x, tag:release-150-24b93f5]-
  • parsedImage: [name:11111111111.dkr.ecr.us-east-1.amazonaws.com/3pt-api-x, tag:release-150-24b93f5]
$ git tag -a release_2019-06-19-09-32-29 24b93f5 -m 'User (Sun Jun 16 10:58:11 2019): [JIRA-3531] coming soon mvp (#518) 24b93f5'`
$ git push --tags #to [email protected]:Organization/repo.git 
$* [new tag] release_2019-06-19-09-32-29 -> release_2019-06-19-09-32-29

NOTE4:

Performance testing it's not included in the automated workflow, can probably be executed from a stand alone job or tool, as a result App / Cluster parameters can be fine tuned in real time based on the current live results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment