DevOps Meetup Karlsruhe - Concourse CI

Abstract

Some years ago, Continuous Integration (CI) or at least Continuous Delivery (CD) was the "latest and greatest". If you managed to set it up for your project you were playing amongst the cool kids. Nowadays having a CI/CD pipeline can be considered mainstream and without a fully automated delivery workflow in place, you are likely to lose important competitive advantage.

One can argue that CI/CD pipelines became a key driver for a team's software delivery performance. They evolved from some hacky shell scripts - that were created once and never ever touched again - to a critical component of your software project that need to be maintained and scaled...

During this presentation we will introduce some patterns and general requirements for modern delivery pipelines and discuss why Concourse might be the right tool to develop and maintain pipelines "that don't suck".

The target audience of the talk are all software and infrastructure developers that want to learn how to use Concourse to build maintainable and scalable delivery pipelines.

CI/CD as a key driver for your team's performance regarding software delivery
from bleeding edge to state-of-the-art
from "do it - no matter how" to "how to do it right"
why is Concourse the right tool for the job?
- everything as code
- no UI
- stateless
- isolated builds, running in Docker container out-of-the-box
how to build pipelines that don't suck
pick up some points of the presentation and explain why Concourse might be a good fit
what shall we learn in the presentation
target audience

About the Speakers

Alexander

Michael studied computer science in Karlsruhe and later on founded the DevOps Meetup Karlsruhe in 2014. After more than 10 years in web development he got more and more involved with infrastructure and cloud topics and finally made this his full time profession when he joined the team behind the Bosch IoT Cloud in 2016. He is inofficially entitled "DevOps Evangelist" by his group lead and loves to challenge others in automation topics and company culture.

Brainstorming

META use themes from aviation for slides...

CI/CD - what's it all about Mimi
- Four key metrics
  - Lead time
  - Deployment frequency
  - Mean time to restore
  - Change fail percentage
Pipelines that don't suck - i.e. a good pipeline is --> https://github.com/sirech/talks/blob/master/2019-04-tw-build_pipelines.pdf
- code Alex
  - needs no modification via UI
  - is kept close to the app
  - is in version control
- maintainable Mimi
  - declarative (?)
  - locally executable
- reliable Alex
  - isolated build (Docker)
  - needs a tradeoff between isolation and speed
- fast Mimi
  - throw hardware at the problem
  - parallelization
  - avoid repeating steps
  - make use of caching
- visual Alex
  - avoid the "god step"
  - avoid logging overdose
  - visualize dependencies
- scalable (in a sense of how to maintain many many pipelines?) Mimi
  - find a tradeoff between reuse and coupling
  - follow a similar structure
  - parametrize steps
Operator's expecations vs developer's expectations Alex
How to set up Concourse Alex
- Test setup with Docker Compose
```
$ wget https://concourse-ci.org/docker-compose.yml
$ docker-compose up -d
$ open http://localhost:8080
# figure out default password 
```
- Deploy with BOSH
  - explain how BOSH works on a single slide (5min)
- Use Cloud installer tools
  - Control Tower (if we have enought time, set this up for demo)
  - BUCC
Concourse Basics and Architecture Mimi
- 3 + 1 main components
  - Web
  - DB
  - Worker(s)
  - fly CLI
- everything runs in (Docker) containers
- yaml-based
- highly opinionated (take it or leave it)
- stateless build jobs
  - no passing of artifacts between jobs
- user management + teams + authentication providers
  - maybe a reference to: requirements for a modern CI/CD server, see https://github.com/cicd-hackathon-stgt/docs
Basic Building Blocks of Concourse Pipelines Alex
- have picture in place, how the building blocks are related
- Pipeline
  - Glue code for blocks below
- Resources
  - artifacts used in the pipeline
  - can be checked (for changes), pulled from and pushed to
  - e.g. git, docker, artifactory, s3
  - see https://concourse-ci.org/included-resources.html
- Jobs
  - determine the action(s) of your pipeline
  - you cannot pass anything between jobs
  - see https://concourse-ci.org/jobs.html
- Tasks
  - smallest configurable unit in a Concourse pipeline
  - a function from inputs to outputs that can either succeed or fail
  - are combined to form a job in your pipeline
  - see https://concourse-ci.org/tasks.html
  - benefit of task.yml files
    - re-use
    - debugging without repiping (read from repo)
How to use the fly cli Mimi
- how to get it
- how to create pipelines
The Concourse Web UI Alex
- main page
- pipeline page
- how to trigger a job
- logs
Pipeline Examples Mimi
- Java build (+ deployment)
- Docker image build
"Advanced Concepts"
- hijacking containers
- handling versions (version files, pull requests)
- credentials management (with Credhub / Vault) Alex
- reusable pipeline tasks
- Pre-flight checks
- Meta Pipelines
- Jsonnet & spruce for templating
- Custom Resources
Recap: which points of "build pipelines that don't suck" are met? Alex
Disadvantages
- small community (compared to Jenkins)
- highly opinionated
- pipelines don't update automatically
  - can be "fixed" via meta-pipeline
- no multi-branch support
  - can be "fixed" via meta-pipeline
- no direct passing of any kind of artifact inbetween jobs
- you can trigger only jobs, not trigger the "full" pipeline
- since everything runs in containers in can become difficult to use Docker inside containers
- operations, if you don't have BOSH in place...
Comparison with Jenkins
- when to use what
- versioning of configuration / configuration as code
- communities / documentation resources
- pipelines as first class citizens
- usage of Docker
- (Jenkins) plugins vs (Concourse) resources
Summary

michaellihs/meetup-ka-concourse.md

DevOps Meetup Karlsruhe - Concourse CI

Abstract

About the Speakers

Brainstorming

References