Skip to content

Instantly share code, notes, and snippets.

@thelicato

thelicato/gitlab_security_pipeline_demo.md

Last active September 7, 2024 17:12
Show Gist options
  • Save thelicato/109d57bb7436520c609b5bab84e80b14 to your computer and use it in GitHub Desktop.
Save thelicato/109d57bb7436520c609b5bab84e80b14 to your computer and use it in GitHub Desktop.
Download ZIP
GitLab Security Pipeline
Raw
gitlab_security_pipeline_demo.md

GitLab Security Pipeline Demo

This READMEexplains the whole process, from start to finish, on how to setup a custom Security Pipeline using GitLab. The starting point is a an already vulnerable application, for this example https://github.com/appsecco/dvna will be used.

The repo is cloned and pushed as-is in a new GitLab Project (empty). The first thing to do is enable a runner for our pipeline, this can be done in the Settings -> CI/CD section of the project

Table of Contents

  1. Add default SAST
  2. Customize Stages
  3. Custom (dummy) reporting
  4. Add DAST
  5. Add Dastardly
  6. Add Container Scan
  7. Other Tools

Add default SAST

The first step explained is the integration of the default SAST template provided by GitLab, to integrate it a file called .gitlab-ci.yml must be created in the root folder of the project with the following content:

# You can override the included template(s) by including variable overrides
# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings
# Note that environment variables can be set in several places
# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
stages:
  - test
 
include:
  - template: Security/SAST.gitlab-ci.yml

References: https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/SAST.gitlab-ci.yml

As you may notice there is a single stage in this pipeline called test, this is the default stage for the SAST template. What if we want to change it?

Customize Stages

We can override every component of a template by overriding the values we want to change. If we want a stage called sast we can do the following:

# You can override the included template(s) by including variable overrides
# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings
# Note that environment variables can be set in several places
# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
stages:
  - sast

include:
  - template: Security/SAST.gitlab-ci.yml

sast:
  stage: sast

Add Secrets Detection

GitLab provides multiple free templates, if we want to add the Secrets Detection template we can do that by just including it:

# You can override the included template(s) by including variable overrides
# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings
# Note that environment variables can be set in several places
# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence
stages:
  - test
  - sast

include:
  - template: Security/Secret-Detection.gitlab-ci.yml
  - template: Security/SAST.gitlab-ci.yml

sast:
  stage: sast

In this case we re-added the test stage and did not change the stage associated with the Secrets Detection template (for no particular reason).

Custom (dummy) reporting

As you may have noticed the artifacts provided by the current steps of the pipeline are reports. GitLab defines a special kind of artifacts called reports. These are the available types:

  • accessibility
  • annotations
  • api_fuzzing
  • browser_performance
  • coverage_report
  • codequality
  • container_scanning
  • coverage_fuzzing
  • cyclonedx
  • dast
  • dependency_scanning
  • dotenv
  • junit
  • load_performance
  • metrics
  • requirements
  • repository_xray
  • sast
  • secret_detection
  • terraform

If we use the free version of GitLab we do not have a dashboard with the results retrieved from these reports. But since these are standardized we can use external tools to obtain some kind of reporting. For example we can use https://github.com/pcfens/sast-parser.git to obtain a HTML report of the SAST results. We can do that manually, but we can also integrate it inside our pipeline by using custom jobs instead of the default SAST template. In fact, since the output of every job of the SAST template, provides the same output filename we need a way to have them all with different (and predictable) names to use them with the sast-parser tool specified above. So we are going to use the same jobs executed in our pipeline, but manually (and changing the filename of the artifact):

# You can override the included template(s) by including variable overrides

# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings

# Note that environment variables can be set in several places

# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence

stages:
  - test
  - sast

include:
  - template: Security/Secret-Detection.gitlab-ci.yml
  # - template: Security/SAST.gitlab-ci.yml

nodejs-scan-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan:4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-nodejs-scan.json
  artifacts:
    reports:
      sast: gl-sast-report-nodejs-scan.json
    paths:
      - gl-sast-report-nodejs-scan.json

semgrep-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:4
  variables:
    SEARCH_MAX_DEPTH: 20
    SAST_ANALYZER_IMAGE_TAG: 4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-semgrep.json
  artifacts:
    reports:
      sast: gl-sast-report-semgrep.json
    paths:
      - gl-sast-report-semgrep.json

sast-report:
  stage: sast
  image: python:alpine3.18
  allow_failure: true
  needs:
    - job: nodejs-scan-sast-custom
      artifacts: true
    - job: semgrep-sast-custom
      artifacts: true
  before_script:
    - apk add git
  script:
    - git clone https://github.com/pcfens/sast-parser.git
    - cd sast-parser
    - pip install -r requirements.txt
    - python3 parse-sast.py ../gl-sast-report-semgrep.json ../gl-sast-report-nodejs-scan.json > ../sast_report.html
  artifacts:
    paths:
      - sast_report.html

Add DAST

Also to add the DAST we can use the default template provided by GitLab. In this case we add a dummy build step and, to avoid losing time with build and push steps, use the default Docker Image provided by AppSecco (appsecco/dvna:sqlite)

# You can override the included template(s) by including variable overrides

# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings

# Note that environment variables can be set in several places

# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence

stages:
  - test
  - sast
  - build
  - dast

include:
  - template: Security/Secret-Detection.gitlab-ci.yml
  # - template: Security/SAST.gitlab-ci.yml
  - template: Security/DAST.gitlab-ci.yml

nodejs-scan-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan:4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-nodejs-scan.json
  artifacts:
    reports:
      sast: gl-sast-report-nodejs-scan.json
    paths:
      - gl-sast-report-nodejs-scan.json

semgrep-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:4
  variables:
    SEARCH_MAX_DEPTH: 20
    SAST_ANALYZER_IMAGE_TAG: 4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-semgrep.json
  artifacts:
    reports:
      sast: gl-sast-report-semgrep.json
    paths:
      - gl-sast-report-semgrep.json

sast-report:
  stage: sast
  image: python:alpine3.18
  allow_failure: true
  needs:
    - job: nodejs-scan-sast-custom
      artifacts: true
    - job: semgrep-sast-custom
      artifacts: true
  before_script:
    - apk add git
  script:
    - git clone https://github.com/pcfens/sast-parser.git
    - cd sast-parser
    - pip install -r requirements.txt
    - python3 parse-sast.py ../gl-sast-report-semgrep.json ../gl-sast-report-nodejs-scan.json > ../sast_report.html
  artifacts:
    paths:
      - sast_report.html

dummy_build:
  stage: build
  script:
    - echo "Build completed!"

dast:
  services: # use services to link your app container to the dast job
    - name: appsecco/dvna:sqlite
      alias: dvna
  rules:
    - when: always
  variables:
    DAST_WEBSITE: http://dvna:9090
    DAST_FULL_SCAN_ENABLED: "true" # do a full scan
    DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler

Add Dastardly

Now we can add an external tool like Dastardly from PortSwigger. For this tool there are a few things to notice:

  • it returns a junit report artifact;
  • it has report: always for the artifact;
  • it has allow_failure: true since it fails if there are vulnerabilities higher than Info;
# You can override the included template(s) by including variable overrides

# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings

# Note that environment variables can be set in several places

# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence

stages:
  - test
  - sast
  - build
  - dast

include:
  - template: Security/Secret-Detection.gitlab-ci.yml
  # - template: Security/SAST.gitlab-ci.yml
  - template: Security/DAST.gitlab-ci.yml

nodejs-scan-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan:4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-nodejs-scan.json
  artifacts:
    reports:
      sast: gl-sast-report-nodejs-scan.json
    paths:
      - gl-sast-report-nodejs-scan.json

semgrep-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:4
  variables:
    SEARCH_MAX_DEPTH: 20
    SAST_ANALYZER_IMAGE_TAG: 4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-semgrep.json
  artifacts:
    reports:
      sast: gl-sast-report-semgrep.json
    paths:
      - gl-sast-report-semgrep.json

sast-report:
  stage: sast
  image: python:alpine3.18
  allow_failure: true
  needs:
    - job: nodejs-scan-sast-custom
      artifacts: true
    - job: semgrep-sast-custom
      artifacts: true
  before_script:
    - apk add git
  script:
    - git clone https://github.com/pcfens/sast-parser.git
    - cd sast-parser
    - pip install -r requirements.txt
    - python3 parse-sast.py ../gl-sast-report-semgrep.json ../gl-sast-report-nodejs-scan.json > ../sast_report.html
  artifacts:
    paths:
      - sast_report.html

dummy_build:
  stage: build
  script:
    - echo "Build completed!"

dast:
  services: # use services to link your app container to the dast job
    - name: appsecco/dvna:sqlite
      alias: dvna
  rules:
    - when: always
  variables:
    DAST_WEBSITE: http://dvna:9090
    DAST_FULL_SCAN_ENABLED: "true" # do a full scan
    DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler

dastardly:
  stage: dast
  image:
    name: public.ecr.aws/portswigger/dastardly:latest
    entrypoint: [""]
  services: # use services to link your app container to the dast job
    - name: appsecco/dvna:sqlite
      alias: dvna
  variables:
    # No need to clone the repo, we exclusively work on artifacts. See
    # https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
    GIT_STRATEGY: none
    BURP_START_URL: "http://dvna:9090"
    BURP_REPORT_FILE_PATH: "$CI_PROJECT_NAME-dastardly-report.xml"
  allow_failure: true
  artifacts:
    when: always
    reports:
      junit: $CI_PROJECT_NAME-dastardly-report.xml
    paths:
      - $CI_PROJECT_NAME-dastardly-report.xml
  script:
    - /usr/local/bin/dastardly-entrypoint.sh dastardly

Add Container Scan

Finally we can add the Container Scanning template for analysis on the Container image created in the build stage:

# You can override the included template(s) by including variable overrides

# SAST customization: https://docs.gitlab.com/ee/user/application_security/sast/#customizing-the-sast-settings
# Secret Detection customization: https://docs.gitlab.com/ee/user/application_security/secret_detection/#customizing-settings
# Dependency Scanning customization: https://docs.gitlab.com/ee/user/application_security/dependency_scanning/#customizing-the-dependency-scanning-settings
# Container Scanning customization: https://docs.gitlab.com/ee/user/application_security/container_scanning/#customizing-the-container-scanning-settings

# Note that environment variables can be set in several places

# See https://docs.gitlab.com/ee/ci/variables/#cicd-variable-precedence

stages:
  - test
  - sast
  - build
  - dast
  - container-scan

include:
  - template: Security/Secret-Detection.gitlab-ci.yml
  # - template: Security/SAST.gitlab-ci.yml
  - template: Security/DAST.gitlab-ci.yml
  - template: Security/Container-Scanning.gitlab-ci.yml

nodejs-scan-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/nodejs-scan:4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-nodejs-scan.json
  artifacts:
    reports:
      sast: gl-sast-report-nodejs-scan.json
    paths:
      - gl-sast-report-nodejs-scan.json

semgrep-sast-custom:
  stage: sast
  image: registry.gitlab.com/gitlab-org/security-products/analyzers/semgrep:4
  variables:
    SEARCH_MAX_DEPTH: 20
    SAST_ANALYZER_IMAGE_TAG: 4
  script:
    - /analyzer run
    - mv gl-sast-report.json gl-sast-report-semgrep.json
  artifacts:
    reports:
      sast: gl-sast-report-semgrep.json
    paths:
      - gl-sast-report-semgrep.json

sast-report:
  stage: sast
  image: python:alpine3.18
  allow_failure: true
  needs:
    - job: nodejs-scan-sast-custom
      artifacts: true
    - job: semgrep-sast-custom
      artifacts: true
  before_script:
    - apk add git
  script:
    - git clone https://github.com/pcfens/sast-parser.git
    - cd sast-parser
    - pip install -r requirements.txt
    - python3 parse-sast.py ../gl-sast-report-semgrep.json ../gl-sast-report-nodejs-scan.json > ../sast_report.html
  artifacts:
    paths:
      - sast_report.html

dummy_build:
  stage: build
  script:
    - echo "Build completed!"

dast:
  services: # use services to link your app container to the dast job
    - name: appsecco/dvna:sqlite
      alias: dvna
  rules:
    - when: always
  variables:
    DAST_WEBSITE: http://dvna:9090
    DAST_FULL_SCAN_ENABLED: "true" # do a full scan
    DAST_BROWSER_SCAN: "true" # use the browser-based GitLab DAST crawler

dastardly:
  stage: dast
  image:
    name: public.ecr.aws/portswigger/dastardly:latest
    entrypoint: [""]
  services: # use services to link your app container to the dast job
    - name: appsecco/dvna:sqlite
      alias: dvna
  variables:
    # No need to clone the repo, we exclusively work on artifacts. See
    # https://docs.gitlab.com/ee/ci/runners/README.html#git-strategy
    GIT_STRATEGY: none
    BURP_START_URL: "http://dvna:9090"
    BURP_REPORT_FILE_PATH: "$CI_PROJECT_NAME-dastardly-report.xml"
  allow_failure: true
  artifacts:
    when: always
    reports:
      junit: $CI_PROJECT_NAME-dastardly-report.xml
    paths:
      - $CI_PROJECT_NAME-dastardly-report.xml
  script:
    - /usr/local/bin/dastardly-entrypoint.sh dastardly

container_scanning:
  stage: container-scan
  variables:
    CS_IMAGE: appsecco/dvna:sqlite

Other Tools

Other tools and templates can be used to enhance the output of this Security Pipeline:

  • Dependency Scanning Template
  • Retire.js tool
  • OWASP Dependency Check
  • Third-Party Tools (Sonarqube, Snyk, Fortify and so on)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment