Skip to content

Instantly share code, notes, and snippets.

@isutton
Last active August 30, 2024 18:09
Show Gist options
  • Save isutton/88ac5fe6a86b16311d81126a20277bb7 to your computer and use it in GitHub Desktop.
Save isutton/88ac5fe6a86b16311d81126a20277bb7 to your computer and use it in GitHub Desktop.
chart-streams original resources

GoReport Code Coverage GoDoc Reference GitHub Status Image Build Status

chartflow

chartflow is a thin layer on top of a Git repository to make it behave as a Helm-Charts repository would. With the the following advantages:

  • Promoting Git repository as source-of-authority over Helm-Charts;
  • Low-friction workflow, index.yaml and Chart tarballs are generated dynamically;
  • Allowing clients to reach branches and commit-ids, with Semantic Versioning;

The basic workflow is represented as:

chartflow diagram

Usage

The usage of chartflow is regular Helm-Chart repository. Therefore, you can employ Helm in command-line to interact with this repository. For instance:

helm repo add cs http://127.0.0.1:8080
helm repo update
helm search ...

Container Image

The container images are stored on quay.io/kubesmith/chartflow. To run it, execute:

podman run --publish=8080:8080 --tmpfs=/var/lib/chartflow quay.io/kubesmith/chartflow:latest
docker run --publish=8080:8080 --tmpfs=/var/lib/chartflow quay.io/kubesmith/chartflow:latest

Configuration

Configuration parameters are exposed as environment variables as well, therefore using the prefix CHART_STREAMS you can combine with the option name. For instance, clone-depth would then become CHART_STREAMS_CLONE_DEPTH as environment variables, and on command line would then become --clone-depth.

The configuration options available are:

Parameter Default Description
repo-url https://github.com/helm/charts.git Git repository URL
relative-dir stable Relative directory in Git repository
clone-depth 1 Amount of commits from Git repository
listen-addr 127.0.0.1:8080 Address the application will be listening on
working-dir /var/lib/chartflow Git repository working directory
log-level info Log verbosity level

Parameter repo-url takes the protocol in consideration, therefore you can use a local Git repository with --repo-url=file://path/to/local/repo, for instance.

In order to best performance results, consider always using parameter working-dir as a tmpfs. With docker/podman, a tmpfs volume can be informed directly on running a container.

Example

As a real world example, serve the last 200 commits of charts repository, with:

docker run \
    --publish="8080:8080" \
    --tmpfs=/var/lib/chartflow \
    quay.io/kubesmith/chartflow:latest \
        --clone-depth=200

Now, add chartflow as a chart repository:

helm repo add cs http://127.0.0.1:8080

And then, you're able to search for grafana in the repository. Notice the cs/grafana where cs is the local name of chartflow based repository:

$ helm search repo cs/grafana
NAME      	CHART VERSION	APP VERSION	DESCRIPTION
cs/grafana	5.0.6        	6.6.2      	The leading tool for querying and visualizing t..

You can also search for --devel (development) versions:

$ helm search repo cs/grafana --versions --devel
NAME      	CHART VERSION                                     	APP VERSION	DESCRIPTION
[...]
cs/grafana	2.1.0-revert-10682-master-f0cd0f9f                	5.4.3      	The leading tool for querying and visualizing t...
cs/grafana	2.0.2-revert-10682-master-fa4468c8                	5.4.3      	The leading tool for querying and visualizing t...
cs/grafana	1.14.1-update-owners-d09fd18b                     	5.2.3      	The leading tool for querying and visualizing t...
cs/grafana	1.14.0-update-owners-2496eaf3                     	5.2.2      	The leading tool for querying and visualizing t...

Therefore, automatically chartflow is displaying branches as development version of your Helm-Charts, you can reach them as, for instance:

helm install grafana cs/grafana:2.0.2-revert-10682-master-fa4468c8

Semantic Versioning

Git repository tree is traversed commit-by-commit starting from latest. On traversing the commits chartflow will identify from which branch each change is coming from, and with this information publish stable and development versions of Helm-Charts, adding semantic versioning representation understood by Helm.

Chart versions are ultimately defined by regular Chart.yaml, however it's enriched when change is located in a branch other than master. In master the exact version present in Chart.yaml is published, while in other branches is enriched with the branch name and short commit identifier.

For instance, lets see how grafana version 0.0.1 would be represented from a Git repository having master a change branches, where change is ahead of master, but grafana version is still the same in Chart.yaml.

Git Reference Name Version
master (or HEAD) 0.0.1
change (latest commit) 0.0.1-change
change (previous commits) 0.0.1-change-12345678

Endpoints

In order to behave as a Helm-Charts repository, chartflow exposes the following endpoints.

/index.yaml

Dynamically render index.yaml payload, representing actual Git repository data as Helm-Charts repository index. Helm clients are frequently downloading this payload in order to check which charts and versions are available in the repository.

/chart/:name/:version

Also generated dynamically, this endpoint exposes the "tarball" presenting the chart name (:name) and version (:version).

Contributing

All development dependencies are located at [Dockerfile.dev][./Dockerfile.dev]. To use it, run:

# build development image
make devcontainer-image
# start shell
make devcontainer-run
# execute an arbitrary command
make devcontainer-run DEVCONTAINER_ARGS='make test'

For Visual Studio Code users, install Remote Containers extension, and re-open the project in the container. Please consider [.devcontainer.json][.devcontainer.json] for details.

Building and Testing

Building the application requires libgit2 installed, please consider Dockerfile.dev for details. In order to build the project run:

make

And in order to run automated tests, execute:

make test

Additionally, consider .editorconfig file for code standards, and .travis.yml for the continuous integration steps.

APP ?= chartflow
MODULE = $(subst -,,$(APP))
IMAGE = quay.io/kubesmith/$(APP)
IMAGE_TAG = $(IMAGE):latest
IMAGE_DEV_TAG = $(IMAGE)-dev:latest
OUTPUT_DIR ?= _output
DEVCONTAINER_ARGS ?=
RUN_ARGS ?= serve
TEST_ARGS ?=
WORKING_DIR ?= /var/tmp/chartflow
COMMON_FLAGS ?= -v -mod=vendor
CHARTS_REPO_ARCHIVE ?= test/charts-repo.tar.gz
CHARTS_REPO_DIR ?= $(OUTPUT_DIR)/charts-repo
TEST_TIMEOUT ?= 10m
TEST_FLAGS ?= -failfast -timeout=$(TEST_TIMEOUT)
UNIT_TEST_TARGET ?= ./cmd/... ./pkg/...
E2E_TEST_TARGET ?= ./test/e2e/...
COVERAGE_DIR ?= $(OUTPUT_DIR)/coverage
LIBGIT_VERSION ?=
LD_LIBRARY_PATH ?= /usr/local/lib
# all variables are exported to environment
.EXPORT_ALL_VARIABLES:
default: build
# initialize Go modules vendor directory
.PHONY: vendor
vendor:
@go mod vendor
# clean up build directory
.PHONY: clean
clean:
@rm -rf $(OUTPUT_DIR)
# compress local charts test repository
archive-charts-repo:
tar zcvpf $(CHARTS_REPO_ARCHIVE) $(CHARTS_REPO_DIR)
# uncompress test charts repository archive tarball
unarchive-charts-repo:
@rm -rf "$(CHARTS_REPO_DIR)"
@tar zxpf $(CHARTS_REPO_ARCHIVE)
# create build and coverage directories
.PHONY: prepare
prepare: unarchive-charts-repo
@mkdir -p $(COVERAGE_DIR) > /dev/null 2>&1 || true
# build application command-line
build: prepare vendor $(OUTPUT_DIR)/$(APP)
# application binary
$(OUTPUT_DIR)/$(APP):
go build $(COMMON_FLAGS) -o="$(OUTPUT_DIR)/$(APP)" cmd/$(MODULE)/*
# installs all development dependencies in the development container
devcontainer-deps:
./hack/fedora.sh
./hack/golang.sh
./hack/libgit2.sh
./hack/libgit2-devel.sh
./hack/yum-clean-up.sh
./hack/helm.sh
# build devcontainer image
devcontainer-image:
docker build --tag="$(IMAGE_DEV_TAG)" --file="Dockerfile.dev" .
# execute devcontainer mounting local project directory
devcontainer-run:
./hack/devcontainer-run.sh
# start a bash shell in devcontainer
devcontainer-exec:
@docker exec --interactive --tty --workdir="/workspaces/$(APP)" $(APP) bash
# installs final application image dependencies
image-deps:
./hack/fedora.sh
./hack/libgit2.sh
./hack/yum-clean-up.sh
# build container image with Docker
image:
docker build --tag="$(IMAGE_TAG)" .
# execute "go run" against cmd
run:
@test -d $(WORKING_DIR) && rm -rf $(WORKING_DIR)
@mkdir $(WORKING_DIR)
go run $(COMMON_FLAGS) cmd/$(MODULE)/* $(RUN_ARGS) --working-dir=$(WORKING_DIR)
# running all test targets
test: test-unit test-e2e
# run unit tests
.PHONY: test-unit
test-unit: prepare
go test \
$(COMMON_FLAGS) \
$(TEST_FLAGS) \
-coverprofile=$(COVERAGE_DIR)/coverage-unit.txt \
$(TEST_ARGS) \
$(UNIT_TEST_TARGET) \
# run end-to-end tests
.PHONY: test-e2e
test-e2e: prepare
go test \
$(COMMON_FLAGS) \
$(TEST_FLAGS) \
-coverprofile=$(COVERAGE_DIR)/coverage-e2e.txt \
$(E2E_TEST_TARGET)
.PHONY: lint
lint:
@golangci-lint run
.PHONY: libgit2
libgit2:
./hack/ubuntu/libgit2.sh
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment