Elastic Beats Development 101

This is a short guide to get up and building Elastic Beats on a new Linux host.

Start a VM

This uses Google Compute Engine (GCE) to start an Ubuntu 20.04 virtual machine. You can use other versions of Linux or different virtualization platforms (or no virtualization), but those are not guaranteed to work with the commands here.

 gcloud auth login
 
 gcloud config set project elastic-beats
 
 gcloud compute instances create $USER-beats-dev \
  --image-project=ubuntu-os-cloud \
  --image-family ubuntu-2004-lts \
  --zone us-central1-a \
  --machine-type=n1-standard-1 \
  --boot-disk-size=100GB
  
 gcloud compute ssh --zone=us-central1-a $USER-beats-dev

NOTE: To clean up this resource afterwards use gcloud compute instances delete --zone us-central1-a $USER-beats-dev.

Install Deps

sudo apt-get update
sudo apt-get install -y vim docker.io make gcc python3-pip python3-venv git libpcap-dev libsystemd-dev librpm-dev

sudo curl -L "https://github.com/andrewkroh/gvm/releases/download/v0.4.1/gvm-linux-amd64" -o /usr/local/bin/gvm
sudo chmod +x /usr/local/bin/gvm

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

sudo usermod -a -G docker $USER

Clone Beats

mkdir -p ~/go/src/github.com/elastic
git clone https://github.com/elastic/beats.git ~/go/src/github.com/elastic/beats

Update .bash_aliases

This configures your bash shell to setup your Go development envionrment on login.

echo 'eval "$(gvm $(cat ~/go/src/github.com/elastic/beats/.go-version))"' >> ~/.bash_aliases
echo 'export PATH=$(go env GOPATH)/bin:${PATH}' >> ~/.bash_aliases
source ~/.bash_aliases

Install Mage

cd ~/go/src/github.com/elastic/beats
make mage

Listing Build Targets and Getting Usage Details

The main build tool is Mage. You can use mage -l to list the available build targets. And mage -h <target> to get detailed usage information about a target.

cd filebeat # or any Beat
mage -l
mage -h exportDashboard

Update Generated Content

mage update

Build

You can use mage -v <targets> to enable verbose output from Mage.

mage build

Unit Test

mage unitTest

Integration Test

mage integTest

Executing Multiple Targets

If you are going to execute multiple targets it's more efficient to run them in a single command.

mage update build integTest

Packaging

mage package

You can control the target platforms with the PLATFORMS environment variable. Use SNAPSHOT=true to build a snapshot. And use VERSION_QUALIFIER=rc1 to add a qualifier to version. Some examples:

PLATFORMS='linux/amd64' SNAPSHOT=true mage package

PLATFORMS='linux/armv7' mage package

PLATFORMS='windows darwin' mage package

Packages are written to build/distributions.

Status: Image is up to date for docker.elastic.co/beats-dev/golang-crossbuild:1.17.6-main-debian8 >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] >> Building using: cmd='build/mage-linux-amd64 buildGoDaemon', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64] /tmp/ccy6qz4n.o: In function `main': god.c:(.text+0x2a6): warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking god.c:(.text+0x23e): warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking W: Failed to fetch http://archive.debian.org/debian/dists/jessie/InRelease W: Failed to fetch http://archive.debian.org/debian/dists/jessie/Release.gpg Unable to connect to archive.debian.org:http: [IP: 193.62.202.28 80] W: Some index files failed to download. They have been ignored, or old ones used instead. >> package: Building filebeat-oss type=docker for platform=linux/amd64 >> package: Building filebeat-oss type=tar.gz for platform=linux/amd64 >> package: Building filebeat-oss type=deb for platform=linux/amd64

useradd: UID 0 is not unique >> package: Building filebeat-oss type=rpm for platform=linux/amd64 useradd: UID 0 is not unique The command '/bin/sh -c for iter in {1..10}; do apt-get update -y && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils && apt-get clean all && exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10; done; (exit $exit_code)' returned a non-zero code: 100 >> Building docker images again (after 10 s) The command '/bin/sh -c for iter in {1..10}; do apt-get update -y && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils && apt-get clean all && exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10; done; (exit $exit_code)' returned a non-zero code: 100 >> Building docker images again (after 10 s) The command '/bin/sh -c for iter in {1..10}; do apt-get update -y && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils && apt-get clean all && exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10; done; (exit $exit_code)' returned a non-zero code: 100 >> Building docker images again (after 10 s) The command '/bin/sh -c for iter in {1..10}; do apt-get update -y && DEBIAN_FRONTEND=noninteractive apt-get install --no-install-recommends --yes ca-certificates curl gawk libcap2-bin xz-utils && apt-get clean all && exit_code=0 && break || exit_code=$? && echo "apt-get error: retry $iter in 10s" && sleep 10; done; (exit $exit_code)' returned a non-zero code: 100 package ran for 17m10.589291866s Error: failed building filebeat-oss type=docker for platform=linux/amd64: failed to build docker: running "docker build -t docker.elastic.co/beats/filebeat-oss:8.2.0-SNAPSHOT build/package/filebeat-oss/filebeat-oss-linux-amd64.docker/docker-build" failed with exit code 100

andrewkroh/elastic-beat-development-101.md