Elastic Beats Development 101

This is a short guide to get up and building Elastic Beats on a new Linux host.

Start a VM

This uses Google Compute Engine (GCE) to start an Ubuntu 20.04 virtual machine. You can use other versions of Linux or different virtualization platforms (or no virtualization), but those are not guaranteed to work with the commands here.

 gcloud auth login
 
 gcloud config set project elastic-beats
 
 gcloud compute instances create $USER-beats-dev \
  --image-project=ubuntu-os-cloud \
  --image-family ubuntu-2004-lts \
  --zone us-central1-a \
  --machine-type=n1-standard-1 \
  --boot-disk-size=100GB
  
 gcloud compute ssh --zone=us-central1-a $USER-beats-dev

NOTE: To clean up this resource afterwards use gcloud compute instances delete --zone us-central1-a $USER-beats-dev.

Install Deps

sudo apt-get update
sudo apt-get install -y vim docker.io make gcc python3-pip python3-venv git libpcap-dev libsystemd-dev librpm-dev

sudo curl -L "https://github.com/andrewkroh/gvm/releases/download/v0.4.1/gvm-linux-amd64" -o /usr/local/bin/gvm
sudo chmod +x /usr/local/bin/gvm

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

sudo usermod -a -G docker $USER

Clone Beats

mkdir -p ~/go/src/github.com/elastic
git clone https://github.com/elastic/beats.git ~/go/src/github.com/elastic/beats

Update .bash_aliases

This configures your bash shell to setup your Go development envionrment on login.

echo 'eval "$(gvm $(cat ~/go/src/github.com/elastic/beats/.go-version))"' >> ~/.bash_aliases
echo 'export PATH=$(go env GOPATH)/bin:${PATH}' >> ~/.bash_aliases
source ~/.bash_aliases

Install Mage

cd ~/go/src/github.com/elastic/beats
make mage

Listing Build Targets and Getting Usage Details

The main build tool is Mage. You can use mage -l to list the available build targets. And mage -h <target> to get detailed usage information about a target.

cd filebeat # or any Beat
mage -l
mage -h exportDashboard

Update Generated Content

mage update

Build

You can use mage -v <targets> to enable verbose output from Mage.

mage build

Unit Test

mage unitTest

Integration Test

mage integTest

Executing Multiple Targets

If you are going to execute multiple targets it's more efficient to run them in a single command.

mage update build integTest

Packaging

mage package

You can control the target platforms with the PLATFORMS environment variable. Use SNAPSHOT=true to build a snapshot. And use VERSION_QUALIFIER=rc1 to add a qualifier to version. Some examples:

PLATFORMS='linux/amd64' SNAPSHOT=true mage package

PLATFORMS='linux/armv7' mage package

PLATFORMS='windows darwin' mage package

Packages are written to build/distributions.

andrewkroh/elastic-beat-development-101.md