elastic-pi.md

Installing elastic beats on Raspberry Pi

At time of writing elastic.co does not provide ARM builds for raspberry. This tutorial describes how to compile e.g. filebeat and run in on Raspberry Pi3.

Prerequisites

You'll need Go (>1.8) and Pyhton with virtualenv.

Go 1.8

Raspian/Debian stretch comes with Go 1.7. To get 1.8 you'll need to add the next version buster to the package sources. Make sure your /etc/apt/preferences looks like this:

Package: *
Pin: release n=stretch
Pin-Priority: 500

Package: *
Pin: release n=buster
Pin-Priority: 10

Then update the package list:

sudo apt update

And install from buster:

sudo apt install -t buster golang

Python

Python 2.x should come preinstalled with Raspbian but you're likely missing virtualenv:

sudo pip install virtualenv

Memory

Compiling with Go, even using a single CPU core can take a lot of memory. Update /etc/dphys-swapfile to 1024MB instead of the default 100MB swap memory and enable it:

sudo nano /etc/dphys-swapfile
sudo dphys-swapfile setup
sudo dphys-swapfile swapon

Building

Get the sources

Set the root path for Go package management:

export GOPATH=~/go

Get sources:

go get github.com/elastic/beats

Goto desired beats folder:

cd ~/go/src/github.com/elastic/beats/filebeat/

Select desired version, should match your elasticsearch backend:

git checkout 6.0

Build binary

Run the go compiler:

GOPATH=~/go make

This wil output the executable filebeat in the current directory, verify by running:

./filebeat -v -e

where -e will output errors to the console instead of syslog. filebeat will now complain about missing config file filebeat.yml.

Copy filebeat.default.yml and modify as required.

Configuration templates

The fields.yml is required to configure the index. To build run:

make python-env
make fields
mv _meta/fields.generated.yml ./fields.yml

Build the Kibana template:

make kibana
mv _meta/kibana .

Now run filebeat setup:

./filebeat setup -v -e

Add-ons

For some beats plugins the elasticsearch core can be supplied with addons. Assuming you're running the elastic server in a docker image named elasticsearch install addons like this:

sudo docker exec -it elasticsearch bash
cd /opt/elasticsearch/
bin/elasticsearch-plugin install ingest-user-agent
bin/elasticsearch-plugin install ingest-geoip

I've whipped up a small docker that does the compile job and packages everything similar to the distributed official images, see https://github.com/andig/beats4pi

Awesome, I'll try it on Asus tinker board tomorrow if it works there :D this monster is hard to manage now https://imgur.com/gHeUok4 hahahaha

@andig
I have successfully made filebeat binary that seems to be working ( did not send data to elastisearch server yet) , there is 1.9 golang in buster now. I have added also swap but when I have compiled metricbeat it never go over 1G of RAM ( our of 2GB on Asus Tinker Board ) so that is not needed I think in this SBC

Use make update to generate configuration files and mapping template as opposed to make fields. Otherwise the mapping template is incomplete.

Thanks man, this is golden!

Compile worked with go1.10.5 binaries downloaded. Great gist!

This worked like a charm - many thanks for a great write-up!

I am getting stuck here:

root@raspberrypi:~/go/src/github.com/elastic/beats/filebeat# GOPATH=~/go make
Installing mage v1.8.0 from vendor dir.
bash: mage: command not found
../dev-tools/make/mage.mk:9: recipe for target 'mage' failed
make: [mage] Error 127 (ignored)

any ideas?

Hi,
Just ran thru this, found later Beats versions (7.x+) need Go 1.11 or better. If running on Raspian 9, you'll also need to add the Buster repos to your sources.list
Thanks a ton for this, massive massive help!!!!

Please update, keep getting an error for outdated Go

You mean on the docker image? Feel free ro open a PR!

Thanks you so much!!!!! I have been trying to compile the filebeat source for hours and tis is the first tutorial that works for me (raspberry pi4 - arm7)

I could get this to run on FIlebeat 7.11, but 6.0 worked fine. I had to install go version 1.8 by dowloading the archived go1.8.linux-armv6l.tar.gz from Golang.org's download page.

Has anyone gotten this to work with the current 7.11 release of Beats, and the current version of go? I am on a Pi 4, which has plenty of resources, but the current versions threw a variety of compile errors related to libraries not being found, and as I don't know go, that was a blocker for me.