How to deploy Functionbeat on AWS

These notes are supplementing my another guide about Minimal Elastic monitoring stack

Forward Kibana port

We need session-manager plugin and AWS CLI packages installed:

$ aws ssm start-session --profile personal --target i-xyz --document-name AWS-StartPortForwardingSession --parameters "localPortNumber=8080,portNumber=8080" --region eu-west-1

Make sure you have valid AWS STS credentials behind personal profile

Define security model

We will follow basic yet effective security model described in here

https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html

Create writer role

https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-put-role.html

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.stack.home/_security/role/beats_writer -H 'Content-Type: application/json' -d'
{
  "cluster": [
    "monitor",
    "read_ilm"
  ],
  "indices": [
    {
      "names": [
        "functionbeat-*"
      ],
      "privileges": [
        "create_doc",
        "create_index",
        "view_index_metadata"
      ],
      "field_security": {
        "grant": [
          "*"
        ],
        "except": [ ]
      },
      "allow_restricted_indices": false
    }
  ],
  "applications": [ ],
  "run_as": [ ],
  "metadata": { },
  "transient_metadata": {
    "enabled": true
  }
}
'

Create writer user

https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-put-user.html

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.stack.home/_security/user/beats_writer -H 'Content-Type: application/json' -d'
{
  "username": "beats_writer",
  "password": "PleaseChangeMe",
  "roles": [
    "beats_writer",
    "remote_monitoring_agent",
    "remote_monitoring_collector"
  ],
  "full_name": "",
  "email": "",
  "metadata": { },
  "enabled": true
}
'

Create ilm policy

https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.stack.home/_ilm/policy/functionbeat-7.15.1 -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_size": "3gb",
            "max_primary_shard_size": "3gb",
            "max_age": "2d"
          },
          "set_priority": {
            "priority": 100
          },
          "shrink": {
            "number_of_shards": 1
          }
        }
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {
            "delete_searchable_snapshot": true
          }
        }
      }
    }
  }
}
'

Prepare index

https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

Initialize index

$ curl -sk -u elastic:PleaseChangeMe -XPUT "https://elastic.stack.home/%3Cfunctionbeat-7.15.1-%7Bnow%2Fd%7D-000001%3E"

Set settings

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.stack.home/_aliases/functionbeat-7.15.1 -H 'Content-Type: application/json' -d'
{
  "actions": [
    {
      "add": {
        "index": "functionbeat-7.15.1-2021.11.03-000001",
        "alias": "functionbeat-7.15.1",
        "is_write_index": true
      }
    }
  ]
}
'

Alternatively, set all at once

https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html

$ curl -sk -u elastic:PleaseChangeMe -X PUT "https://elastic.stack.home/%3Cfunctionbeat-7.15.1-%7Bnow%2Fd%7D-000001%3E" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index.lifecycle.name": "functionbeat-7.15.1",
    "index.lifecycle.rollover_alias": "functionbeat-7.15.1"
  },
  "aliases": {
    "functionbeat-7.15.1": {
      "is_write_index": true
    }
  }
}
'

Install functionbeat

Follow installation guide

https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-installation-configuration.html

You can use EC2 instance with instance-profile and associated functionbeat-deployment-role IAM role

functionbeat-deployment-role (ec2 role)

AmazonSSMManagedInstanceCore
https://www.elastic.co/guide/en/beats/functionbeat/current/iam-permissions.html
"ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVpcs"

We also need to create functionbeat-lambda-execution-role Lambda IAM role.

functionbeat-lambda-execution-role

CloudWatchReadOnlyAccess
AWSLambdaVPCAccessExecutionRole

Prepare configuration file

https://www.elastic.co/guide/en/beats/functionbeat/current/configuration-functionbeat-options.html

https://www.elastic.co/guide/en/beats/functionbeat/current/ilm.html

https://www.elastic.co/guide/en/beats/functionbeat/current/configuration-template.html

functionbeat:
  provider.aws:
    endpoint: "s3.amazonaws.com"
    region: "eu-west-1"
    deploy_bucket: "functionbeat-deploy-random-bucket"
    functions:
    - name: cloudwatch-dummy
      enabled: true
      type: cloudwatch_logs
      description: "lambda function for cloudwatch logs"
      memory_size: 128MiB
      role: arn:aws:iam::123456789012:role/functionbeat-lambda-execution-role
      virtual_private_cloud:
        security_group_ids: ["sg-xyz"]
        subnet_ids: ["subnet-xyz", "subnet-zyx"]
      triggers:
      - log_group_name: /aws/lambda/source-log-group

setup:
  dashboards.enabled: false
  ilm:
    enabled: true
    rollover_alias: "functionbeat-7.15.1"
    pattern: "{now/d}-000001"
    policy_name: "functionbeat-7.15.1"
    check_exists: false
    overwrite: false
  template:
    enabled: false
    overwrite: false
    settings:
      index.number_of_shards: 1
      index.number_of_replicas: 1

processors:
- add_host_metadata: ~
- add_cloud_metadata:
    providers: ["aws"]
- add_fields:
    target: "project"
    fields:
      team:
        name: "abc"
      component:
        name: "abc"

logging:
  level: info
  selectors: ["*"]

output.elasticsearch:
  hosts: ["https://elastic.stack.home"]
  protocol: "https"
  username: "beats_writer"
  password: "PleaseChangeMe"
  ssl:
    certificate_authorities: ["elasticsearch-ca.pem"]
    verification_mode: "certificate"

Download certificate

$ echo -n | openssl s_client -connect elastic.stack.home -servername elastic.stack.home 2>/dev/null | openssl x509 > elasticsearch-ca.pem

Verify configuration

$ ./functionbeat test config
$ ./functionbeat test output

Import index template

https://www.elastic.co/guide/en/beats/functionbeat/current/functionbeat-template.html

$ ./functionbeat export template > functionbeat.index.template.json

adjust aliases, check index parameters (policy, names, etc.) in the index template file

{
  "index_patterns": [
    "functionbeat-*"
  ],
  "settings": {
    "index": {
      "lifecycle": {
        "name": "functionbeat-7.15.1",
        "rollover_alias": "functionbeat-7.15.1"
      },
      "number_of_replicas": 1,
      "number_of_shards": 1,
    }
  },
  "aliases": { }
}

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.stack.home/_template/functionbeat-7.15.1 -H 'Content-Type: application/json' [email protected]

Run deployment

Change ssl configuration (functionbeat deployment tool doesn't inject CA certificate file into the lambda function zip package)

  ssl:
    verification_mode: "none"

$ ./functionbeat setup -e
$ ./functionbeat -v -e -d "*" deploy cloudwatch-dummy

To remove deployment

$ ./functionbeat -v -e -d "*" remove cloudwatch-provisioner

Extra: configure watermarks

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.cluster.home/_cluster/settings -H 'Content-Type: application/json' -d'
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "10gb",
    "cluster.routing.allocation.disk.watermark.high": "5gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "1gb",
    "cluster.info.update.interval": "1m"
  }
}
'

$ curl -sk -u elastic:PleaseChangeMe -XPUT https://elastic.cluster.home/functionbeat-7.15.1/_settings -H 'Content-Type: application/json' -d'
{
  "index.blocks.read_only_allow_delete": null
}
'

avoidik/README.md