Offline ElastiFlow Installation on CentOS 8

Required Files

Download the following files and copy to the CentOS server:

Tune the Linux Kernel

1. Add the parameters required by Elasticsearch

Create the following file:

sudo vi /etc/sysctl.d/70-elasticsearch.conf

Add the following contents to this file:

vm.max_map_count=262144

2. Tune network parameters for better throughput

Create the following file:

sudo vi /etc/sysctl.d/60-net.conf

Add the following contents to this file:

net.core.netdev_max_backlog=4096
net.core.rmem_default=262144
net.core.rmem_max=67108864
net.ipv4.udp_rmem_min=131072
net.ipv4.udp_mem=2097152 4194304 8388608

3. Reboot

Reboot the system for these changes to take effect.

Disable the Firewall

The easiest way to get started is to disable the firewall. Alternatively the firewall can be configured to allow access to any required ports.

sudo systemctl stop firewalld
sudo systemctl disable firewalld

Install Elasticsearch

1. Install the Elasticsearch RPM

sudo yum install -y elasticsearch-7.10.2-x86_64.rpm

2. Configure JVM Heap Size

If a JVM is started with unequal initial and max heap sizes, it may pause as the JVM heap is resized during system usage. For this reason it’s best to start the JVM with the initial and maximum heap sizes set to equal values.

Edit /etc/elasticsearch/jvm.options and set -Xms and -Xmx to about one third of the system memory, but do not exceed 31g. For example:

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms12g
-Xmx12g

3. Increase System Limits

You should specify system limits in a systemd configuration file for the elasticsearch service.

sudo mkdir /etc/systemd/system/elasticsearch.service.d
sudo vi /etc/systemd/system/elasticsearch.service.d/elasticsearch.conf

Add the following contents to this file:

[Service]
LimitNOFILE=131072
LimitNPROC=8192
LimitMEMLOCK=infinity
LimitFSIZE=infinity
LimitAS=infinity

4. Edit elasticsearch.yml

Edit the Elasticsearch configuration:

vi /etc/elasticsearch/elasticsearch.yml

Replace the contents of the file, editing as necessary for your environment:

cluster.name: elastiflow

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 0.0.0.0
http.port: 9200

discovery.type: 'single-node'

indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000

action.destructive_requires_name: 'true'

reindex.remote.whitelist: '*:*'

xpack.monitoring.enabled: 'true'
xpack.monitoring.collection.enabled: 'true'
xpack.monitoring.collection.interval: 30s

xpack.security.enabled: 'true'
xpack.security.audit.enabled: 'false'

node.ml: 'false'
xpack.ml.enabled: 'false'

xpack.watcher.enabled: 'false'

xpack.ilm.enabled: 'true'

xpack.sql.enabled: 'true'

5. Enable and Start Elasticsearch

Execute the following commands:

sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

Confirm Elasticsearch started successfully by executing:

sudo systemctl status elasticsearch

6. Set Passwords for Elasticsearch Built-in Accounts

Execute the following command for to setup passwords:

sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive

The following will be displayed:

Initiating the setup of passwords for reserved users elastic,apm_system,kibana,kibana_system,logstash_system,beats_system,remote_monitoring_user.
You will be prompted to enter passwords as the process progresses.
Please confirm that you would like to continue [y/N]

Answer y, then enter and confirm passwords for the built-in Elasticsearch accounts.

7. Verify Elasticsearch

Ensure that the Elasticsearch REST API is available by running the following:

curl -XGET "http://elastic:[email protected]:9200"

The output should be:

{
  "name" : "NAME_OF_YOUR_HOST",
  "cluster_name" : "elastiflow",
  "cluster_uuid" : "tCeuQg-QSnUUIDtE2pYnRA",
  "version" : {
    "number" : "7.10.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "747e1cc71def077253878a59143c1f785afa92b9",
    "build_date" : "2021-01-13T00:42:12.435326Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Install Kibana

1. Install the Kibana RPM

sudo yum install -y kibana-7.10.2-x86_64.rpm

2. Edit kibana.yml

Edit the Kibana configuration:

vi /etc/kibana/kibana.yml

Replace the contents of the file, editing as necessary (especially elasticsearch.password) for your environment:

telemetry.enabled: false
telemetry.optIn: false
newsfeed.enabled: false

server.host: '0.0.0.0'
server.port: 5601

server.maxPayloadBytes: 8388608

elasticsearch.hosts: ['http://127.0.0.1:9200']
elasticsearch.username: 'kibana_system'
elasticsearch.password: 'PASSWORD'
elasticsearch.requestTimeout: 132000
elasticsearch.shardTimeout: 120000

console.enabled: true

kibana.defaultAppId: 'dashboard/4a608bc0-3d3e-11eb-bc2c-c5758316d788'

kibana.autocompleteTimeout: 2000
kibana.autocompleteTerminateAfter: 500000

xpack.maps.showMapVisualizationTypes: true

xpack.security.enabled: true
xpack.security.audit.enabled: false

monitoring.enabled: true
monitoring.kibana.collection.enabled: true
monitoring.kibana.collection.interval: 30000

monitoring.ui.enabled: true
monitoring.ui.min_interval_seconds: 20

xpack.apm.enabled: false
xpack.apm.ui.enabled: false
xpack.ccr.enabled: false
xpack.cloud.enabled: false
xpack.code.enabled: false
xpack.fleet.enabled: false
xpack.graph.enabled: false
xpack.grokdebugger.enabled: false
xpack.infra.enabled: false
xpack.logstash.enabled: false
xpack.ml.enabled: false
xpack.remote_clusters.enabled: false
xpack.reporting.enabled: false
xpack.searchprofiler.enabled: false
xpack.siem.enabled: false
xpack.transform.enabled: false
xpack.upgrade_assistant.enabled: false
xpack.uptime.enabled: false
xpack.watcher.enabled: false

3. Enable and Start Kibana

Execute the following commands:

sudo systemctl daemon-reload
sudo systemctl enable kibana
sudo systemctl start kibana

Confirm Elasticsearch started successfully by executing:

sudo systemctl status kibana

You should now be able to access Kibana at http://IP_OF_KIBANA_HOST:5601.

4. Import Kibana Objects

Importing via the User Interface

To import the configuration, in Kibana go to Stack Management --> Saved Objects and click Import in the upper right corner.

A side-bar will appear. Again click Import at the top of the side-bar.

Select the file which you downloaded, and click the Import button at the bottom of the side-bar.

The configuration will be imported and you will see all of the imported objects.

Close the side-bar. You will also see all of the imported objects in the Saved Objects list.

Importing via the API

curl -XPOST "https://username:password@IPORHOSTOFKIBANA:5601/api/saved_objects/_import?overwrite=true" -k -H "kbn-xsrf: true" -H "securitytenant: global" --form [email protected]

5. Apply Recommended Kibana Advanced Settings

You may find that modifying a few of the Kibana advanced settings will produce a more user-friendly experience while using ElastiFlow™. These settings are made in Kibana, under Stack Management --> Kibana --> Advanced Settings.

Advanced Setting	Value	Why make the change?
filters:pinnedByDefault	`true`	Pinning a filter allows it to persist when you are changing dashbaords. This is very useful when drilling-down into something of interest and you want to change dashboards for a different perspective of the same data. This is the #1 setting we recommend changing.
defaultRoute	see description	If your primary or only use-case for Kibana is ElastiFlow, set this the URL path for the dashboard to which you which to load immediately after logging in, or when returning to "home". The format of this value is `/app/dashboards#/view/4a608bc0-3d3e-11eb-bc2c-c5758316d788`.
doc_table:highlight	`false`	There be a query performance penalty that comes with using the highlighting feature. As it isn't very useful for this use-case, it is better to just turn it off.
state:storeInSessionStorage	`true`	Kibana URLs can get pretty large. Especially when working with Vega visualizations. This will likely result in error messages for users of Internet Explorer. Using in-session storage will fix this issue for these users.
timepicker:timeDefaults	see below	The Time Picker Quick Range to use when Kibana is started without one.
timepicker:quickRanges	see below	The default options in the Time Picker are less than optimal, for most logging and monitoring use-cases. Fortunately Kibana now allows you to customize the time picker. Our recommended settings can be found below.
format:number:defaultPattern	`0,0.[00]`	Default numeral format for the "number" format.
format:percent:defaultPattern	`0,0.[00]%`	Default numeral format for the "percent" format.

Recommended Time Picker Time Defaults (timepicker:timeDefaults)

We find that the following Time Picker Time Default provides more useful views of the data for network flow related use-cases.

{
  "from": "now-1h/m",
  "to": "now"
}

Recommended Time Picker Quick Ranges (timepicker:quickRanges)

We find that the following set of Time Picker Quick Ranges provides more useful views of the data for network flow related use-cases.

[
  {
    "from": "now-15m/m",
    "to": "now/m",
    "display": "Last 15 minutes"
  },
  {
    "from": "now-30m/m",
    "to": "now/m",
    "display": "Last 30 minutes"
  },
  {
    "from": "now-1h/m",
    "to": "now/m",
    "display": "Last 1 hour"
  },
  {
    "from": "now-2h/m",
    "to": "now/m",
    "display": "Last 2 hours"
  },
  {
    "from": "now-4h/m",
    "to": "now/m",
    "display": "Last 4 hours"
  },
  {
    "from": "now-12h/m",
    "to": "now/m",
    "display": "Last 12 hours"
  },
  {
    "from": "now-24h/m",
    "to": "now/m",
    "display": "Last 24 hours"
  },
  {
    "from": "now-48h/m",
    "to": "now/m",
    "display": "Last 48 hours"
  },
  {
    "from": "now-7d/m",
    "to": "now/m",
    "display": "Last 7 days"
  },
  {
    "from": "now-30d/m",
    "to": "now/m",
    "display": "Last 30 days"
  },
  {
    "from": "now-60d/m",
    "to": "now/m",
    "display": "Last 60 days"
  },
  {
    "from": "now-90d/m",
    "to": "now/m",
    "display": "Last 90 days"
  },
  {
    "from": "now/d",
    "to": "now/d",
    "display": "Today"
  },
  {
    "from": "now/w",
    "to": "now/w",
    "display": "This week"
  },
  {
    "from": "now/M",
    "to": "now/M",
    "display": "This month"
  },
  {
    "from": "now/d",
    "to": "now",
    "display": "Today so far"
  },
  {
    "from": "now/w",
    "to": "now",
    "display": "Week to date"
  },
  {
    "from": "now/M",
    "to": "now",
    "display": "Month to date"
  }
]

Install libpcap-devel

The collector requires that libpcap-devel is installed.

1. Install the libpcap-devel RPM

sudo yum install -y libpcap-devel-1.9.1-4.el8.x86_64.rpm

Install the ElastiFlow Unified Flow Collector

1. Install the flow_collector RPM

sudo yum install -y flow-collector-5.0.2-1.x86_64.rpm

2. Configure the Flow Collector

The Unified Flow Collector will be installed to run as a daemon manged by systemd. Configuration of the collector is provided via environment variables and, depending on the enabled options, via various configuration files which by default are located within /etc/elastiflow.

To configure the environment variables, edit the file /etc/systemd/system/flowcoll.service.d/flowcoll.conf. For details on all of the configuration options, please refer to the Configuration Reference.

3. Enable and Start the Flow Collector

Execute the following commands:

sudo systemctl daemon-reload
sudo systemctl enable flowcoll
sudo systemctl start flowcoll

Confirm Elasticsearch started successfully by executing:

sudo systemctl status flowcoll

robcowart/elastiflow_offline_install_centos8.md

Offline ElastiFlow Installation on CentOS 8

Required Files

Tune the Linux Kernel

1. Add the parameters required by Elasticsearch

2. Tune network parameters for better throughput

3. Reboot

Disable the Firewall

Install Elasticsearch

1. Install the Elasticsearch RPM

2. Configure JVM Heap Size

3. Increase System Limits

4. Edit elasticsearch.yml

5. Enable and Start Elasticsearch

6. Set Passwords for Elasticsearch Built-in Accounts

7. Verify Elasticsearch

Install Kibana

1. Install the Kibana RPM

2. Edit kibana.yml

3. Enable and Start Kibana

4. Import Kibana Objects

Importing via the User Interface

Importing via the API

5. Apply Recommended Kibana Advanced Settings

Recommended Time Picker Time Defaults (timepicker:timeDefaults)

Recommended Time Picker Quick Ranges (timepicker:quickRanges)

Install libpcap-devel

1. Install the libpcap-devel RPM

Install the ElastiFlow Unified Flow Collector

1. Install the flow_collector RPM

2. Configure the Flow Collector

3. Enable and Start the Flow Collector