System Prerequisites

Elasticsearch relies on mmapfs for index storage. The default OS limits on mmap counts are typically insufficient and can trigger out-of-memory exceptions. To permanently increase this limit, modify /etc/sysctl.conf by adjusting the vm.max_map_count parameter. After rebooting, verify the change by executing sysctl vm.max_map_count.

Containerized Deployment

Pull the official image:

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.1.0

Establish an isolated network for the stack components:

docker network create es-net

Launch the Elasticsearch container. The initial startup generates the elastic user password and a Kibana enrollment token, which are printed to the console output:

docker run --name es-node1 --net es-net -p 9200:9200 -p 9300:9300 -it docker.elastic.co/elasticsearch/elasticsearch:8.1.0

Extract the generated TLS certificate to your local host:

docker cp es-node1:/usr/share/elasticsearch/config/certs/http_ca.crt .

Validate the connection using the extracted certificate and the auto-generated credentials:

curl --cacert http_ca.crt -u elastic https://localhost:9200

Elastic Agent and Fleet Overview

Starting with version 7.14, Elastic Agent became generally available, and 7.16 introduced Fleet. Fleet serves as a control plane within Kibana, pushing configuration policies to a Fleet Server, which utilizes Elasticsearch as its backend data store. Elastic Agents deployed on edge hosts pull these policies from the Fleet Server, replacing the need for manual Logstash and Beats configurations.

RPM Installation: Elasticsearch

Download the package:

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.16.2-x86_64.rpm

Install the RPM:

sudo rpm -ivh elasticsearch-7.16.2-x86_64.rpm

The default systemd service unit uses Type=notify, which frequently causes startup timeouts. Changing it to Type=simple resolves this issue. Modify /usr/lib/systemd/system/elasticsearch.service:

[Service]
Type=simple
# Type=notify has been overridden
RuntimeDirectory=elasticsearch
PrivateTmp=true
Environment=ES_HOME=/usr/share/elasticsearch
Environment=ES_PATH_CONF=/etc/elasticsearch
Environment=PID_DIR=/var/run/elasticsearch
Environment=ES_SD_NOTIFY=true
EnvironmentFile=-/etc/sysconfig/elasticsearch

WorkingDirectory=/usr/share/elasticsearch
User=elasticsearch
Group=elasticsearch

ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet
StandardOutput=journal
StandardError=inherit

LimitNOFILE=65535
LimitNPROC=4096
LimitAS=infinity
LimitFSIZE=infinity
TimeoutStopSec=0
KillSignal=SIGTERM
KillMode=process
SendSIGKILL=no
SuccessExitStatus=143

To enable external access, update /etc/elasticsearch/elasticsearch.yml:

network.host: 10.0.0.5
discovery.seed_hosts: ["10.0.0.5"]

Verify the instance is running:

curl http://10.0.0.5:9200

RPM Installation: Kibana

Import the Elastic GPG key and configure the repository:

rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

cat <<EOF >/etc/yum.repos.d/kibana.repo
[kibana-7.x]
name=Kibana repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Install the package:

sudo yum install kibana

The RPM setup automatically generates the systemd unit file at /etc/systemd/system/kibana.service. Point Kibana to the Elasticsearch node by editing /etc/kibana/kibana.yml:

elasticsearch.hosts: ["http://10.0.0.5:9200"]

Enable and start the service:

systemctl daemon-reload
systemctl enable kibana
systemctl start kibana

RPM Installation: Filebeat

Download and install the RPM:

curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.16.2-x86_64.rpm
sudo rpm -vi filebeat-7.16.2-x86_64.rpm

Create a dedicated service account, as the default systemd unit lacks user specification:

groupadd filebeat
useradd -r -g filebeat -s /sbin/nologin filebeat

Append user and group directives to the systemd unit at /usr/lib/systemd/system/filebeat.service:

[Service]
User=filebeat
Group=filebeat
Environment="GODEBUG='madvdontneed=1'"
Environment="BEAT_LOG_OPTS="
Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat.yml"
Environment="BEAT_PATH_OPTS=--path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat"
ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
Restart=always

Assign appropriate ownership to the required directories:

chown -R filebeat:filebeat /etc/filebeat/filebeat.yml
chown -R filebeat:filebeat /etc/filebeat/
chown -R filebeat:filebeat /var/lib/filebeat/
chown -R filebeat:filebeat /usr/share/filebeat/
mkdir -p /var/log/filebeat
chown -R filebeat:filebeat /var/log/filebeat

Filebeat Configuration and Keystore

Validate the configuration syntax:

sudo -u filebeat filebeat test config -c /etc/filebeat/filebeat.yml

Utilize the Keystore to securely manage Elasticsearch credentials. The keystore must be created by the user running the Filebeat process:

sudo -u filebeat filebeat keystore create
sudo -u filebeat filebeat keystore add ES_PASS

Update /etc/filebeat/filebeat.yml with the output destination and reference the stored secret:

output.elasticsearch:
  hosts: ["10.0.0.5:9200"]
  username: "elastic"
  password: "${ES_PASS}"

setup.kibana:
  host: "10.0.0.5:5601"

Enable the required data collection modules:

sudo -u filebeat filebeat modules enable system nginx mysql

Define the log paths within the module configuration, such as /etc/filebeat/modules.d/nginx.yml. Ensure strict YAML syntax, including spaces after colons, to avoid parsing errors:

var.paths: [/var/log/nginx/access.log*]

Load the predefined index templates and dashboards into Elasticsearch and Kibana:

sudo -u filebeat filebeat setup -e

Path Configuration Pitfall

A common issue during RPM setup involves the conflict between the wrapper script /bin/filebeat and the raw binary /usr/share/filebeat/bin/filebeat. The RPM package installs a wrapper at /bin/filebeat that properly sets the default path variables. However, this wrapper is omitted from the official directory layout documentation, leading users to invoke the raw binary directly.

When the raw binary is executed without explicit path flags, it defaults path.home to its own location, resulting in module loading failures such as: Not loading modules. Module directory not found: /usr/share/filebeat/bin/module.

To resolve this, always invoke the wrapper script for CLI operations, or explicitly define the paths within /etc/filebeat/filebeat.yml:

path.home: /usr/share/filebeat
path.config: /etc/filebeat
path.data: /var/lib/filebeat
path.logs: /var/log/filebeat

If explicit paths are defined in the configuration file, ensure that identical flags are passed during keystore operations to maintain path consistency and prevent credential resolution failures.

Elasticsearch Maintenance Operations

Retrieve a list of all indices:

curl -u elastic:password -XGET '10.0.0.5:9200/_cat/indices?v&pretty'

Adjust the replica count for a specific index:

curl -u elastic:password -H "Content-Type: application/json" -XPUT '10.0.0.5:9200/app-logs-7.16.2-2022.01.01-000001/_settings' -d '{"number_of_replicas":0}'

Insert a test document into a sample index:

curl -u elastic:password -H "Content-Type: application/json" -XPOST '10.0.0.5:9200/sample_index/_doc' -d '{"sample_field": 1}'

Query the sample index:

curl -u elastic:password -XGET '10.0.0.5:9200/sample_index/_search'

Resolving Field Limit Errors

When an index exceeds the default 1000 field mapping threshold, indexing operations will fail with a Limit of total fields [1000] in index [indexname] has been exceeded exception. Increase the allowable field count via the index settings API:

curl -XPUT 'http://10.0.0.5:9200/indexname/_settings' -H "Content-Type: application/json" -d '{"index.mapping.total_fields.limit": 10000}'