Deploying an Elasticsearch Cluster with Kibana: Installation Guide

Understanding Elasticsearch

ElasticSearch is a search server built on Apache Lucene that provides a REST API interface. As a highly scalable open-source full-text search and analytics engine, it enables rapid storage, search, and analysis of large datasets.

Key characteristics of Elasticsearch include: distributed architecture, high availability, asynchronous writes, multiple API options, and document-oriented storage.

Core concepts: near real-time performance, clusters, nodes (data storage units), indices, shards (index partitioning), and replicas (multiple copies of shards). Elasticsearch is used by major platforms like Wikipedia, Stack Overflow, and GitHub.

Elasticsearch Cluster Setup

Environment Requirements

ElasticSearch cluster installation requires JDK. This guide uses Elasticsearch version 6.5.4 with corresponding Kibana 6.5.4. Note that Kibana version should not be lower than Elasticsearch. JDK version required is 1.8.

Download links:

• ElasticSearch-6.5.4: https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.tar.gz
• Kibana-6.5.4: https://artifacts.elastic.co/downloads/kibana/kibana-6.5.4-linux-x86_64.tar.gz
• JDK1.8: http://www.oracle.com/technetwork/java/javase/downloads

ElasticSearch has several important node attributes: master node, data node, coordinating node, and ingest node. This guide focuses primarily on master and data nodes, with other attributes explained in the configuration section.

Linux System Configuration

Before installing Elasticsearch, adjust the Linux environment to prevent potential issues:

1. Adjust Memory Limits

Edit sysctl.conf file:

vim /etc/sysctl.conf

Add the following configuration at the end:

vm.max_map_count = 655360
vm.swappiness=1

Save and exit, then apply changes with:

sysctl -p

2. Modify Thread Count Limit

Edit the thread limit file (90-nproc.conf may vary by system):

vim /etc/security/limits.d/90-nproc.conf

Change:

soft nproc 2048

To:

soft nproc 4096

3. Adjust File Descriptor Limit

Edit limits.conf:

vim /etc/security/limits.conf

Add at the end:

hard nofile 65536
soft nofile 65536

4. Firewall Settings

For simplicity, this guide disables the firewall. In production environments, proper port configuration should be implemented instead.

CentOS 6:

service iptables status
service iptables stop
service iptables start
service iptables restart
chkconfig iptables off
chkconfig iptables on

CentOS 7:

systemctl stop firewalld.service

JDK Installation

1. File Preparation

Extract the downloaded JDK:

tar -xvf jdk-8u144-linux-x64.tar.gz

Move to /opt/java directory and rename:

mv jdk1.8.0_144 /opt/java
mv jdk1.8.0_144 jdk1.8

2. Environment Configuration

Check if JDK is already installed:

java -version

If installed but incompatible, remove it:

rpm -qa | grep java
rpm -e --nodeps "java-package-info"

After confirming removal, extract the JDK:

tar -xvf jdk-8u144-linux-x64.tar.gz
mv jdk1.8.0_144 /opt/java
mv jdk1.8.0_144 jdk1.8

Edit profile file:

vim /etc/profile

Add the following configuration:

export JAVA_HOME=/opt/java/jdk1.8
export JRE_HOME=/opt/java/jdk1.8/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$RE_HOME/lib
export PATH=.:${JAVA_HOME}/bin:$PATH

Apply changes:

source /etc/profile

Verify installation:

java -version

Elasticsearch Installation

1. File Preparation

Extract the downloaded Elasticsearch:

tar -xvf elasticsearch-6.5.4.tar.gz

Move to /opt/elk directory and rename:

mv elasticsearch-6.5.4 /opt/elk
mv elasticsearch-6.5.4 masternode

2. Configuration Setup

ElasticSearch requires non-root user operation. Create a dedicated user:

adduser elastic
chown -R elastic:elastic /opt/elk/masternode

Create data and log directories:

su elastic
mkdir /home/elk
mkdir /home/elk/masternode
mkdir /home/elk/masternode/data
mkdir /home/elk/masternode/logs
mkdir /home/elk/datanode1
mkdir /home/elk/datanode1/data
mkdir /home/elk/datanode1/logs

Master Node Configuration

Edit configuration files:

cd /opt/elk/
vim masternode/config/elasticsearch.yml
vim masternode/config/jvm.options

Master node elasticsearch.yml configuration:

cluster.name: production-cluster
node.name: master-node
path.data: /home/elk/masternode/data
path.logs: /home/elk/masternode/logs
network.host: 0.0.0.0
network.publish_host: 192.168.1.100
transport.tcp.port: 9301
http.port: 9201
discovery.zen.ping.unicast.hosts: ["192.168.1.100:9301","192.168.1.101:9301","192.168.1.102:9301"]
node.master: true
node.data: false
node.ingest: false
index.number_of_shards: 5
index.number_of_replicas: 1
discovery.zen.minimum_master_nodes: 1
bootstrap.memory_lock: true
http.max_content_length: 1024mb

Parameter explanations:

• cluster.name: Cluster identifier, should be consistent across all nodes
• node.name: Unique identifier for this node
• path.data: Directory for storing index data
• path.logs: Directory for storing log files
• network.host: IP address binding (0.0.0.0 for all interfaces)
• network.publish_host: IP address for node communication
• transport.tcp.port: Port for inter-node communication
• http.port: Port for REST API access
• discovery.zen.ping.unicast.hosts: Initial list of master nodes for discovery
• node.master: Whether this node can become master
• node.data: Whether this node stores data
• node.ingest: Whether this node processes pipelines
• index.number_of_shards: Default number of shards per index
• index.number_of_replicas: Default number of replicas per shard
• discovery.zen.minimum_master_nodes: Minimum master nodes to avoid split-brain
• bootstrap.memory_lock: Prevent swapping for performance
• http.max_content_length: Maximum size of HTTP requests

Node role combinations:

1. Master + Data: Both coordinates cluster operations and stores data (not recommended for production)
2. Master Only: Coordinates cluster operations without storing data
3. Data Only: Stores data without coordinating cluster operations
4. Coordinating Only: Handles request routing without storing data or coordinating cluster operations
5. Ingest Only: Processes documents before indexing

jvm.options configuration for master node:

-Xms2g
-Xmx2g

Data Node Configuration

Copy master node configuration for data node:

cd /opt/elk/
cp -r masternode/ datanode1
vim datanode1/config/elasticsearch.yml
vim datanode1/config/jvm.options

Data node elasticsearch.yml configuration:

cluster.name: production-cluster
node.name: data-node-1
path.data: /home/elk/datanode1/data
path.logs: /home/elk/datanode1/logs
network.host: 0.0.0.0
network.publish_host: 192.168.1.101
transport.tcp.port: 9300
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.1.100:9301","192.168.1.101:9300","192.168.1.102:9300"]
node.master: false
node.data: true
node.ingest: false
index.number_of_shards: 5
index.number_of_replicas: 1
discovery.zen.minimum_master_nodes: 1
bootstrap.memory_lock: true
http.max_content_length: 1024mb

jvm.options configuration for data node:

-Xms8g
-Xmx8g

Verify permissions for both nodes:

chown -R elastic:elastic /opt/elk/masternode
chown -R elastic:elastic /opt/elk/datanode1

3. Deploy to Additional Nodes

For additional nodes, either repeat the installation process or transfer files using:

JDK environment transfer:

scp -r /opt/java root@slave1:/opt
scp -r /opt/java root@slave2:/opt

ElasticSearch environment transfer:

scp -r /opt/elk root@slave1:/opt
scp -r /home/elk root@slave1:/opt
scp -r /opt/elk root@slave2:/opt
scp -r /home/elk root@slave2:/opt

4. Start Elasticsearch

Start each node using the elastic user:

su elastic
cd /opt/elk
./masternode/bin/elasticsearch -d
./datanode1/bin/elasticsearch -d

Verify successful startup:

jps

Or access via browser at http://IP:9200 or http://IP:9201

Kibana Installation

1. File Preparation

Extract the downloaded Kibana:

tar -xvf kibana-6.5.4-linux-x86_64.tar.gz

Move to /opt/elk and rename:

mv kibana-6.5.4-linux-x86_64 /opt/elk
mv kibana-6.5.4-linux-x86_64 kibana6.5.4

2. Configuration Setup

Enter the directory and edit kibana.yml:

cd /opt/elk/kibana6.5.4
vim config/kibana.yml

Change the server host:

server.host: "192.168.1.100"

Add the following line to disable authentication:

xpack.security.enabled: false

3. Start Kibana

Start Kibana as root user:

nohup ./bin/kibana >/dev/null 2>&1 &

Access Kibana at http://IP:5601

Troubleshooting Common Issues

1. max virtual memory areas vm.max_map_count [65530] is too low

Cause: Memory limit too low

Solution: Increase maximum memory limit as described in Linux configuration section 1

2. max number of threads [2048] for user [elastic] is too low

Cause: Thread count limit too low

Solution: Increase thread limit as described in Linux configuration section 2

3. max file descriptors [65535] for elasticsearch process likely too low

Cause: File descriptor limit too low

Solution: Increase file descriptor limit as described in Linux configuration section 3

4. ERROR:bootstrap checks failed

Cause: Memory not locked

Solution: Add bootstrap.memory_lock: true to elasticsearch.yml configuration file