System Prerequisites

The following software versions are required for this deployment:

• Operating System: CentOS 6.5
• Java Development Kit: 1.8
• Elasticsearch: 5.2.2
• Logstash: 5.2.2
• Kibana: 5.2.2

Java Runtime Configuration

Download the 64-bit JDK archive and transfer it to the server. Extract the contents to the designated installation directory.

cd /opt/java
tar -xzvf jdk-8u111-linux-x64.tar.gz

Define the necessary environment variables by editing the system profile. Append the following configuration to /etc/profile:

JAVA_HOME=/opt/java/jdk1.8.0_111
JRE_HOME=/opt/java/jdk1.8.0_111/jre
CLASSPATH=.:$JAVA_HOME/lib:/dt.jar:$JAVA_HOME/lib/tools.jar
PATH=$PATH:$JAVA_HOME/bin
export JAVA_HOME
export JRE_HOME

Apply the changes immediately:

source /etc/profile

OS Kernel and User Limits

Adjust the process and file descriptor limits to accommodate Elasticsearch requirements. Edit /etc/security/limits.conf and insert:

* soft nproc 65536
* hard nproc 65536
* soft nofile 65536
* hard nofile 65536

Update the kernel virtual memory map count in /etc/sysctl.conf:

vm.max_map_count=262144

Reload the system configuration:

sysctl -p

Create a dedicated group and user for the ELK stack to ensure proper permission isolation:

groupadd es_admin
useradd -g es_admin es_user
mkdir -p /srv/elastic
chown -R es_user:es_admin /srv/elastic

Elasticsearch Service Setup

Switch to the dedicated user account before proceeding with the installation:

su - es_user

Download the Elasticsearch archive, upload it to /srv/elastic, and extract it. Navigate to the configuration directory to modify config/elasticsearch.yml:

network.host: 172.16.0.10
http.port: 9200
bootstrap.system_call_filter: false
http.cors.enabled: true
http.cors.allow-origin: "*"
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*,logstash*

For testing environments, reduce the heap size in config/jvm.options to conserve memory:

-Xms512m
-Xmx512m

Launch the Elasticsearch daemon in the background:

./bin/elasticsearch -d

Logstash Pipeline Configuration

Create a configuration directory for Logstash pipelines. Define a new configurasion file named nginx_traffic.conf within /srv/elastic/logstash-5.2.2/config.d/.

input {
    file {
        path => [ "/var/www/nginx/logs/access.log" ]
        start_position => "beginning"
        ignore_older => 0
        type => "web-access"
    }
}

filter {
    if [type] == "web-access" {
        grok {
            match => [
                "message","%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NOTSPACE:http_x_forwarded_for}"
            ]
        }

        urldecode {
            all_fields => true
        }

        date {
            locale => "en"
            match => ["timestamp" , "dd/MMM/YYYY:HH:mm:ss Z"]
        }

        geoip {
            source => "clientip"
            target => "geoip"
            database => "/srv/elastic/logstash-5.2.2/data/GeoLite2-City.mmdb"
            add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
            add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
        }

        mutate {
            convert => [ "[geoip][coordinates]", "float" ]
            convert => [ "response","integer" ]
            convert => [ "bytes","integer" ]
            replace => { "type" => "nginx_web" }
            remove_field => "message"
        }
    }
}

output {
    elasticsearch {
        hosts => ["172.16.0.10:9200"]
        index => "web-traffic-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}

The ppieline consists of three stages:

1. Input: Captures logs from local files or message queues.
2. Filter: Parses raw data, extracts fields, and enriches records (e.g., GeoIP).
3. Output: Sends processed events to Elasticsearch or other sinks.

Validate the configuration syntax before running:

/srv/elastic/logstash-5.2.2/bin/logstash -t -f /srv/elastic/logstash-5.2.2/config.d/nginx_traffic.conf

Start the Logstash process:

nohup /srv/elastic/logstash-5.2.2/bin/logstash -f /srv/elastic/logstash-5.2.2/config.d/nginx_traffic.conf &

Kibana Dashboard Initialization

Navigate to the Kibana installation directory and launch the service:

cd /srv/elastic/kibana-5.2.2
nohup ./bin/kibana &

Once the process is running, the web interface becomes available to visualize the indices generated by Logstash, such as web-traffic-YYYY.MM.DD.