Practical Log Collection for Security Operations Using Elastic Stack
Effective incident response and attack attribution rely heavily on comprehensive log data. To detect, analyze, and block adversaries early in their lifecycle, organizations must collect diverse logs across endpoints and infrastructure.
Deploying Elasticsearch and Kibana
RPM-Based Installation (Recommended for Stability)
Due to instability observed with Docker deployments, a native RPM installation is preferred:
# Download and verify Elasticsearch
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.1-x86_64.rpm
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.12.1-x86_64.rpm.sha512
shasum -a 512 -c elasticsearch-8.12.1-x86_64.rpm.sha512
sudo rpm -i elasticsearch-8.12.1-x86_64.rpm
# Download and verify Kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.12.1-x86_64.rpm
wget https://artifacts.elastic.co/downloads/kibana/kibana-8.12.1-x86_64.rpm.sha512
shasum -a 512 -c kibana-8.12.1-x86_64.rpm.sha512
sudo rpm -i kibana-8.12.1-x86_64.rpm
After installation, configure both services to bind to all interfaces:
/etc/elasticsearch/elasticsearch.yml
http.host: 0.0.0.0
transport.host: 0.0.0.0
/etc/kibana/kibana.yml
server.host: "0.0.0.0"
Enable and start the services:
sudo systemctl daemon-reload
sudo systemctl enable --now elasticsearch.service
sudo systemctl enable --now kibana.service
Verify Elasticsearch connectivity:
export ELASTIC_PASSWORD="<generated_password>"
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:$ELASTIC_PASSWORD https://localhost:9200
Setting Up Fleet and Elastic Agent
In Kibana, navigate to Management → Fleet and deploy a Fleet Server (typically on port 8220 over HTTPS). Note: Fleet Server and Elastic Agent cannot coexist on the same host.
Once the Fleet Server is active, enroll additional Elastic Agents. To enhance security visibility, install integrations via Security → Manage → Add security integrations, enabling detection rules such as reverse shell monitoring.
Ensure detection rules are enabled under Security → Rules to populate the Detection & Response dashboard.
Collecting Windows Event Logs with Winlogbeat
Windows event logs require structured collection. Winlogbeat is used to forward logs to Elasticsearch.
Sample winlogbeat.yml configuration:
winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: System
- name: Security
- name: Microsoft-Windows-Sysmon/Operational
- name: Windows PowerShell
event_id: 400, 403, 600, 800
- name: Microsoft-Windows-PowerShell/Operational
event_id: 4103, 4104, 4105, 4106
- name: ForwardedEvents
tags: [forwarded]
setup.kibana:
host: "kibana_host:5601"
output.elasticsearch:
hosts: ["elasticsearch_host:9200"]
username: "elastic"
password: "<password>"
pipeline: "winlogbeat-%{[agent.version]}-routing"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
Validate the configuration:
./winlogbeat.exe test config -c ./winlogbeat.yml -e
Start the service and verify data ingestion by listing Elasticsearch indices:
curl http://elasticsearch_host:9200/_cat/indices?v | grep winlogbeat
If data appears in Elasticsearch but not in Kibana, manually create an index pattern:
- Go to Discover → Settings (gear icon) → Index Patterns
- Create a new pattern matching
winlogbeat-* - Set
@timestampas the time field
Data should now be queryable in Discover, enabling analysis of Security, Sysmon, and PowerShell events for threat hunting and forensic investigations.