Overview of Prometheus and Its Capabilities

Prometheus is an open-source monitoring and alerting toolkit designed too collect and analyze system metrics. Originally developed by SoundCloud, this tool enables efficient tracking and visualization of system performance via time series datasets. Below is a summary of its key features:

Multi-Dimensional Data Model

• Metric Representation: Provides key-value pair-based time series description.
• Dimensional Analysis: Tracks system metrics across multiple dimensions such as region, instance, and job.

PromQL, Visualization, and Storage

• PromQL: A flexible query language for slicing and dicing metric data.
• Dashboards: Comes with built-in expression browser and integrates seamlessly with Grafana forr enhanced visualizations.
• Efficient Data Storage: Uses both in-memory and on-disk time series storage, supporting sharded and federated architectures.

Flexibility and Ecosystem

• Ease of Deployment: Prometheus binaries are built in Go and come precompiled for straightforward deployment. Each Prometheus server operates independently.
• Alerting System: The integrated Alertmanager allows defining precise conditions to trigger notifications based on PromQL expressions.
• Client Libraries: Supported across several languages like Go, Python, end Java, enabling easy integration with in applications.
• Exporters: Prebuilt exporters allow integration with third-party systems like Docker, JMX, and HAProxy, bridging non-Prometheus data sources.

---

Prometheus Architecture

Core Components

1. Prometheus Server: The core engine for collecting and storing metrics.
2. Push Gateway: Facilitates metrics collection for short-lived jobs.
3. Exporters: Specialized tools for data ingestion, e.g., HAProxy and JVM metrics.
4. Alertmanager: Handles notifications triggered by defined alerts.

Logical Framework

Prometheus can be viewed as an Online Analytical Processing (OLAP) system with components for storage, computation, and visualization.

Storage and Computation Layer

• TSDB (Time-Series Database): Handles core data storage and query processing.
• Service Discovery: Dynamically identifies monitoring targets.
• Retrievers: Pulls metrics data from configurable collection endpoints.

Application Layer

• Alerts: Manages notification workflows, supports integrations with PagerDuty and SMTP notifications.
• Visualization: Native Web UI or Grafana integration.

Collection Layer

Metrics collection is divided into:

• Short-Lived Jobs: Data is pushed via the API through Push Gateway.
• Long-Lived Jobs: Data is actively fetched from exporters.

---

Installation of Prometheus

Steps to Deploy Prometheus

1. Download Prometheus Binaries: Prometheus Downloads, e.g., version 2.25.0.
2. Create Prometheus User:

   useradd prometheus
   passwd prometheus
   visudo
   prometheus   ALL=(ALL)   NOPASSWD:ALL
   

3. Upload and Extract Files:

   tar -xvzf prometheus-2.25.0.linux-amd64.tar.gz -C /opt/
   ln -s /opt/prometheus-2.25.0.linux-amd64 /opt/prometheus
   

4. Verify Installation:

   ./prometheus --version
   

5. Configuration Setup: Update the prometheus.yml file:

   global:
     scrape_interval: 15s
     evaluation_interval: 15s
   scrape_configs:
     - job_name: 'prometheus'
       static_configs:
       - targets: ['localhost:9090']
   

---

Integrating Hadoop Cluster Monitoring

Monitoring ZooKeeper Nodes

1. Configure ZooKeeper Metrics Exporter: Update zoo.cfg:

   metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
   metricsProvider.httpPort=7000
   metricsProvider.exportJvmInfo=true
   

2. Distribute Configurations Across Nodes:

   scp /opt/apache-zookeeper-3.6.1-bin/conf/zoo.cfg ha-node2:/opt/apache-zookeeper-3.6.1-bin/conf
   

3. Update Prometheus Configuration:

   scrape_configs:
     - job_name: 'zk_cluster'
       static_configs:
       - targets: ['ha-node1:7000', 'ha-node2:7000']
   

4. Test the Setup:

   curl ha-node1:7000/metrics
   

Monitoring Hadoop Metrics Using Exporters

1. Deploy Hadoop Exporters:

   useradd prometheus_client
   tar -zxvf hadoop_jmx_exporter.tar.gz -C /opt
   

2. Update Node Information in Config File: Modify cluster_config.json for coordination.
3. Integrate with Prometheus:

   scrape_configs:
     - job_name: 'hadoop'
       static_configs:
       - targets: ['ha-node1:9131', 'ha-node2:9131']
   

---

Visualizing Metrics via Grafana

Installing Grafana

Use Prometheus as a data source for more advanced visualizations:

1. Download Grafana: Grafana Downloads
2. Setup Grafana User:

   useradd grafana
   

3. Start Grafana:

   ./bin/grafana-server web
   

4. Access Grafana: Go to http://ha-node1:3000 and log in using default credentials admin/admin.

Creating Dashboards

1. Add Prometheus as a data source in Grafana by selecting Prometheus in Data Sources.
2. Use PromQL queries to build panels:

   rate(http_requests_total[5m])
   

3. Save and configure dashboard layouts for monitoring.

---

Summary

By combining Prometheus's robust metric collection and querying capabilities with Grafana's advanced dashboarding, a powerful monitoring solution for systems like Hadoop clusters can be achieved. This enables administrators to gain real-time insights and respond proactively to system alerts and performance needs.