Apache ZooKeeper serves as a centralized coordination service for distributed systems, enabling reliable configuration management, naming, synchronization, and group services. It operates as a hierarchical key-value store with strong consistency guarantees and event-driven notifications.

ZooKeeper clusters follow a leader-follower architecture where a single node acts as the leader while others serve as followers. Quorum-based consensus ensures availability: a majority of nodes (e.g., at least two out of three) must be operational for the ensemble to remain functional. All servers maintain identical copies of the data tree, ensuring linearizable reads and strict ordering of client updates via a FIFO execution model. Operations are atomic—either fully applied or completely aborted.

Internally, ZooKeeper employs a file-system-like abstraction backed by ephemeral znodes and watch mechanisms. Clients register watches on znodes to receive asynchronous notifications upon changes. When services register or deregister, ZooKeeper propagates updates to all interested clients through these watches, supporting dynamic discovery and failover scenarios such as leader election, distributed locks, and service registry.

Leader election in ZooKeeper relies on versioned proposals and epoch-based voting. During initial startup, each server broadcasts its myid and proposes itself as leader; the candidate with the highest myid wins if no prior state exists. In subsequent elections, servers compare (epoch, zxid, myid) tuples lexicographically: higher epoch takes precedence; ties are resolved first by larger transaction ID (zxid), then by greater server ID (myid). This ensures monotonic leadership progression and avoids split-brain conditions.

To deploy a three-node ZooKeeper ensemble:

# On each host, disable security constraints
sudo systemctl stop firewalld
sudo setenforce 0

# Extract and relocate distribution
cd /opt
tar -xzf apache-zookeeper-3.8.3-bin.tar.gz
sudo mv apache-zookeeper-3.8.3-bin /usr/local/zk

# Configure zoo.cfg
sudo tee /usr/local/zk/conf/zoo.cfg << 'EOF'
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/usr/local/zk/data
dataLogDir=/usr/local/zk/logs
clientPort=2181
server.1=192.168.170.111:2888:3888
server.2=192.168.170.113:2888:3888
server.3=192.168.170.114:2888:3888
EOF

# Create required directories and identity files
sudo mkdir -p /usr/local/zk/{data,logs}
echo "1" | sudo tee /usr/local/zk/data/myid  # on first node
echo "2" | sudo tee /usr/local/zk/data/myid  # on second node
echo "3" | sudo tee /usr/local/zk/data/myid  # on third node

# Install init script
sudo tee /etc/init.d/zk << 'EOF'
#!/bin/bash
# chkconfig: 2345 20 90
ZK_HOME="/usr/local/zk"
case "$1" in
  start) $ZK_HOME/bin/zkServer.sh start ;;
  stop)  $ZK_HOME/bin/zkServer.sh stop ;;
  restart) $ZK_HOME/bin/zkServer.sh restart ;;
  status) $ZK_HOME/bin/zkServer.sh status ;;
  *) echo "Usage: $0 {start|stop|restart|status}" ;;
esac
EOF

sudo chmod +x /etc/init.d/zk
sudo chkconfig --add zk
sudo service zk start

Apache Kafka is a distributed streaming platform designed for high-throughput, low-latency message publishing and consumption. Unlike traditional message brokers, Kafka treats messages as immutable, append-only commit logs partitioned across brokers, enabling horizontal scalability and fault tolerance.

Key advantages include decoupling producers from consumers, buffering traffic spikes, enabling asynchronous processing, improving system resilience via persistent storage, and supporting elastic scaling through parallelism.

Kafka supports two core messaging patterns:

• Point-to-point: Each message is consumed exactly once and removed after acknowledgment.
• Publish-subscribe: Messages persist until retention policies expire; multiple independent consumers may read the same stream.

Its architecture centers around several abstractions:

• Broker: A Kafka server instance managing partitions and replicas.
• Producer: Publishes records to topics.
• Consumer: Reads records from topics using offset tracking.
• Topic: A named category or feed to which records are published.
• Partition: An ordered, immutable sequence within a topic; enables parallelism and replication.
• Replica: A copy of a partition across brokers for redundancy.
• Offset: A unique position identifier per partition used by consumers to track progress.
• Controller: A broker elected via ZooKeeper to manage partition leadership and reassignments.

Partitioning improves throughput by distributing load and allows independent scaling per topic segment. Routing decisions follow this logic:

• If a partition is explicitly specified, it is used directly.
• Otherwise, if a key is provided, its hash modulo the number of partitions determines placement.
• If neither is present, round-robin assignment applies.

Kafka depends on ZooKeeper for cluster metadata management, controller election, and consumer group coordination. Starting with Kafka 3.3+, ZooKeeper dependency is optional via KRaft mode—but legacy deployments still rely heavily on it.

To configure a Kafka cluster integrated with the above ZooKeeper ensemble:

# Extract and relocate
cd /opt
tar -xzf kafka_2.13-3.4.0.tgz
sudo mv kafka_2.13-3.4.0 /usr/local/kafka

# Customize server.properties per node
sudo tee /usr/local/kafka/config/server.properties << 'EOF'
broker.id=0
listeners=PLAINTEXT://192.168.170.111:9092
advertised.listeners=PLAINTEXT://192.168.170.111:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/usr/local/kafka/logs
num.partitions=1
log.retention.hours=168
log.segment.bytes=1073741824
zookeeper.connect=192.168.170.111:2181,192.168.170.113:2181,192.168.170.114:2181
EOF

# Set environment variables
echo 'export KAFKA_HOME=/usr/local/kafka' | sudo tee -a /etc/profile
echo 'export PATH=$PATH:$KAFKA_HOME/bin' | sudo tee -a /etc/profile
source /etc/profile

# Install service wrapper
sudo tee /etc/init.d/kafka << 'EOF'
#!/bin/bash
# chkconfig: 2345 22 88
KAFKA_HOME="/usr/local/kafka"
case "$1" in
  start) $KAFKA_HOME/bin/kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties ;;
  stop)  $KAFKA_HOME/bin/kafka-server-stop.sh ;;
  restart) $0 stop && sleep 2 && $0 start ;;
  status) ps aux | grep -q 'kafka.Kafka' && echo 'running' || echo 'stopped' ;;
  *) echo "Usage: $0 {start|stop|restart|status}" ;;
esac
EOF

sudo chmod +x /etc/init.d/kafka
sudo chkconfig --add kafka
sudo service kafka start

# Create and verify topic
kafka-topics.sh --create \
  --bootstrap-server 192.168.170.111:9092 \
  --replication-factor 2 \
  --partitions 3 \
  --topic demo-topic

kafka-topics.sh --list --bootstrap-server 192.168.170.111:9092

# Test producer/consumer flow
kafka-console-producer.sh \
  --bootstrap-server 192.168.170.111:9092 \
  --topic demo-topic

kafka-console-consumer.sh \
  --bootstrap-server 192.168.170.111:9092 \
  --topic demo-topic \
  --from-beginning

Common deployment issues include misconfigured zookeeper.connect URIs (e.g., wrong port or unreachable hosts), mismatched broker.ids, duplicate identifiers, firewall interference, or insufficient disk space. For example, InvalidReplicationFactorException typically indicates fewer live brokers than requested replicas—verify process status, network reachability, and configuration uniqueness across nodes.