Apache Kafka employs log semgentation to efficiently organize message data on disk. This mechanism enables optimized write, read, and cleanup operations for persistent logs.

Log Segmentation Principles:

1. Each Kafka topic partition maintains a collection of persistent log files called segments
2. Segments contain message ranges determined by either:
   • Time-based policies (e.g., retaining only the last N days of data)
   • Size-based policies (creating new segments when reaching maximum configured size)

Message Retrieval System:

• Messages are uniquely identified by monotonically increasing offsets
• Each segment has a accompanying index file for rapid message location
• Indexes use sparse storage, containing only enough entries for efficient lookup

Segment Maintainance Process:

• Periodic cleanup tasks evaluate segments against retention policies
• Eligible segments are marked for deletion and eventually removed from disk (configurable via file.delete.delay.ms)

Configuration examples using Kafka CLI tools:

# Set 24-hour retention policy
bin/kafka-configs.sh --zookeeper localhost:2181 \
  --alter --entity-type topics \
  --entity-name my-topic \
  --add-config retention.ms=86400000

# Inspect segment details
bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe

The describe command outputs comprehensive segment information including partition offsets and storage metrics.