What is epoll?

epoll is a Linux-specific I/O multiplexing mechanism designed to overcome the limitations of older solutions like select and poll. Optimized for handling large-scale network workloads, it delivers significant performance improvements for applications managing thousands of concurrent connections.

Key advantages over select/poll:

• No hard connection limit: Unlike select (capped at 2048 by default), epoll supports far more concurrent connections, limited only by system resources.
• Scalable efficiency: epoll's performance scales with the number of active connections, not the total number of open connections. This makes it drastically more efficient for workloads where most sockets are idle.
• Shared memory optimization: epoll uses a shared memory region between user and kernel space, eliminating redundant data copies that slow down select and poll.

Kernel Space vs. User Space

In Linux, each process is allocated a 4GB virtual address space, split into two distinct regions:

• Kernel Space: The top 1GB of virtual memory, reserved exclusively for the operating system kernel. It houses kernel code, device drivers, and critical data structures, and is shared across all processes.
• User Space: The remaining 3GB, where user applications run. Each process has its own isolated user space for code, data, and stack.

Processes can transition to kernel space via system calls (e.g., epoll_ctl, read), allowing them to access kernel-managed resources in a controlled manner.

Why Separate Kernel and User Space?

Modern CPUs implement privilege levels to protect the system from unsafe operations. High-risk instructions (such as modifying hardware registers or accessing physical memory directly) are restricted to kernel mode. By splitting the address space, we ensure user applications cannot execute dangerous commands that could crash the system or corrupt data.

Kernel Mode vs. User Mode

• Kernel Mode: When a process runs in kernel space, it has unrestricted access to all CPU instructions and system resources. This mode is reserved for critical operations like memory menagement and device I/O.
• User Mode: Processes in user space operate with limited privileges. They cannot directly access hardware or modify kernel data, and any attempt to do so triggers a system call to switch to kernel mode.

Core epoll Components

File Descriptors (fd)

A file descriptor is a non-negative integer acting as an index to a kernel-managed table of open files or sockets. When you open a file or create a socket, the kernel returns an fd that you use for all subsequent operations (e.g., read, write).

• Special FDs: 0 (standard input), 1 (standard output), and 2 (standard error) are reserved for default I/O streams.
• FD Limit: The maximum number of open fds per process can be checked with the ulimit -n command (default is often 256, but can be adjusted system-wide or per process).

epoll Functions and Structures

To use epoll, you’ll work with three core components, all defined in the <sys/epoll.h> header:

1. Create an epoll Instance

epoll_create1 (the modern replacement for the deprecated epoll_create) initializes an epoll instance and returns its file descriptor:

#include <sys/epoll.h>
#include <stdio.h>
#include <stdlib.h>

// Create epoll instance with CLOEXEC flag to auto-close on exec
int epoll_fd = epoll_create1(EPOLL_CLOEXEC);
if (epoll_fd == -1) {
    perror("epoll_create1 failed");
    exit(EXIT_FAILURE);
}

2. Manage Events with epoll_ctl

epoll_ctl registers, modifies, or removes file descriptors fromm the epoll instance. It takes the epoll fd, an operation code, the target socket fd, and an epoll_event struct:

// Define event to monitor incoming data with edge-triggered mode
struct epoll_event event_config;
event_config.events = EPOLLIN | EPOLLET;
event_config.data.fd = client_socket;

// Add the client socket to the epoll instance
int op_result = epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_socket, &event_config);
if (op_result == -1) {
    perror("epoll_ctl add failed");
    close(client_socket);
    close(epoll_fd);
    exit(EXIT_FAILURE);
}

Common operations for epoll_ctl:

• EPOLL_CTL_ADD: Register a new file descriptor with the epoll instance.
• EPOLL_CTL_MOD: Update the event mask for an existing file descriptor.
• EPOLL_CTL_DEL: Remove a file descriptor from the epoll instance.

3. The epoll_event Struct

This struct holds the event type and associated user data, using a union to support multiple data types:

typedef union epoll_data {
    void* ptr;    // Pointer to user-defined data
    int fd;       // Associated file descriptor
    uint32_t u32; // 32-bit integer value
    uint64_t u64; // 64-bit integer value
} epoll_data_t;

struct epoll_event {
    uint32_t events;    // Bitmask of epoll events to monitor
    epoll_data_t data;  // User data linked to the event
};

Key Epoll Events

The events field uses a bitmask to specify wich events to watch for:

• EPOLLIN: Data is available for reading.
• EPOLLOUT: The socket is ready for writing data.
• EPOLLERR: An error occurred on the file descriptor.
• EPOLLHUP: The connection was closed (hang up).
• EPOLLET: Enable edge-triggered mode (instead of the default level-triggered).
• EPOLLONESHOT: Disable the file descriptor after the first event is triggered (must be rearmed manually for future events).

Wait for Ready Events with epoll_wait

epoll_wait blocks until events are ready or a timeout occurs. It copies ready events into a user-provided array for processing:

#define MAX_READY_EVENTS 1024
struct epoll_event ready_events[MAX_READY_EVENTS];

// Wait for up to MAX_READY_EVENTS ready events, timeout after 5000ms
int num_ready = epoll_wait(epoll_fd, ready_events, MAX_READY_EVENTS, 5000);
if (num_ready == -1) {
    perror("epoll_wait failed");
    close(epoll_fd);
    exit(EXIT_FAILURE);
}

// Process each ready event
for (int i = 0; i < num_ready; ++i) {
    int active_fd = ready_events[i].data.fd;
    if (ready_events[i].events & EPOLLIN) {
        // Handle incoming data on active_fd
    } else if (ready_events[i].events & (EPOLLERR | EPOLLHUP)) {
        // Clean up closed or errored connections
        close(active_fd);
    }
}

• Timeout Values:
  • -1: Block indefinitely until events are ready.
  • 0: Return immediately (non-blocking mode).
  • Positive integer: Wait up to the specified number of milliseconds before timing out.
• Return Value: Number of ready events, 0 if no events were ready before timeout, or -1 if an error occurred.

Edge-Triggered (ET) vs. Level-Triggered (LT) Modes

epoll supports two distinct trigger modes for notifying applications of events:

• Level-Triggered (LT, Default): The kernel continues to notify the application of an event until it’s fully processed. For example, if data remains unread in the buffer, epoll_wait will return the file descriptor again in subsequent calls.
• Edge-Triggered (ET): The kernel only notifies the application once when the event state changes (e.g., new data arrives). The application must read all available data in one go, as no further notifications will be sent until more data is received. ET mode is more efficient but requires careful handling to avoid missing events.