Accessing Physical Memory with /dev/mem and mmap on Linux

Overview

For low-level debugging and register poking, user-space can reach device registers or RAM by mapping physical addresses through /dev/mem. Tools like devmem do this by invoking mmap on the /dev/mem character device, establishing a virtual mapping to the requested physical range. This article walks through how to use devmem, how to reproduce its behavior from user-space code, and how the kernel implements the mapping path down to remap_pfn_range.

Using BusyBox devmem

BusyBox provides a tiny utility named devmem (misc utilities option) that performs read/write operations on physical addresses via /dev/mem.

• Command synopsis:
  • devmem ADDRESS [WIDTH [VALUE]]
• Behavior:
  • If VALUE is omitted, devmem reads from ADDRESS.
  • If VALUE is supplied, devmem writes VALUE using WIDTH and prints the result (varies by BusyBox version).
• WIDTH specifies the access size in bits (8/16/32/64).

Example reads:

• devmem 0x44e07134 16 → reads a 16-bit value
• devmem 0x44e07134 32 → reads a 32-bit value
• devmem 0x44e07134 8 → reads an 8-bit value

User-space mapping with mmap

mmap and munmap are the standard interfaces for memory mapping in user proceses:

• void* mmap(void* hint, size_t len, int prot, int flags, int fd, off_t off);
• int munmap(void* addr, size_t len);

Parameter notes:

• hint: preferred virtual address; nullptr lets the kernel choose.
• len: byte length of the mapping (rounded up to whole pages).
• prot: page protections (PROT_READ, PROT_WRITE, PROT_EXEC, PROT_NONE).
• flags: mapping behavior (e.g., MAP_SHARED, MAP_PRIVATE, MAP_FIXED, MAP_ANONYMOUS).
• fd: file descriptor of the object to map (e.g., /dev/mem); ignored for anonymous mappings.
• off: file offset (must be page aligned for file-backed mappings). For /dev/mem, off encodes the physical address rounded down to page size.

Illustrative mapping example (reads/writes a physical address via /dev/mem):

#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <unistd.h>

static void die(const char* msg) {
    perror(msg);
    exit(1);
}

int main(int argc, char** argv) {
    if (argc < 3) {
        fprintf(stderr, "usage: %s <phys_addr_hex> <width_bits> [value_hex]\n", argv[0]);
        return 2;
    }

    off_t phys = strtoull(argv[1], NULL, 0);
    int width = atoi(argv[2]);
    int do_write = (argc >= 4);

    long pagesz = sysconf(_SC_PAGESIZE);
    off_t page_base = phys & ~((off_t)pagesz - 1);
    off_t page_off  = phys - page_base;

    int fd = open("/dev/mem", do_write ? O_RDWR | O_SYNC : O_RDONLY | O_SYNC);
    if (fd < 0) die("open /dev/mem");

    size_t map_len = page_off + sizeof(uint64_t); // enough to cover max width
    void* va = mmap(NULL, map_len, do_write ? (PROT_READ | PROT_WRITE) : PROT_READ,
                    MAP_SHARED, fd, page_base);
    if (va == MAP_FAILED) die("mmap");

    volatile uint8_t* p8  = (volatile uint8_t*) ((uint8_t*)va + page_off);
    volatile uint16_t* p16 = (volatile uint16_t*) ((uint8_t*)va + page_off);
    volatile uint32_t* p32 = (volatile uint32_t*) ((uint8_t*)va + page_off);
    volatile uint64_t* p64 = (volatile uint64_t*) ((uint8_t*)va + page_off);

    if (!do_write) {
        switch (width) {
            case 8:  printf("0x%02X\n",  *p8);  break;
            case 16: printf("0x%04X\n", *p16); break;
            case 32: printf("0x%08X\n", *p32); break;
            case 64: printf("0x%016llX\n", (unsigned long long)*p64); break;
            default: fprintf(stderr, "invalid width\n"); munmap(va, map_len); close(fd); return 2;
        }
    } else {
        unsigned long long val = strtoull(argv[3], NULL, 0);
        switch (width) {
            case 8:  *p8  = (uint8_t)val;  printf("0x%02X\n",  *p8);  break;
            case 16: *p16 = (uint16_t)val; printf("0x%04X\n", *p16); break;
            case 32: *p32 = (uint32_t)val; printf("0x%08X\n", *p32); break;
            case 64: *p64 = (uint64_t)val; printf("0x%016llX\n", (unsigned long long)*p64); break;
            default: fprintf(stderr, "invalid width\n"); munmap(va, map_len); close(fd); return 2;
        }
    }

    munmap(va, map_len);
    close(fd);
    return 0;
}

The mapping permissions in the example mirror devmem’s behavior: if a write value is given, the mapping uses PROT_READ|PROT_WRITE; otherwise, it is read-only.

Kernel path from mmap to physical mapping

On ARM (and similarly on other architectures), user-space mmap ultimate reaches a file-backed implementation that sets up the VMA and page tables. The high-level path is:

• user-space → swi/svc → sys_mmap2 (for 4K-based offsets) → sys_mmap_pgoff → vm_mmap_pgoff → do_mmap_pgoff → mmap_region → file->f_op->mmap → driver’s mmap handler (for /dev/mem, mmap_mem) → remap_pfn_range

Syscall numbers (ARM excerpt)

• __NR_mmap, __NR_munmap, and __NR_mmap2 are defined in arch/arm/include/uapi/asm/unistd.h.
• sys_mmap2 adapts a 4K-based offset to sys_mmap_pgoff. Conceptually:

// Pseudocode for the ARCH glue (simplified)
long sys_mmap2_wrapper(args..., unsigned long off_4k) {
    if (PAGE_SHIFT > 12 && (off_4k & ((1UL << (PAGE_SHIFT - 12)) - 1)))
        return -EINVAL;
    return sys_mmap_pgoff(args..., off_4k >> (PAGE_SHIFT - 12));
}

Entering the generic mmap implementation

sys_mmap_pgoff dispatches to the core implementation in mm/mmap.c. The signature in include/linux/syscalls.h resembles:

asmlinkage long sys_mmap_pgoff(unsigned long uaddr,
                               unsigned long len,
                               unsigned long prot,
                               unsigned long flags,
                               unsigned long fd,
                               unsigned long pgoff);

The syscall wrapper validates the file descriptor (if not MAP_ANONYMOUS), adjusts flag combinations, and calls vm_mmap_pgoff. A condensed view of SYSCALL_DEFINE6(mmap_pgoff) is:

SYSCALL_DEFINE6(mmap_pgoff,
    unsigned long, uaddr, unsigned long, len,
    unsigned long, prot,  unsigned long, flags,
    unsigned long, fd,    unsigned long, pgoff)
{
    struct file *filp = NULL;
    unsigned long ret = -EBADF;

    if (!(flags & MAP_ANONYMOUS)) {
        filp = fget(fd);
        if (!filp) goto out;
        if (is_file_hugepages(filp))
            len = ALIGN(len, huge_page_size(hstate_file(filp)));
        if ((flags & MAP_HUGETLB) && !is_file_hugepages(filp)) {
            ret = -EINVAL;
            goto out_put;
        }
    }

    flags &= ~(MAP_EXECUTABLE | MAP_DENYWRITE);
    ret = vm_mmap_pgoff(filp, uaddr, len, prot, flags, pgoff);

out_put:
    if (filp) fput(filp);
out:
    return ret;
}

vm_mmap_pgoff sets up the mapping under mm->mmap_sem and optionally pre-populates pages:

unsigned long vm_mmap_pgoff(struct file *filp, unsigned long uaddr,
                            unsigned long len, unsigned long prot,
                            unsigned long flags, unsigned long pgoff)
{
    unsigned long res;
    struct mm_struct *mm = current->mm;
    unsigned long populate;

    res = security_mmap_file(filp, prot, flags);
    if (res)
        return res;

    down_write(&mm->mmap_sem);
    res = do_mmap_pgoff(filp, uaddr, len, prot, flags, pgoff, &populate);
    up_write(&mm->mmap_sem);

    if (populate)
        mm_populate(res, populate);

    return res;
}

Establishing the VMA and choosing the address range

do_mmap_pgoff determines the target virtual address interval and maps either a file or anonymous memory, depending on the presence of filp:

unsigned long do_mmap_pgoff(struct file *filp, unsigned long uaddr,
                            unsigned long len, unsigned long prot,
                            unsigned long flags, unsigned long pgoff,
                            unsigned long *populate)
{
    struct mm_struct *mm = current->mm;
    vm_flags_t vmf;

    *populate = 0;

    // Pick a free range in the process address space
    uaddr = get_unmapped_area(filp, uaddr, len, pgoff, flags);

    vmf = calc_vm_prot_bits(prot) |
          calc_vm_flag_bits(flags) |
          mm->def_flags |
          VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC;

    if (filp) {
        if ((flags & MAP_TYPE) == MAP_SHARED)
            vmf |= VM_SHARED | VM_MAYSHARE;
    } else {
        if ((flags & MAP_TYPE) == MAP_SHARED) {
            pgoff = 0;
            vmf |= VM_SHARED | VM_MAYSHARE;
        } else {
            pgoff = uaddr >> PAGE_SHIFT;
        }
    }

    return mmap_region(filp, uaddr, len, vmf, pgoff);
}

Backing the VMA and invoking the file’s mmap hook

mmap_region handles accounting, overlap resolution, VMA allocation/merge, and calls into the file’s mmap operation:

unsigned long mmap_region(struct file *filp, unsigned long start,
                          unsigned long len, vm_flags_t vmf,
                          unsigned long pgoff)
{
    struct mm_struct *mm = current->mm;
    struct vm_area_struct *vma, *prev;
    struct rb_node **rb_link, *rb_parent;

    if (!may_expand_vm(mm, len >> PAGE_SHIFT)) {
        if (!(vmf & MAP_FIXED))
            return -ENOMEM;
        if (!may_expand_vm(mm, (len >> PAGE_SHIFT) - count_vma_pages_range(mm, start, start + len)))
            return -ENOMEM;
    }

    // Evict overlap if necessary
    while (find_vma_links(mm, start, start + len, &prev, &rb_link, &rb_parent)) {
        if (do_munmap(mm, start, len))
            return -ENOMEM;
    }

    // Try merging with an adjacent VMA
    vma = vma_merge(mm, prev, start, start + len, vmf, NULL, filp, pgoff, NULL);
    if (vma)
        return vma->vm_start;

    // Allocate and initialize a new VMA
    vma = kmem_cache_zalloc(vm_area_cachep, GFP_KERNEL);
    if (!vma)
        return -ENOMEM;

    vma->vm_mm = mm;
    vma->vm_start = start;
    vma->vm_end = start + len;
    vma->vm_flags = vmf;
    vma->vm_page_prot = vm_get_page_prot(vmf);
    vma->vm_pgoff = pgoff;
    INIT_LIST_HEAD(&vma->anon_vma_chain);

    if (filp) {
        if (vmf & VM_DENYWRITE) {
            int err = deny_write_access(filp);
            if (err) { kmem_cache_free(vm_area_cachep, vma); return err; }
        }
        vma->vm_file = get_file(filp);
        {
            int err = filp->f_op->mmap(filp, vma);
            if (err) { /* undo setup, free vma, etc. */ return err; }
        }
    } else if (vmf & VM_SHARED) {
        int err = shmem_zero_setup(vma);
        if (err) { kmem_cache_free(vm_area_cachep, vma); return err; }
    }

    // Link the VMA into mm (vma_link and friends)
    // ...

    return vma->vm_start;
}

/dev/mem’s mmap handler and remapping physical frames

For /dev/mem, the character device implements .mmap to attach the VMA direct to physical frames. The driver lives in drivers/char/mem.c. A simplified version of its operations and mmap callback is:

static const struct file_operations memory_fops = {
    .llseek            = memory_lseek,
    .read              = read_mem,
    .write             = write_mem,
    .mmap              = mmap_mem,        // /dev/mem mmap entry point
    .open              = open_mem,
    .get_unmapped_area = get_unmapped_area_mem,
};

static int mmap_mem(struct file *filp, struct vm_area_struct *vma)
{
    size_t span = vma->vm_end - vma->vm_start;

    if (!valid_mmap_phys_addr_range(vma->vm_pgoff, span))
        return -EINVAL;

    if (!private_mapping_ok(vma))
        return -ENOSYS;

    if (!range_is_allowed(vma->vm_pgoff, span))
        return -EPERM;

    if (!phys_mem_access_prot_allowed(filp, vma->vm_pgoff, span, &vma->vm_page_prot))
        return -EINVAL;

    vma->vm_page_prot = phys_mem_access_prot(filp, vma->vm_pgoff, span, vma->vm_page_prot);

    vma->vm_ops = &mmap_mem_ops; // e.g., open/close, access hooks

    // Map physical frames (PFNs) into user VMA
    if (remap_pfn_range(vma, vma->vm_start, vma->vm_pgoff, span, vma->vm_page_prot))
        return -EAGAIN;

    return 0;
}

Key details:

• vma->vm_pgoff holds the starting PFN (physical frame number); user-space’s mmap offfset encodes the physical address divided by PAGE_SIZE.
• remap_pfn_range wires the VMA too the specified physical range and tags the VMA as I/O (VM_IO), ensuring it’s treated as device memory.
• vma->vm_page_prot is adjusted to reflect device memory attributes and requested protections.

Once the VMA is established, any load/stores through the mapped virtual addresses in user-space read/write the underlying physical locations governed by the VMA’s protections and system security policy.