Overview

When using OpenSSL APIs, proper memory management is critical. Failing to release OpenSSL objects leads to memory leaks. Even experienced developers can forget to call release functions occasionally.

OpenSSL previously provided built-in memory leak detection through the crypto-mdebug feature. This mechanism tracked memory allocations internally—when memory was allocated, OpenSSL recorded it; when freed, the record was removed. By querying the allocation records before program exit, one could identify unfreed memory.

However, this feature was deprecated in OpenSSL 3.0, and the diagnostic functions now have empty implementations.

Examining Available Features

To view OpenSSL feature configurations:

perl configdata.pm --dump > features.txt

The resulting file contains lists of enabled and disabled features. To enable crypto-mdebug during compilation:

perl Configure VC-WIN64A --debug enable-crypto-mdebug zlib-dynamic

Initial Approach: CRYPTO_get_alloc_counts

After the deprecation, I attempted using CRYPTO_get_alloc_counts() to detect memory leaks by comparing allocation counts before and after OpenSSL API calls.

Implementation

memory_tracker.h

#ifndef MEMORY_TRACKER_H
#define MEMORY_TRACKER_H

#include <stdbool.h>

typedef struct {
    int malloc_count_start;
    int realloc_count_start;
    int free_count_start;

    int malloc_count_end;
    int realloc_count_end;
    int free_count_end;
} AllocCounter;

void tracker_start(void);
bool tracker_end(bool enable_assert);

#endif

memory_tracker.c

#include "memory_tracker.h"
#include <openssl/crypto.h>
#include <stdio.h>
#include <assert.h>

static AllocCounter g_alloc_state;

void tracker_start(void)
{
    CRYPTO_get_alloc_counts(
        &g_alloc_state.malloc_count_start,
        &g_alloc_state.realloc_count_start,
        &g_alloc_state.free_count_start
    );
}

bool tracker_end(bool enable_assert)
{
    bool leak_detected = false;
    int net_allocs_start = 0;
    int net_allocs_end = 0;

    CRYPTO_get_alloc_counts(
        &g_alloc_state.malloc_count_end,
        &g_alloc_state.realloc_count_end,
        &g_alloc_state.free_count_end
    );

    net_allocs_start = g_alloc_state.malloc_count_start 
                     + g_alloc_state.realloc_count_start 
                     - g_alloc_state.free_count_start;

    net_allocs_end = g_alloc_state.malloc_count_end 
                   + g_alloc_state.realloc_count_end 
                   - g_alloc_state.free_count_end;

    leak_detected = (net_allocs_start != net_allocs_end);

    if (leak_detected) {
        printf("Leak detected: start=%d, end=%d\n", 
               net_allocs_start, net_allocs_end);
    }

    if (enable_assert) {
        assert(!leak_detected);
    }

    return !leak_detected;
}

Problem with This Approach

This method proved unreliable. OpenSSL internally creates default contexts during initialization, which allocate memory even before user APIs are called. Additionally, internal allocations ocurr within various OpenSSL functions, making it impossible to distinguish between legitimate internal allocations and actual leaks.

Working Solution: Memory Function Hooking

The more effective approach involves intercepting OpenSSL's memory allocation functions using CRYPTO_set_mem_functions(). This provides detailed information about each allocation, including source file and line number.

Implementation

mem_hook.h

#ifndef MEM_HOOK_H
#define MEM_HOOK_H

void install_mem_hook(void);
void uninstall_mem_hook(void);

#endif

mem_hook.cpp

#include "mem_hook.h"
#include <openssl/crypto.h>
#include <openssl/init.h>
#include <cstdio>
#include <cstdint>
#include <map>
#include <assert.h>

struct AllocRecord {
    uint64_t address;
    size_t size;
    const char* filename;
    int line_number;
    uint64_t sequence;
};

static std::map<uint64_t, AllocRecord*> g_alloc_map;

// Original function pointers
static CRYPTO_malloc_fn orig_malloc = nullptr;
static CRYPTO_realloc_fn orig_realloc = nullptr;
static CRYPTO_free_fn orig_free = nullptr;
static OPENSSL_INIT_SETTINGS* g_init_settings = nullptr;
static uint64_t g_alloc_sequence = 0;

static void* hook_malloc(size_t size, const char* file, int line);
static void* hook_realloc(void* ptr, size_t size, const char* file, int line);
static void hook_free(void* ptr, const char* file, int line);

void install_mem_hook(void)
{
    CRYPTO_get_mem_functions(&orig_malloc, &orig_realloc, &orig_free);
    CRYPTO_set_mem_functions(hook_malloc, hook_realloc, hook_free);

    g_init_settings = OPENSSL_INIT_new();
    
    uint64_t flags = OPENSSL_INIT_LOAD_CRYPTO_STRINGS |
                     OPENSSL_INIT_ADD_ALL_CIPHERS |
                     OPENSSL_INIT_ADD_ALL_DIGESTS |
                     OPENSSL_INIT_LOAD_CONFIG |
                     OPENSSL_INIT_ASYNC |
                     OPENSSL_INIT_NO_ATEXIT;

    int result = OPENSSL_init_crypto(flags, g_init_settings);
    assert(1 == result);
}

void uninstall_mem_hook(void)
{
    OPENSSL_cleanup();

    if (!g_alloc_map.empty()) {
        printf("Memory leaks detected:\n");
        for (const auto& entry : g_alloc_map) {
            AllocRecord* rec = entry.second;
            printf("  %p %zu bytes at %s:%d\n",
                   (void*)rec->address,
                   rec->size,
                   rec->filename ? rec->filename : "unknown",
                   rec->line_number);
        }
        assert(false && "OpenSSL memory leaks detected");
    }

    if (g_init_settings) {
        OPENSSL_INIT_free(g_init_settings);
        g_init_settings = nullptr;
    }

    CRYPTO_set_mem_functions(orig_malloc, orig_realloc, orig_free);
}

static void* hook_malloc(size_t size, const char* file, int line)
{
    void* ptr = malloc(size);
    if (ptr) {
        AllocRecord* rec = new AllocRecord();
        rec->address = reinterpret_cast<uint64_t>(ptr);
        rec->size = size;
        rec->filename = file;
        rec->line_number = line;
        rec->sequence = ++g_alloc_sequence;
        g_alloc_map[rec->address] = rec;
    }
    return ptr;
}

static void* hook_realloc(void* ptr, size_t size, const char* file, int line)
{
    void* new_ptr = realloc(ptr, size);
    if (new_ptr) {
        if (ptr) {
            auto it = g_alloc_map.find(reinterpret_cast<uint64_t>(ptr));
            if (it != g_alloc_map.end()) {
                delete it->second;
                g_alloc_map.erase(it);
            }
        }

        AllocRecord* rec = new AllocRecord();
        rec->address = reinterpret_cast<uint64_t>(new_ptr);
        rec->size = size;
        rec->filename = file;
        rec->line_number = line;
        rec->sequence = ++g_alloc_sequence;
        g_alloc_map[rec->address] = rec;
    }
    return new_ptr;
}

static void hook_free(void* ptr, const char* file, int line)
{
    if (ptr) {
        free(ptr);
        auto it = g_alloc_map.find(reinterpret_cast<uint64_t>(ptr));
        if (it != g_alloc_map.end()) {
            delete it->second;
            g_alloc_map.erase(it);
        }
    }
}

Usage Example

#include "mem_hook.h"
#include <openssl/bio.h>

void test_bio_allocation(bool should_free)
{
    BIO* bio = BIO_new_mem_buf("Test Data\n", 10);
    
    if (should_free) {
        BIO_free(bio);
    }
}

int main(void)
{
    install_mem_hook();
    
    test_bio_allocation(false);  // Intentionally leak
    
    uninstall_mem_hook();
    
    return 0;
}

When running this program, the output shows the exact source location of each unfreed allocation:

Memory leaks detected:
  0x7ff8a1c23000 128 bytes at crypto\bio\bio_lib.c:83
  0x7ff8a1c2e040 16 bytes at crypto\bio\bss_mem.c:111
  0x7ff8a1c3a080 32 bytes at crypto\bio\bss_mem.c:119
  0x7ff8a1c3a120 32 bytes at crypto\buffer\buffer.c:35
Assertion failed: false && "OpenSSL memory leaks detected"

Why This Works

The hook-based approach succeeds where simpler methods fail because:

1. Granular tracking: Every allocation and deallocation is intercepted
2. Source information: File and line number are captured at allocation time
3. Clear attribution: Multiple allocations from a single API call appear consecutively in output, making it easy to identify which function caused the leak
4. Post-cleanup validation: By calling uninstall_mem_hook() after OPENSSL_cleanup(), internal OpenSSL allocations are filtered out, leaving only user-caused leaks

Important Notes

• Initialize OpenSSL explicitly with OPENSSL_init_crypto() before hooking to avoid initialization-related allocations contaminating results
• Always call OPENSSL_cleanup() before checking for leaks to allow OpenSSL to release its internal structures
• Release the OPENSSL_INIT_SETTINGS object to avoid false positives
• Restore original functions after detection completes

Conclusion

While the deprecated crypto-mdebug feature and related APIs are no longer available, memory leak detection in OpenSSL applications remains possible through memory function hooking. This approach provides detailed, actionable information that significantly accelerates debugging of OpenSSL memory management issues.