Fading Coder

One Final Commit for the Last Sprint

Home > Tech > Content

Reversing Custom XTEA Implementations in Binary Challenges

Tech 1

The XTEA (eXtended Tiny Encryption Algorithm) operates on 64-bit data blocks divided into two 32-bit halves, utilizing a 128-bit key split into four 32-bit words. Unlike its predecessor TEA, XTEA introduces a more complex key schedule and alternates the update operations between the left and right halves within each round. This structure mitigates certain related-key attacks while maintaining a lightweight footprint suitable for embedded systems and obfuscation routines.

Standard implementations typically run for 32 rounds with a fixed delta constant derived from the golden ratio (0x9E3779B9). The encryption process accumulates this delta into a sum variable, which dictates both the round progression and the key index selection.

#include <stdint.h>

#define STANDARD_ROUNDS 32
#define GOLDEN_RATIO_DELTA 0x9E3779B9

void xtea_encrypt(uint32_t block[2], const uint32_t secret_key[4]) {
    uint32_t left = block[0];
    uint32_t right = block[1];
    uint32_t accumulator = 0;

    for (uint8_t round = 0; round < STANDARD_ROUNDS; ++round) {
        left += (((right << 4) ^ (right >> 5)) + right) ^ (accumulator + secret_key[accumulator & 3]);
        accumulator += GOLDEN_RATIO_DELTA;
        right += (((left << 4) ^ (left >> 5)) + left) ^ (accumulator + secret_key[(accumulator >> 11) & 3]);
    }

    block[0] = left;
    block[1] = right;
}

void xtea_decrypt(uint32_t block[2], const uint32_t secret_key[4]) {
    uint32_t left = block[0];
    uint32_t right = block[1];
    uint32_t accumulator = GOLDEN_RATIO_DELTA * STANDARD_ROUNDS;

    for (uint8_t round = 0; round < STANDARD_ROUNDS; ++round) {
        right -= (((left << 4) ^ (left >> 5)) + left) ^ (accumulator + secret_key[(accumulator >> 11) & 3]);
        accumulator -= GOLDEN_RATIO_DELTA;
        left -= (((right << 4) ^ (right >> 5)) + right) ^ (accumulator + secret_key[accumulator & 3]);
    }

    block[0] = left;
    block[1] = right;
}

Reverse engineering tasks frequently modify core constants and shift parameters to bypass signature-based detection. Common alterations include changing the iteration count, replacing the golden ratio delta with an arbitrary integer, and adjusting bitwise shift offsets. When analyzing such variants, the decryption routine must precisely mirror the encryption logic in reverse order, ensuring the accumulator is initialized to delta * rounds and decremented appropriately.

Implementing these custom variants in Python requires explicit handling of 32-bit unsigned integer overflow, as Python natively supports arbitrary-precision integers. The ctypes library provides c_uint32, which automatically truncates values to 32 bits during arithmetic operations, accurately simulating C/C++ behavior.

from ctypes import c_uint32
import struct

CUSTOM_ROUNDS = 33
CUSTOM_DELTA = 999999999

def custom_xtea_decrypt(cipher_block, round_keys):
    left = c_uint32(cipher_block[0])
    right = c_uint32(cipher_block[1])
    # Initialize accumulator to delta * rounds, adjusted for the specific variant's off-by-one behavior if present
    state_sum = c_uint32((CUSTOM_DELTA * CUSTOM_ROUNDS) + 1)

    for _ in range(CUSTOM_ROUNDS):
        right.value -= (((left.value << 3) ^ (left.value >> 4)) + left.value) ^ (state_sum.value + round_keys[(state_sum.value >> 11) & 3])
        state_sum.value -= CUSTOM_DELTA
        left.value -= (((right.value << 3) ^ (right.value >> 4)) + right.value) ^ (state_sum.value + round_keys[state_sum.value & 3])

    return left.value, right.value

def process_ciphertext(raw_bytes, key_schedule):
    # Convert little-endian byte chunks into 32-bit integers
    num_blocks = len(raw_bytes) // 8
    plaintext_words = []

    for idx in range(num_blocks):
        chunk = raw_bytes[idx * 8 : (idx + 1) * 8]
        # Unpack two 32-bit little-endian unsigned integers
        word_a, word_b = struct.unpack('<II', chunk)
        decrypted_a, decrypted_b = custom_xtea_decrypt((word_a, word_b), key_schedule)
        plaintext_words.extend([decrypted_a, decrypted_b])

    # Convert back to bytes
    return struct.pack('<' + 'I' * len(plaintext_words), *plaintext_words)

# Example usage context
sample_cipher = bytes([130, 67, 163, 137, 111, 186, 128, 200, 248, 180, 86, 189, 179, 65, 178, 141,
                       218, 68, 14, 4, 3, 46, 56, 222, 18, 84, 173, 137, 149, 48, 99, 33,
                       223, 13, 148, 17, 220, 178, 208, 17])
master_key = [5, 20, 13, 14]

recovered_data = process_ciphertext(sample_cipher, master_key)

When reverse engineering binaries containing modified XTEA, static analysis should focus on identifying the loop boundary, the constant added to the accumulator, and the bitwise shift values applied to the block halves. Dynamic analysis or hardware breakpoints on the key array or accumulator variable can quickly reveal the custom parameters. The decryption script must replicate the exact overflow behavior and key indexing logic observed in the disassembly, particularly noting how the sum variable masks into the key array indices (sum & 3 and (sum >> 11) & 3). Proper alignment of byte order during the conversion between raw ciphertext and 32-bit words is critical, as endianness mismatches will produce garbled output even with a mathematically correct algorithm.

Related Articles

Understanding Strong and Weak References in Java

Strong References Strong reference are the most prevalent type of object referencing in Java. When an object has a strong reference pointing to it, the garbage collector will not reclaim its memory. F...

Comprehensive Guide to SSTI Explained with Payload Bypass Techniques

Introduction Server-Side Template Injection (SSTI) is a vulnerability in web applications where user input is improper handled within the template engine and executed on the server. This exploit can r...

Implement Image Upload Functionality for Django Integrated TinyMCE Editor

Django’s Admin panel is highly user-friendly, and pairing it with TinyMCE, an effective rich text editor, simplifies content management significantly. Combining the two is particular useful for bloggi...

Leave a Comment

Anonymous

◎Feel free to join the discussion and share your thoughts.