Overview

This guide demonstrates how to establish a differential testing environment using NEMU as the device under test (DUT) and QEMU as the reference implementation (REF). The setup employs socket communication for synchronization and comparison.

NEMU Compilation Setup

Configuration

Execute the following commands to install dependencies and configure NEMU:

apt install build-essential man gcc gdb git libreadline-dev libsdl2-dev zstd libzstd-dev
git clone https://github.com/OpenXiangShan/NEMU.git
cd NEMU/
export NEMU_HOME=/path/to/NEMU
make riscv64-benos_defconfig
make menuconfig

Configure the following settings:

• Build Options → Optimization Level: Set to O0
• Build Options → Enable link-time optimization: Disabled
• Build Options → Enable debug information: Enabled
• Testing and Debugging → Enable differential testing: Enabled with socket communication
• Testing and Debugging → Enable debug features: Instruction tracing and watchpoint enabled
• Testing and Debugging → Enable rich Log: Instruction tracing enabled

Build Configuration Modifications

Remove all -Werror flags from NEMU/scripts/build.mk:

CFLAGS  := -O2 -MMD -Wall $(INCLUDES) $(CFLAGS)
CXXFLAGS  := -O2 -MMD -Wall --std=c++17 $(XINCLUDES) $(CFLAGS)

Register Check Function Modification

Update isa_difftest_checkregs in NEMU/src/isa/riscv64/difftest/dut.c to check only gpr[32] and PC registers:

bool isa_difftest_checkregs(CPU_state *ref_r, vaddr_t pc) {
  csr_prepare();
#ifdef CONFIG_DIFFTEST_REF_SPIKE
  cpu.mip &= 0xffffff4f;
#endif
  if(cpu.mip != ref_r->mip) ref_r->mip = cpu.mip;
  if (memcmp(&cpu.gpr[1], &ref_r->gpr[1], ARRLEN(cpu.gpr) - sizeof(cpu.gpr[0])) || (cpu.pc != ref_r->pc)) {
    int i;
    for (i = 1; i < ARRLEN(cpu.gpr); i ++) {
      difftest_check_reg(reg_name(i, 4), pc, ref_r->gpr[i]._64, cpu.gpr[i]._64);
    }
    difftest_check_reg("pc", pc, ref_r->pc, cpu.pc);
    return false;
  }
#ifdef CONFIG_DIFFTEST_STORE_COMMIT
  return difftest_check_store(pc);
#else
  return true;
#endif
}

PMP Permission Check Update

Modify isa_pmp_check_permission in NEMU/src/isa/riscv64/system/mmu.c:

bool isa_pmp_check_permission(paddr_t addr, int len, int type, int out_mode) {
  return true;
}

Compilation

make -j`nproc`

The compiled executable riscv64-nemu-interpreter will be generated in NEMU/build/.

QEMU Socket Diff Compilation

ISA Configuraton

Set ISA to riscv64 in NEMU/scripts/isa.mk:

ISA ?= riscv64

Build Flags Modification

Add -ldl to LDFLAGS in NEMU/scripts/build.mk and update optimization flags:

CFLAGS  := -O0 -g -MMD -Wall $(INCLUDES) $(CFLAGS)
CXXFLAGS  := -O0 -g -MMD -Wall --std=c++17 $(XINCLUDES) $(CFLAGS)
LDFLAGS := -O0 -g $(LDFLAGS) -ldl

Header File Adjustments

Add #include <difftest.h> to NEMU/tools/qemu-socket-diff/include/common.h and remove it from NEMU/tools/qemu-socket-diff/src/diff-test.c.</difftest.h>

Register Structure Update

Remove fpr[32] from union isa_gdb_regs in NEMU/tools/qemu-socket-diff/include/isa/riscv64.h:

union isa_gdb_regs {
  struct {
    uint64_t gpr[32];
    uint64_t pc;
  };
  struct {
    uint32_t array[DIFFTEST_REG_SIZE/sizeof(uint32_t)];
  };
};

Register Retrieval Function

Update gdb_getregs in NEMU/tools/qemu-socket-diff/src/gdb-host.c:

bool gdb_getregs(union isa_gdb_regs *r) { 
  gdb_send(conn, (const uint8_t *)"g", 1);
  size_t size;
  uint8_t *reply = gdb_recv(conn, &size);

  int i;
  uint8_t *p = reply;
  uint8_t c;
  assert(size % (sizeof(uint64_t) * 2) == 0);
  assert((size / 2) <= sizeof(union isa_gdb_regs));
  for (i = 0; i < size / (sizeof(uint64_t) * 2); i ++) {
    c = p[16];
    p[16] = '\0';
    ((uint64_t*)(r->array))[i] = gdb_decode_hex_str(p);
    p[16] = c;
    p += 16;
  }
 
  free(reply);
  return true; 
}

Register Copy Function

Modify difftest_regcpy in NEMU/tools/qemu-socket-diff/src/diff-test.c:

void difftest_regcpy(void *dut, bool direction) {
  union isa_gdb_regs qemu_r;
  gdb_getregs(&qemu_r);
  int dut_pc_off = 32 * sizeof(uint64_t) * 2 + 18 * sizeof(uint64_t);
  if (direction == DIFFTEST_TO_REF) {
    memcpy(qemu_r.gpr, dut, (32 * sizeof(uint64_t)));
    memcpy(&(qemu_r.pc), (dut + dut_pc_off), sizeof(uint64_t));
    gdb_setregs(&qemu_r);
  } else {
    memcpy(dut, qemu_r.gpr, (32 * sizeof(uint64_t)));
    memcpy((dut + dut_pc_off), &(qemu_r.pc), sizeof(uint64_t));
  }
}

QEMU Instruction Logging

Update execlp call in difftest_init to enable instruction logging:

execlp(ISA_QEMU_BIN, ISA_QEMU_BIN, ISA_QEMU_ARGS "-S", "-gdb", buf, "-nographic", "-d", "in_asm", "-D", "./qemu.log", NULL);

Compilation

cd tools/qemu-socket-diff/
make -j`nproc`

The riscv64-qemu-so shared library will be generated in NEMU/tools/qemu-socket-diff/build.

Running DiffTest

Copy the following files to NEMU/build directory:

• riscv64-qemu-so
• benos_payload.bin
• qemu-system-riscv64
• opensbi-riscv64-generic-fw_dynamic.bin

Execute NEMU with DiffTest enabled:

./riscv64-nemu-interpreter -b benos_payload.bin -d ./riscv64-qemu-so -l nemu.log

NEMU will fork a process to launch QEMU and begin instruction-by-instruction comparison. Any discrepancies will be logged to nemu.log.

If encountering port conflicts, terminate existing QEMU processes:

kill $(ps aux | grep qemu | grep -v grep | awk '{print $2}')

Successful Execution

When both implementations produce identical results, BenOS will output "Welcome RISC-V!" via serial port, and nemu.log will confirm instruction matching.

Detection of Differences

When discrepancies are detected, execution halts and register differences are reported in nemu.log. For example:

pc is different after executing instruction at pc = 0x00000000800000a8, right = 0x0000000000000000, wrong = 0x0000000080200000

This indicates a mismatch in PC register values after mret instruction execution.