Prerequisites

Install CUDA Toolkit and cuDNN. Download the prebuilt ONNX Runtime binaries that match your CUDA major version—for example, use the CUDA 12.x package when your toolkits 12.1. Extract the archive and register the library folder in your system PATH.

For GPU execution, the following dynamic libraries must be visible to your application at runtime. Place them in the same directory as you're compiled binary to prevent Windows from resolving an incompatible copy in System32:

• onnxruntime.dll
• onnxruntime_providers_shared.dll
• onnxruntime_providers_cuda.dll

In Visual Studio, add the ORT include directory to C++ > General > Additional Include Directories, the lib directory to Linker > General > Additional Library Directories, and link against onnxruntime.lib.

Session Initialization

#include <onnxruntime_cxx_api.h>
#include <cuda_provider_factory.h>

Ort::Env ort_environment(ORT_LOGGING_LEVEL_WARNING, "deploy");

Ort::SessionOptions session_cfg;
uint32_t cuda_device = 0;
Ort::ThrowOnError(OrtSessionOptionsAppendExecutionProvider_CUDA(session_cfg, cuda_device));

std::wstring model_path = L"model.onnx";
Ort::Session infer_session(ort_environment, model_path.c_str(), session_cfg);

The Ort::Env object is process-wide, whereas multiple Ort::Session instances may be created to load distinct networks.

Input Preparation

Verify exact input node names, shapes, and types with a model viewer such as Netron. The snippet below prepares a float32 NCHW image and a single int32 auxiliary value.

#include <opencv2/dnn.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/imgproc.hpp>
#include <vector>

cv::Mat frame = cv::imread("sample.jpg");
cv::Mat resized;
cv::resize(frame, resized, cv::Size(224, 224), 0, 0, cv::INTER_AREA);

cv::Mat chw;
cv::dnn::blobFromImage(resized, chw, 1.0 / 255.0, cv::Size(), cv::Scalar(), true, false, CV_32F);

size_t pixel_count = static_cast<size_t>(chw.total() * chw.channels());
std::vector<float> image_data(pixel_count);
std::memcpy(image_data.data(), chw.ptr<float>(), pixel_count * sizeof(float));

auto allocator = Ort::MemoryInfo::CreateCpu(OrtArenaAllocator, OrtMemTypeDefault);

std::vector<int64_t> image_shape = {1, 3, 224, 224};
Ort::Value image_input = Ort::Value::CreateTensor<float>(
    allocator, image_data.data(), image_data.size(), image_shape.data(), image_shape.size());

std::vector<int64_t> aux_shape = {1};
std::vector<int32_t> aux_value = {0};
Ort::Value aux_input = Ort::Value::CreateTensor<int32_t>(
    allocator, aux_value.data(), aux_value.size(), aux_shape.data(), aux_shape.size());

std::vector<const char*> input_nodes = {"input", "s_id"};
std::vector<const char*> output_nodes = {"logits"};

std::vector<Ort::Value> inputs;
inputs.emplace_back(std::move(image_input));
inputs.emplace_back(std::move(aux_input));

Ort::Value does not support copy assignment; transfer ownership with std::move or emplace_back.

Inference

std::vector<Ort::Value> outputs = infer_session.Run(
    Ort::RunOptions{nullptr},
    input_nodes.data(),
    inputs.data(),
    inputs.size(),
    output_nodes.data(),
    output_nodes.size()
);

Reading Results

float* logits = outputs.front().GetTensorMutableData<float>();
auto shape_info = outputs.front().GetTensorTypeAndShapeInfo();
size_t elements = static_cast<size_t>(shape_info.GetElementCount());
for (size_t i = 0; i < elements; ++i) {
    std::printf("%f\n", logits[i]);
}

Troubleshooting

Ort::Global::api_ is null This occurs when Windows loads a conflicting onnxruntime.dll from System32. Copy the three provider DLLs into your executable folder so the loader resolves them locally first.

ThrowStatus exceptions CUDA kernel failures, tensor type mismatches, or null environment handles typically trigger this. Isolate the cause by catching Ort::Exception and logging the embedded message, or by stepping through the call stack during a debug session.