Modern GPUs from various manufacturers share similar core architectures despite differences in hardware and software implementations. The NVIDIA ecosystem, being closed-source, presents challenges in understanding GPU scheduling strateiges. This analysis covers three key aspects: the CUDA programming model, GPU hardware architecture, and CUDA scheduling framework.

CUDA programs involve both host (CPU) and device (GPU) components. The host manages data and control flow, while the device handles parallel computations. Kernels, marked with __global__, define paralel tasks executed on the GPU. Threads are organized into blocks and grids for parallel execution.

__global__ void vectorAdd(float *A, float *B, float *C, int N) {
    int idx = blockIdx.x * blockDim.x + threadIdx.x;
    if (idx < N) C[idx] = A[idx] + B[idx];
}

Thread organization follows a hierarchy:

• Grid: All threads for a kernel
• Block: Thread groups within a grid
• Thread: Basic execution unit

Hardware mapping:

• Blocks execute on Streaming Multiprocessors (SMs)
• Threads correspond to CUDA cores
• Warps (32-thread groups) are the scheduling units

Key scheduling concepts:

1. Streams: Enable concurrent kernel execution

cudaStream_t stream;
cudaStreamCreate(&stream);
kernel<<<blocks, threads, 0, stream>>>(...);
cudaStreamDestroy(stream);

2. Warp Scheduling: SM switches between warps to hide latency
3. Occupancy: Measure of active warps per SM

GPU scheduling frameworks include:

• Time slicing for resource sharing
• Multi-Process Service (MPS) for concurrent execution
• Multi-Instance GPU (MIG) for hardware partitioning