Sign in to track progress, unlock prep layers, and browse your pipeline.Sign in
Optimize CUDA GEMM with tiling and coalescing
CUDA Execution Model, Memory Hierarchy, and GEMM Kernel Design
Part 1: Concepts
Explain:
- CUDA execution model: grid, blocks, warps, threads (SIMT). Describe how threads map to hardware, warp scheduling, and synchronization.
- Memory hierarchy: registers, shared memory (including bank conflicts), global memory (including coalescing), L1/L2 caches, constant/texture memory, and local memory (including register spill).
- Occupancy: Define it, explain how it is limited by registers, shared memory, and the number of threads, and discuss why higher occupancy is not always better.
Part 2: Single-Precision GEMM Kernel (C = A × B)
Design and iteratively optimize an efficient single-precision General Matrix Multiplication (sgemm) kernel for row-major matrices. The optimization steps are:
- Start with a naive kernel where each thread computes one element C[i, j].
- Optimize the kernel using tiling and shared memory.
- Ensure coalesced global loads and stores.
- Avoid shared memory bank conflicts.
- Unroll inner loops.
- Utilize vectorized loads/stores (e.g.,
float4), including handling of data alignment. - Choose appropriate grid and block sizes, as well as tile shapes. Explain the trade-offs involved, particularly concerning register and shared memory pressure versus occupancy.
- Handle edge cases where matrix dimensions M, N, or K are not exact multiples of the chosen tile sizes.
- Compare the achieved throughput (in GFLOP/s) of your optimized kernel against a
cuBLAS sgemmbaseline.
Part 3: Overlap and Measurement
- Discuss strategies for overlapping host–device data transfers with compute operations, including CUDA streams, double buffering, pinned memory, and
cublasSetStream. - Explain how to measure performance metrics such as achieved occupancy, arithmetic intensity (FLOPs/byte), and memory bandwidth. Include relevant formulas and practical tools for measurement.
AI interview prep
Core dilemma
Maximizing GPU utilization and throughput for a compute-intensive operation like GEMM by meticulously managing the complex CUDA memory hierarchy, execution model, and data movement to approach theoretical peak performance.
Mid-level signals3
Senior signals3
Red flags3
3 tradeoffs and 3 stress tests locked
Interview prep
Unlock the interviewer X-Ray rubric, tradeoff matrix, stress tests, clarifying questions, and key concepts.
Optimize CUDA GEMM with tiling and coalescing
CUDA Execution Model, Memory Hierarchy, and GEMM Kernel Design
Part 1: Concepts
Explain:
- CUDA execution model: grid, blocks, warps, threads (SIMT). Describe how threads map to hardware, warp scheduling, and synchronization.
- Memory hierarchy: registers, shared memory (including bank conflicts), global memory (including coalescing), L1/L2 caches, constant/texture memory, and local memory (including register spill).
- Occupancy: Define it, explain how it is limited by registers, shared memory, and the number of threads, and discuss why higher occupancy is not always better.
Part 2: Single-Precision GEMM Kernel (C = A × B)
Design and iteratively optimize an efficient single-precision General Matrix Multiplication (sgemm) kernel for row-major matrices. The optimization steps are:
- Start with a naive kernel where each thread computes one element C[i, j].
- Optimize the kernel using tiling and shared memory.
- Ensure coalesced global loads and stores.
- Avoid shared memory bank conflicts.
- Unroll inner loops.
- Utilize vectorized loads/stores (e.g.,
float4), including handling of data alignment. - Choose appropriate grid and block sizes, as well as tile shapes. Explain the trade-offs involved, particularly concerning register and shared memory pressure versus occupancy.
- Handle edge cases where matrix dimensions M, N, or K are not exact multiples of the chosen tile sizes.
- Compare the achieved throughput (in GFLOP/s) of your optimized kernel against a
cuBLAS sgemmbaseline.
Part 3: Overlap and Measurement
- Discuss strategies for overlapping host–device data transfers with compute operations, including CUDA streams, double buffering, pinned memory, and
cublasSetStream. - Explain how to measure performance metrics such as achieved occupancy, arithmetic intensity (FLOPs/byte), and memory bandwidth. Include relevant formulas and practical tools for measurement.