CUDA & GPU Programming

The core NVIDIA-candidate course: CUDA's programming model, memory hierarchy, coalescing, kernels, profiling, and matrix multiplication optimization.

Lesson 1: What is a GPU?
Lesson 2: Why a GPU? Throughput vs Latency
Lesson 3: Your First Kernel — __global__ and <<<>>>
Lesson 4: Sizing the Grid — Enough Threads for N
Lesson 5: Indexing: threadIdx, blockIdx, blockDim
Lesson 6: Two Separate Memories — Host, Device & cudaMalloc
Lesson 7: Host/Device Memory: cudaMalloc, cudaMemcpy
Lesson 8: Grid-Stride Loops & Arbitrary N
Lesson 9: 2D Grids & Blocks
Lesson 10: Warps & SIMT: the 32-thread unit
Lesson 11: Warp Divergence
Lesson 12: Error Handling & Synchronization
Lesson 13: The Memory Hierarchy
Lesson 14: Global Memory Coalescing — the concept
Lesson 15: Diagnosing & Fixing Strided Access
Lesson 16: Shared Memory Basics
Lesson 17: __syncthreads() and Race Conditions
Lesson 18: Shared Memory Bank Conflicts
Lesson 19: Constant Memory & Registers
Lesson 20: Parallel Reduction: tree-based sum
Lesson 21: Optimizing Reduction: sequential addressing
Lesson 22: Global Memory Races — Why Atomics
Lesson 23: Atomic Operations: atomicAdd & histograms
Lesson 24: Parallel Scan (Prefix Sum) — the Idea
Lesson 25: Parallel Scan — the Doubling Kernel
Lesson 26: Matrix Multiply — the Naive GPU Kernel
Lesson 27: Tiled Matrix Multiplication
Lesson 28: Occupancy: active warps per SM
Lesson 29: Bandwidth & Arithmetic Intensity
Lesson 30: The Roofline Model
Lesson 31: Profiling with Nsight
Lesson 32: Streams & Concurrency
Lesson 33: Async Transfers & Pinned Memory
Lesson 34: Unified Memory with cudaMallocManaged
Lesson 35: cuBLAS, Thrust & When Not to Write a Kernel
Lesson 36: Capstone — End-to-End Kernel Optimization