CUDA-Learn-Notes

What's Changed

• [HGEMM] Add MMA 16816 swizzle, Up to 115 TFLOPS by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/98

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.13...v2.4.15

What's Changed

• [Mat][Trans] Add f32/f32x4 row/col first kernel by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/89
• [Docs][Contribute] Add How to contribute Notes by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/90
• [HGEMM] optimize SMEM padding, up to 113 TFLOPS by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/92
• [Mat][Trans] Add f32x4_shared/bcf row/col first kernel. by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/91
• [Docs] rename mat_transpose -> mat-transpose by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/93
• [HGEMM] Add GeForce RTX 3080 Laptop benchmark by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/94
• [HGEMM] update HGEMM benchmark option by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/95
• [HGEMM] Refactor HGEMM WMMA 161616 kernels by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/96
• [HGEMM] Update HGEMM WMMA Benchmark by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/97

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.12...v2.4.13

What's Changed

• [SGEMM] SGEMM TF32 Thread Block Swizzle by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/84
• [HGEMM] mma4x4_warp4x4_stages with swizzle by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/86
• [SWISH] support Swish F32/F16 kernel by @wangzijian1010 in https://github.com/DefTruth/CUDA-Learn-Notes/pull/85
• [SGEMM] Update SGEMM TF32 Benchmark by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/87

New Contributors

• @wangzijian1010 made their first contribution in https://github.com/DefTruth/CUDA-Learn-Notes/pull/85

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.11...v2.4.12

What's Changed

• [Docs] Update README.md by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/81
• [HEGMM] HGEMM WMMA Thread Block Swizzle by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/82
• [HGEMM] make thread block swizzle stride as N/4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/83

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.10...v2.4.11

What's Changed

• [HGEMM] HGEMM WMMA Stage mma4x2+warp4x4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/76
• [SGEMM] Add SGEMM WMMA TF32 Stage2/3 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/77
• [SGEMM] Add cuBLAS SGEMM F32/TF32 baseline by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/78
• [SGEMM] Add Kernel cudaFuncSetAttribute hint by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/79
• [RoPE] Add minimal RoPE f32/f32x4 pack impl by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/80

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.9...v2.4.10

What's Changed

• [HGEMM] Add HGEMM WMMA Double Buffers by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/69
• [Embedding] Add embedding kernel f32/x4/x4_pack, f16/x8/x8_pack by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/68
• [HGEMM] Add HGEMM mma4x2, warp2x4x2 kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/70
• [HGEMM] HGEMM WMMA with Reg double buffers by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/71
• [HGEMM] Add HGEMM WMMA Stage 3/4 Kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/74
• [Softmax] Add online softmax f32x4 pack kernel by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/73
• [HEGMM][Bugfix] fix HGEMM Stage cp.async error by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/75

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.8...v2.4.9

What's Changed

• [GELU] Add f32/x4, f16/x2/x8/x8pack kernel. by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/66
• [HGEMM] HGEMM Tensor Cores Support Part-1 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/67

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.7...v2.4.8

What's Changed

• [SGEMM][Async] Add naive copy async SGEMM by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/64
• [SGEMM][Async] Add K16 + Copy Async Kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/65

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.6...v2.4.7

What's Changed

• [Softmax] Add online softmax according to Nvidia Paper by @bear-zd in https://github.com/DefTruth/CUDA-Learn-Notes/pull/60
• [HGEMM][Async] support K16/32 pack+cp.async+dbuf by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/62
• [Softmax][Bugfix] fixed softmax compile error by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/63

New Contributors

• @bear-zd made their first contribution in https://github.com/DefTruth/CUDA-Learn-Notes/pull/60

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.5...v2.4.6

What's Changed

• [FlashAttention] Refactor FlashAttention PyTorch bindings by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/55
• [SGEMM] test bank conflicts free with smem offset by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/56
• [HGEMM] HEGMM kernel with double buffers by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/57
• [Docs] Add docs for HGEMM/SGEMM double buffers by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/58
• [HGEMM] Add PyTorch HGEMM profile by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/59

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.4...v2.4.5

What's Changed

• [SGEMM] Add naive sgemm kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/51
• [SGEMM] bank conflicts free & double buffers by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/52
• [Misc][Benchmark] optimize benchmarks by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/53
• [HGEMM] Pack sliced_k f16x4/fp16x8 HGEMM by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/54

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.3...v2.4.4

What's Changed

• [LayerNorm][FP16] support fp16x8_pack_f32 kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/48
• [Softmax][FP16] Pack f16x8 softmax kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/49

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.2...v2.4.3

What's Changed

• [RMSNorm][FP16] Pack f16x8 rmsnorm by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/47

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4.1...v2.4.2

What's Changed

• [Nsight] Add nsys/ncu usage, ptx/sass by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/44
• [DotProd][FP16] support f16x8_pack kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/45
• [LayerNorm][FP16] Add pack support for f16x8 LD/ST by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/46

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.4...v2.4.1

What's Changed

• [Reduce][Kernel] Pack f16/bf16x8 & fp8/i8x16 LD/ST by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/43

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.3.1...v2.4

What's Changed

• [FA2][Half] Add FA2 f16_mma_m16n8k16 kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/35
• [Refactor][7/N] CUDA Learn Notes refactor Part-7 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/36
• Clamped input range in Sigmoid kernel to prevent overflow by @Phoenix8215 in https://github.com/DefTruth/CUDA-Learn-Notes/pull/37
• [Sigmoid][F16] Add f16x8_pack kernel, boost 1.5x ~ by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/39
• [Elementwise][Half] support f16x8_pack kernel, boost 1.1x by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/40
• [FlashAttention] replace FLOAT4 with LDST128BITS macro by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/41
• [RELU][FP16] Add f16x8_pack kernel, boost 2.1x by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/42

New Contributors

• @Phoenix8215 made their first contribution in https://github.com/DefTruth/CUDA-Learn-Notes/pull/37

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.3...v2.3.1

What's Changed

• [Refactor][6/N] CUDA Learn Notes refactor Part-6 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/17
• [Refactor][5/N] CUDA Learn Notes refactor Part-6 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/18
• [LayerNorm][Half] support fp16x8 packed LayerNorm by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/19
• [Reduce][Half] add HALF2 & BFLOAT2 macro by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/21
• [RMSNorm][Half] support fp16x8 packed RMSNorm by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/22
• [Bugfix][Kernel] fixed some kernel blocks calculate errors by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/23
• [Elementwise][Half] support fp16x8 packed Elementwise by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/24
• [Elementwise][Half] support fp16x8 packed Elementwise by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/25
• [RELU][Half] support fp16x8 RELU kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/26
• [RMSNorm] support f16x8_f32 RMSNorm by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/28
• [RMSNorm][Kernel] Add FLOAT2/HALF2_VARIANCE macro by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/29
• [LayerNorm][Kernel] Add HALF2 SUM/SUB/VAR macro by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/30
• [HGEMM] Add slicked_k&t_8x8_sliced_k_f16x4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/31
• [HGEMV][Half] support hgemv k32/k128/f16 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/32
• [FlashAttention] Refactor flash_attn_1_fwd_f32 kernel by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/33
• Bump up to v2.3 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/34

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.2...v2.3

What's Changed

• [Refactor][5/N] CUDA Learn Notes refactor Part-5 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/15
• Bump up to v2.2 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/16

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/2.1...v2.2

What's Changed

• [Refactor][4/N] CUDA Learn Notes refactor Part-4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/10
• [Refactor][4/N] CUDA Learn Notes refactor Part-4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/11
• [Refactor][4/N] CUDA Learn Notes refactor Part-4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/12
• [Refactor][4/N] CUDA Learn Notes refactor Part-4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/13
• [Refactor][4/N] CUDA Learn Notes refactor Part-4 by @DefTruth in https://github.com/DefTruth/CUDA-Learn-Notes/pull/14

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v2.0...2.1

Full Changelog: https://github.com/DefTruth/CUDA-Learn-Notes/compare/v0.8...v2.0