| Date |
Lecturer |
Topics |
Slides |
| Jan 20 |
Minjia Zhang |
Course Introduction and Logistics |
pdf |
| Jan 22 |
Minjia Zhang |
Deep Learning Basics |
pdf |
| Jan 27 |
Minjia Zhang |
Transformers Deep Dive |
pdf |
| Jan 29 |
Minjia Zhang |
Arithmetic Intensity |
pdf |
| Feb 3 |
Minjia Zhang |
Distributed Training Overview, Parameter Server, Asynchronous Training
| pdf |
| Feb 5 |
Minjia Zhang |
All-Reduce Data Parallelism, Communication Terminologies |
pdf |
| Feb 10 |
Minjia Zhang |
Tensor Slicing Model Parallelism |
pdf |
| Feb 12 |
Minjia Zhang |
Pipeline Parallelism |
pdf |
| Feb 17 |
Minjia Zhang |
Multi-Dimensional Parallelism |
pdf |
| Feb 19 |
Minjia Zhang |
Mixed Precision Training |
pdf |
| Feb 24 |
Minjia Zhang |
Memory Optimization, Rematerialization |
| Feb 26 |
Minjia Zhang |
ZeRO-style Data Parallelism |
| March 1 |
Minjia Zhang |
Training with Heterogeneous Memory |
| Mar 3 |
Minjia Zhang |
Course Project Proposal Feedback |
|
| Mar 5 |
Minjia Zhang |
Course Project Proposal Feedback |
|
| Mar 10 |
(Guest Lecture) |
TBD |
|
| Mar 12 |
(Guest Lecture) |
TBD |
|
| Spring Break (March 14-22) |
| Mar 24 |
Minjia Zhang |
Inference Overview |
| Mar 26 |
Minjia Zhang |
GPU Memory Hierarchy, FlashAttention Part 1 |
| March 31 |
Minjia Zhang |
FlashAttention Part 2, LLM Inference
|
| Apr 2 |
Minjia Zhang |
LLM Batching
|
| Apr 7 |
Minjia Zhang |
Continus Batching |
| Apr 9 |
Minjia Zhang |
Continus Batching |
| Apr 14 |
Minjia Zhang |
Paged Attention |
| Apr 16 |
Minjia Zhang |
Adaptive KV |
| Apr 21 |
Minjia Zhang |
Quantization |
| May 9 |
|
Final Project Due Date
|
|
-->