| Date |
Lecturer |
Topics |
Slides |
| Jan 20 |
Minjia Zhang |
Course Introduction and Logistics |
pdf |
| Jan 22 |
Minjia Zhang |
Deep Learning Basics |
pdf |
| Jan 27 |
Minjia Zhang |
Transformers Deep Dive |
pdf |
| Jan 29 |
Minjia Zhang |
Arithmetic Intensity |
pdf |
| Feb 3 |
Minjia Zhang |
Distributed Training Overview, Parameter Server, Asynchronous Training
| pdf |
| Feb 5 |
Minjia Zhang |
All-Reduce Data Parallelism, Communication Terminologies |
pdf |
| Feb 10 |
Minjia Zhang |
Tensor Slicing Model Parallelism |
pdf |
| Feb 12 |
Minjia Zhang |
Pipeline Parallelism |
pdf |
| Feb 17 |
Minjia Zhang |
Multi-Dimensional Parallelism |
pdf |
| Feb 19 |
Minjia Zhang |
Mixed Precision Training |
pdf |
| Feb 24 |
Minjia Zhang |
Memory Optimization, Rematerialization |
pdf |
| Feb 26 |
Minjia Zhang |
Memory Optimization, Rematerialization (Cont.) |
pdf |
| March 3 |
Minjia Zhang |
ZeRO-style Data Parallelism |
pdf |
| March 5 |
Minjia Zhang |
Training with Heterogeneous Memory |
pdf |
| Mar 10 |
Zechun Liu |
Scaling Down: Optimizing Foundation Models for Edge Deployment |
|
| Mar 12 |
Yida Wang |
Push the science and maximize performance with AWS Trainium |
|
| Spring Break (March 16-20) |
| Mar 24 |
Minjia Zhang |
Project Feedback Session |
| Mar 26 |
Minjia Zhang |
Inference Overview (Part 1: System) |
pdf |
| Mar 31 |
Minjia Zhang |
Inference Overview (Part 2: Algorithm) |
pdf |
| April 2 |
Minjia Zhang |
LLM Inference Basic, Continuous Batching
|
pdf |
| Apr 6 |
Minjia Zhang |
Continus Batching
|
pdf |
| Apr 8 |
Minjia Zhang |
Continus Batching |
| Apr 9 |
Minjia Zhang |
Continus Batching |
| Apr 14 |
Minjia Zhang |
Paged Attention |
| Apr 16 |
Minjia Zhang |
Adaptive KV |
| Apr 21 |
Minjia Zhang |
Quantization |
| May 9 |
|
Final Project Due Date
|
|
-->