Lecture 11#
Topic#
In this lecture you will learn about transforrmers and how they revolutionized natural language processing.
Lecture Slides#
Exercises#
There are no exercise notebooks for this lecture.
Suggested Homework#
Read the illustrated transformer blogpost
Read the PicoGPT source code
Additional materials#
Pico GPT#
PicoGPT is a minimal Python implementation of GPT style transformer models that implements the full transformer architecture in only 40 lines of instructional numpy code. This is very similar to how we implemented RNNs and MLPs in this lecture.
Videos from cs224n at Stanford#
You already know this lecture series from last week, but here is the video on transformers
GPT from scratch in Pytorch#
You already know Andrej Karpathy from his great blogpost. He also has a video where he implements transformers in Pytorch.