# Lecture 11

## Topic

In this lecture you will learn about transforrmers and how they revolutionized
natural language processing.


## Lecture Slides

```{raw} html
<iframe src="_static/bld/pdfs/lecture_11.pdf" width="700" height="415"></iframe>
```

```{eval-rst}
:download:`Download the slides <_static/bld/pdfs/lecture_11.pdf>`
```

## Exercises

There are no exercise notebooks for this lecture.


## Suggested Homework

- Read the illustrated transformer blogpost
- Read the PicoGPT source code

## Additional materials

### Pico GPT

[PicoGPT](https://github.com/jaymody/picoGPT/tree/29e78cc52b58ed2c1c483ffea2eb46ff6bdec785) is a minimal Python implementation of GPT style transformer models that implements the full transformer architecture in only 40 lines of instructional numpy code. This is very similar to how we implemented RNNs and MLPs in this lecture.


### Videos from cs224n at Stanford

You already know this lecture series from last week, but here is the video on transformers

<iframe width="560" height="315" src="https://www.youtube.com/embed/ptuGllU5SQQ" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

### GPT from scratch in Pytorch

You already know Andrej Karpathy from his great blogpost. He also has a video where he
implements transformers in Pytorch.

<iframe width="560" height="315" src="https://www.youtube.com/embed/kCc8FmEb1nY" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>