Intro to deep learning#
This is the course web-page for Introduction to Deep Learning and NLP at University of Bonn in spring 2023.
Last year has shown multiple breakthroughs in deep learning, bringing large language models to the mainstream. OpenAI’s ChatGPT, Microsoft’s new Bing Search and GitHub Copilot, and Deep Mind’s AlphaCode are the most prominent. While they still have many flaws, they show a potential to transform many sectors of the economy, replace some workers and make other vastly more productive.
NLP also has an immense potential to change research in economics. Most economists use small and expensive structured datasets. NLP offers a way to work with novel data sources that often can be scraped for free from the web. Examples are classifying speeches along the political spectrum, classifying tweets to measure opinions, extracting concepts mentioned in free-form survey replies, or translating questionnaires or datasets into different languages.
This class is an introduction to deep learning and NLP for economists. Starting from zero, the first half of the course focuses on learning the practical skills needed to incorporate NLP into empirical workflows. We will use Huggingface’s transformers library and only work with pre-trained models for this. The second half of the class zooms in and focuses on understanding what language models are, how they differ, and how they are trained. We will write some purely didactical code in numpy and implement a few simple models in PyTorch. The main focus of the second half is to build enough understanding to work effectively with pre-trained models. It is beyond our scope and computational resources to actually train large models.
Below you find some general materials and the materials for each lecture. Please make sure to thouroughly read the logistics and installation page before you approach us with any questions.