Lecture Schedule & Material

Topics covered: 

• Review of the impact deep networks have had in the application fields of speech, computer vision and NLP. 
• Review of the course's syllabus.
• Review of the course's assignments, project and assessment.

Slides: Lecture1.pdf Download Lecture1.pdf, Lecture1_CourseAdmin.pdf Download Lecture1_CourseAdmin.pdf

Video of the lecture from 2021:DD2424 2021 videos available on this page

Interesting extra reading material:

Topics covered: 

• Supervised learning  =  minimizing loss + regularization.
• Learning linear classifiers.
• Binary SVM classifiers as an unconstrained optimization problem.
• Gradient Descent, SGD, mini-batch optimization.  
• Multi-class classification with one layer networks.
• Different loss-functions.

Slides:  Lecture2.pdf Download Lecture2.pdf

Video of the lecture from 2021: DD2424 2021 videos available on this page

Suggested Reading Material: Sections 5.1.4, 5.2, 5.2.2, 5.7.2 from "Deep Learning" by Goodfellow, Bengio and Courville. Link to Chapter 5 of Deep Learning (Links to an external site.) Links to an external site.

Sections 8.1.3, 8.3.1 from the book give amore detailed description and analysis of mini-batch gradient descent and SGD than given in the lecture notes. Link to Chapter 8 of Deep Learning (Links to an external site.) Links to an external site. .

The suggested readings from chapter 5 should be familiar to those who have already taken courses in ML. Lecture 2 should be more-or-less self-contained. But the reading material should flesh out some of the concepts referred to in the lecture.

During the lecture generalization, over-paramaterized models and double descent came up. Here are some slides I made for another lecture showing the phenomenon: double_descent_slides.pdf

Topics covered: 

Slides: Lecture3.pdf Download Lecture3.pdf

Suggested Reading Material:

• Nice review of the back-prop algorithm written by Boaz Barak Yet another backpropagation tutorial
• Overview of PyTorch Autograd Engine - Link to a blog post describing how PyTorch does automatic gradient calculations under the hood - i.e. how it does the Jacobian and gradient matrix multiplications we descriptions in the lecture in an efficient way w.r.t. memory and computation.

• Variations of gradient descent.
• Learning rate schedulers.
• Regularization - Dropout
• Practicalities of training neural networks and hyper-parameter optimization.

• Details of the convolution layer in Convolutional Networks.
• Gradient computations for a convolutional layers.
• Common operations in ConvNets - max-pooling etc

• Improving neural networks by preventing co-adaptation of feature detectors  Links to an external site.- In the lecture got a question about co-adaptation. This linked blog could help explain the issue.

• Review of the modern top performing deep ConvNets - AlexNet, VGGNet, GoogLeNet, ResNet
• Practicalities of training deep neural networks - data augmentation, transfer learning and stacking convolutional filters.

• RNNs
• Back-prop for RNNs
• RNNs for synthesis problems.
• Problem of exploding and vanishing gradients in RNN.
• LSTMs

• Neural Machine Translation
• Sequence to Sequence
• Seq2Seq with attention
• Self-attention

• Self-Atttention review
• Transformer encoder-decoder
• Pre-training language models for NLP
• Transformers for Computer vision

• Motivation for self-supervised learning
• Pre-text tasks
• Contrastive Representative Learning
• SimCLR & MoCo

• High-level review of image generation
• High-level review of image generation via diffusion
• Adversarial examples