Lecture Schedule and Material

Below is a summary of the lecture topics and links to the lecture slides. I will try and make all slides available before the lecture begins.  We might vary the order of the lecture topics (probability of this happening is larger for the later lectures). The topics of Lectures 1-5 are fairly set though.

I am not recording the lectures however the lectures I recorded last year are available on this page: Videos of DD2424 lectures 2021

Lecture 1

Title:  The Deep Learning Revolution

Date & Time: Tuesday, March 22, 08:00-10:00 

Location: Alfvénsalen

More details and material

 

• Topics covered: 

  • Review of the impact deep networks have had in the application fields of speech, computer vision and NLP. 
  • Review of the course's syllabus.
  • Review of the course's assignments, project and assessment.

  Slides:  Lecture1.pdf Download Lecture1.pdf , Lecture1_CourseAdmin.pdf Download Lecture1_CourseAdmin.pdf 

  Video of the lecture from last year:DD2424 2021 videos available on this page

  Interesting extra reading material:

  State of AI Report 2021 by Nathan Benaich and Ian Hogarth available at https://www.stateof.ai/. Documenting giving an overview of the current start of AI. I mention the document in the lecture.

 

Lecture 2

Title:  Learning Linear Binary & Linear Multi-class Classifiers from Labelled Training Data

 (mini-batch gradient descent optimization applied to "Loss + Regularization" cost functions)

Date & Time: Wednesday, March 23, 15:00-17:00  

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Supervised learning  =  minimizing loss + regularization.
  • Learning linear classifiers.
  • Binary SVM classifiers as an unconstrained optimization problem.
  • Gradient Descent, SGD, mini-batch optimization.  
  • Multi-class classification with one layer networks.
  • Different loss-functions.

  Slides:  Lecture2.pdf Download Lecture2.pdf>

  Video of the lecture from last year:

  Suggested Reading Material: Sections 5.1.4, 5.2, 5.2.2, 5.7.2 from "Deep Learning" by Goodfellow, Bengio and Courville. Link to Chapter 5 of Deep Learning (Links to an external site.) Links to an external site.

  Sections 8.1.3, 8.3.1 from the book give amore detailed description and analysis of mini-batch gradient descent and SGD than given in the lecture notes. Link to Chapter 8 of Deep Learning (Links to an external site.) Links to an external site. .

  The suggested readings from chapter 5 should be familiar to those who have already taken courses in ML. Lecture 2 should be more-or-less self-contained. But the reading material should flesh out some of the concepts referred to in the lecture.

  During the lecture generalization, over-paramaterized models and double descent came up. Here are some slides I made for another lecture showing the phenomenon: double_descent_slides.pdf Download double_descent_slides.pdf

 

Lecture 3

Title:  Back Propagation 

Date & Time: Friday, March 25, 13:00-15:00 

Location: Alfvénsalen Links to an external site.

More details and material

• Topics covered: 

  • Chain rule of differentiation.
  • Chain rule for vector inputs and outputs
  • Computational graph.
  • Back propagation (In more detail then you probably ever expected!)

  Slides: Lecture3.pdf Download Lecture3.pdf

  Suggested Reading Material: Nice review of the back-prop algorithm written by Boaz Barak Yet another backpropagation tutorial Links to an external site.

 

Lecture 4

Title:  k-layer Neural Networks 

Date & Time: Tuesday, March 29, 08:00-10:00  

Location: Alfvénsalen

More details and material

• Topics covered: 

  • k-layer Neural Networks.
  • Activation functions.
  • Backprop for k-layer neural networks.
  • Problem of vanishing and exploding gradients.
  • Importance of careful initialization of network's weight parameters.
  • Batch normalization + Backprop with Batch normalisation

  Slides:  Lecture4.pdf Download Lecture4.pdf

 

Lecture 5

Title:  Training & Regularization of Neural Networks 

Date & Time: Wednesday, March 30, 15:00-17:00

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Variations of gradient descent.
  • Learning rate schedulers.
  • Regularization - Dropout
  • Practicalities of training neural networks and hyper-parameter optimization.

  Slides:  Lecture5.pdf Download Lecture5.pdf

 

Lecture 6

Title:  All about Convolutional Networks 

Date & Time:  Tuesday,  April 5, 08:00-10:00  

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Details of the convolution layer in Convolutional Networks.
  • Gradient computations for a convolutional layers.
  • Common operations in ConvNets - max-pooling etc

  Slides:  Lecture6.pdf Download Lecture6.pdf

 

Lecture 7

Title:  Even more about ConvNets   

Date & Time: Wednesday,  April 6, 15:00-17:00

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Review of the modern top performing deep ConvNets - AlexNet, VGGNet, GoogLeNet, ResNet
  • Practicalities of training deep neural networks - data augmentation, transfer learning and stacking convolutional filters.

  Slides: Lecture7.pdf Download Lecture7.pdf

 

Lecture 8

Title: Deep Learning Frameworks and Computational Facilities  

Date & Time: Tuesday, Apr 12, 08:00-10:00

Location: Alfvénsalen

Part 1 - will be a normal lecture from 08:15-09:00

Part 2 - Will be a live Q&A session from 09:15 - 10:00 with Sebastian Bujwid about Deep Learning Libraries. It is assumed that you will have watched the provided videos before the Q&A session.

 

More details and material

• Topics covered: 

  • Part 1 - The Transposed Convolution operation, Bottleneck ConvNets - Semantic Segmentation
  • Part 2 - Deep Learning Libraries

  Slides Part 1: Lecture8.pdf Download Lecture8.pdf

  Slides Part 2: DeepLearningLibraries.pdf Download DeepLearningLibraries.pdf

  Videos to accompany Part 2: 

  • Part 1 - Introduction:

  • Part 2- AutoDiff:

  • Part 3 - Code Examples:

  • Part 4 - Training Efficiency:

  • Part 5 - Technical Details:

  • Part 6 - Using Google Clound Platform:

 

Lecture 9

Title:  Networks for Sequential Data: RNNs & LSTMs 

Date & Time:  Wednesday, April 13, 15:00-17:00

Location: Alfvénsalen

More details and material

• Topics covered: 

  • RNNs
  • Back-prop for RNNs
  • RNNs for synthesis problems.
  • Problem of exploding and vanishing gradients in RNN.
  • LSTMs

  Slides: Lecture9.pdf Download Lecture9.pdf

 

 

Lecture 10

Title: Translation, Attention, Self-Attention

Date & Time:  Tuesday,  April 26, 08:00-10:00

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Neural Machine Translation
  • Sequence to Sequence
  • Seq2Seq with attention
  • Self-attention

  Slides:  Lecture10.pdf Download Lecture10.pdf

 

Lecture 11

Title: Transformer Networks

Date & Time:  Wednesday, April 27, 15:00-17:00

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Self-Atttention
  • Transformer encoder-decoder
  • Pre-training language models for NLP
  • Transformers for Computer vision

  Slides:  Lecture11.pdf Download Lecture11.pdf

 

Lecture 12

Title: How to generate realistic images using deep learning?  

Date & Time:  Tuesday, May 3, 08:00-10:00

Location: Alfvénsalen

Slides: DGMs.pdf Download DGMs.pdf

Will be a live Q&A session with Hossein Azizpour. It is assumed that you will have watched the provided videos before the Q&A session.

More details and material

• Topics covered: 

  • Introduction to (Deep) Generative Modeling
  • Variational Auto-encoders
  • Generative Adversarial Networks
  • Brief mention of other methods (PixelCNN, Glow)

  Slides: DGMs.pdf Download DGMs.pdf

  Video: 

   

 

Lecture 13

Title:  Self-supervised learning & Adversarial examples

Date & Time: Wednesday, May 4, 15:00-17:00

Location: Alfvénsalen

More details and material

• Topics covered: 

  • Motivation for self-supervised learning
  • Pre-text tasks
  • Contrastive Representative Learning
  • SimCLR & MoCo

  Slides:Lecture13.pdf Download Lecture13.pdf

  Few Extra Slides on Adversarial examples: AdversarialExamplesSlides.pdf Download AdversarialExamplesSlides.pdf

  Interesting links: 

  • I got a question on whether Transformer networks are more robust to adversarial examples than ConvNets. There is a flurry of research in this area at the moment. Here is a link to a NeurIPs 2021 paper (the answer is somewhat nuanced as some simple design choices in the architecture like choice of activation function can have a large impact on the ability protect against adversarial attacks). Please check out

    Are Transformers More Robust Than CNNs? Links to an external site. by Yutong Bai, Jieru Mei, Alan Yuille, and Cihang Xie, NeurIPs, 2021

  Slides from the Astra Zeneca presentation:AZ Deep Learning Applications.pptx Download AZ Deep Learning Applications.pptx