Module 2

Teachers

Pontus Giselsson - pontusg [at] control.lth.se (examiner)
Manu Upadhyaya - manu.upadhyaya [at] control.lth.se (teaching assistant)

Aim

This module is about implicit regularization of stochastic gradient descent in overparameterized deep learning. We will have three lectures:

Lectures

Lecture 1 - Intro and deep learning generalization (slides Download slides)

• Setting the stage - video
• Deep learning generalization aspects relevant for SGD - video
  
  • norm of weights
  • flatness of minima

Lecture 2 - Stochastic gradient descent (slides Download slides)

• Qualitative convergence behavior - video
• Escaping "bad" minima - video

Lecture 3 - Convergence to minimum norm solution (slides Download slides)

• Variable metric methods - video
• Convergence to minimum metric-norm solution - video

Assignment

The assignment is available here. Deadline is June 11.

Schedule

The schedule for May 18-19 is (starting time is sharp):

• May 18, 10am-12am:  Lecture 1 - https://lu-se.zoom.us/j/2834723313
• May 18, 2pm - 4pm:    Lecture 2 - https://lu-se.zoom.us/j/2834723313
• May 19, 10am-12am:  Exercise session- https://lu-se.zoom.us/j/2834723313
• May 19, 2pm - 4pm:    Lecture 3 - https://lu-se.zoom.us/j/2834723313