Relevant parts of Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, http://www.deeplearningbook.org (Links to an external site.),

Lecture 1
- Chapter 1: an introduction to deep learning and some history
- Chapter 5-5.1, 5.7.1: machine learning basics.
- Chapter 5.10: building a machine learning algorithm.
Lecture 2
- Maximum likelihood estimation, Chapter 5.5.
- Chapter 6-6.4: Deep feedforward networks.
- Empirical risk minimization and surrogate losses, Chapter 8-8.1.2.
- To understand the term cross-entropy (and learn about other important concepts): Chapter 3.13.
- Gradient descent optimization: Chapter 4.3 (you can skip 4.3.1).
Lecture 3
- Chapter 5.9, 6.5, 8.1.3, 8.3, 8.5
Lecture 4
- Chapter 5.2, 5.3, 7.1 and 8.7
Lectures 5 and 6
- Chapter 9, except 9.6 and 9.8-9.10

We don't have slides from the videos by Andrew Ng, but slides from most of the other videos are listed here:

We believe it can be valuable to look at different explanations of the same material. We have therefore collected a list of resources that you may find interesting. Note that we do not expect you to look at them and that they are merely included for those who are interested.

Entire courses

A free online course on deep learning: http://www.fast.ai (Links to an external site.).
The Stanford course on CNNs: http://cs231n.stanford.edu (Links to an external site.)
A course from NYU: https://atcold.github.io/pytorch-Deep-Learning/ (Links to an external site.)
A course from EPFL: https://fleuret.org/ee559/ (Links to an external site.)

YouTube

Deep learning is a popular topic and many people are producing instruction videos. Here is a selected list:

Transformers

Even though transformers is beyond the scope of this module, it is reasonable to mention them as an important alternative (or complement) to CNNs for computer vision. Below are a small number of references that you may find inspiring.

Attention is all you need: a landmark paper that introduced transformers as an architecture for machine translation. Lennart has recorded a series of videos on transformers that carefully introduces and motivates the architecture.
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale: demonstrates that transformers can perform at par with the state of the art CNNs on image classification (without involving any convolutions).
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation: one of many papers that leverages on transformers to do semantic/instance/panoptic segmentation.
End-to-end object detection with transformers: a novel architecture for object detection using transformers.

Even though many recent papers on computer vision involve transformers, most of them also make use of CNNs. Learning about CNNs therefore remains a natural first step when studying deep learning for computer vision.

Miscellaneous

Two pages that contain nice visualisations of CNNs:
- A cheat sheet for various aspects of CNNs.
- Explanations and illustrations of padding, strides, transposed convolutions, etc: PDF and webpage.
A compilation of online demonstrations: https://p.migdal.pl/interactive-machine-learning-list/ (Links to an external site.)
If you want to stay tuned with recent developments in deep learning, you can for instance use
- Twitter list with machine learning researchers and profiles (see Members/Medlemmar): twitter.com/DL_ML_Loop/lists/deep-learning-loop (Links to an external site.)
- arXiv alerts
- Quality discussion forum (subreddit) for machine learners: reddit.com/r/MachineLearning/ (Links to an external site.)
- sign up for a weekly newsletter such as https://www.deeplearningweekly.com/ or https://us13.campaign-archive.com/home/?u=67bd06787e84d73db24fb0aa5&id=6c9d98ff2c (Links to an external site.) (Links to an external site.)
Chris Olah has written several illuminating posts on the fundamentals of neural networks, backprop, LSTMs, etc. http://colah.github.io (Links to an external site.)
Another popular free book on deep learning: neuralnetworksanddeeplearning.com (Links to an external site.)
Vector, Matrix, and Tensor Derivatives: compsci697l.github.io/docs/vecDerivs.pdf (Links to an external site.)
Sebastian Ruder's blog contains several well written texts, e.g., a summary of various methods for optimization in deep learning, http://ruder.io/optimizing-gradient-descent/. (Links to an external site.)