Reading directions and slides from videos
Relevant parts of Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning, MIT Press, 2016, http://www.deeplearningbook.org (Links to an external site.),
- Lecture 1
- Chapter 1: an introduction to deep learning and some history
- Chapter 5-5.1, 5.7.1: machine learning basics.
- Chapter 5.10: building a machine learning algorithm.
- Lecture 2
- Maximum likelihood estimation, Chapter 5.5.
- Chapter 6-6.4: Deep feedforward networks.
- Empirical risk minimization and surrogate losses, Chapter 8-8.1.2.
- To understand the term cross-entropy (and learn about other important concepts): Chapter 3.13.
- Gradient descent optimization: Chapter 4.3 (you can skip 4.3.1).
- Lecture 3
- Chapter 5.9, 6.5, 8.1.3, 8.3, 8.5
- Lecture 4
- Chapter 5.2, 5.3, 7.1 and 8.7
- Lectures 5 and 6
- Chapter 9, except 9.6 and 9.8-9.10
We don't have slides from the videos by Andrew Ng, but slides from most of the other videos are listed here:
- Lecture 2: L2 Opt. based and cross-entropy.pdf Download L2 Opt. based and cross-entropy.pdf
- Lecture 3: L3 Gradient descent and backprop.pdf Download L3 Gradient descent and backprop.pdf
- Lecture 5: L5.pdf Download L5.pdf
- Lecture 6: L6.pdf Download L6.pdf
We believe it can be valuable to look at different explanations of the same material. We have therefore collected a list of resources that you may find interesting. Note that we do not expect you to look at them and that they are merely included for those who are interested.
Entire courses
- A free online course on deep learning: http://www.fast.ai (Links to an external site.).
- The Stanford course on CNNs: http://cs231n.stanford.edu (Links to an external site.)
- A course from NYU: https://atcold.github.io/pytorch-Deep-Learning/ (Links to an external site.)
- A course from EPFL: https://fleuret.org/ee559 (Links to an external site.)
YouTube
Deep learning is a popular topic and many people are producing instruction videos. Here is a selected list:
- In the course, we use many videos from Andrew Ng's courses on Coursera. You can find more videos in the playlists in the Deeplearning.ai channel: https://www.youtube.com/channel/UCcIXc5mJsHVYTZR1maL5l9w (Links to an external site.)
- 3Blue1Brown (Grant Sanderson) produces beautifully animated videos on math and science (highly recommended). The basics of deep learning is covered in a short series of videos: But what is a Neural Network?
- Geoffrey Hinton also has a neural networks course on Coursera that you can find on YouTube.
- Yannic Kilcher has a channel Links to an external site. where he explains recent papers.
- If you feel that instruction videos tend to be slow, Henry AI labs Links to an external site. might be an excellent channel to follow.
Transformers
Even though transformers is beyond the scope of this module, it is reasonable to mention them as an important alternative (or complement) to CNNs for computer vision. Below are a small number of references that you may find inspiring.
- Attention is all you need Links to an external site.: a landmark paper that introduced transformers as an architecture for machine translation. Lennart has recorded a series of videos on transformers Links to an external site. that carefully introduces and motivates the architecture.
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Links to an external site.: demonstrates that transformers can perform at par with the state of the art CNNs on image classification (without involving any convolutions).
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation Links to an external site.: one of many papers that leverages on transformers to do semantic/instance/panoptic segmentation.
- End-to-end object detection with transformers Links to an external site.: a novel architecture for object detection using transformers.
Even though many recent papers on computer vision involve transformers, most of them also make use of CNNs. Learning about CNNs therefore remains a natural first step when studying deep learning for computer vision.
Miscellaneous
- Two pages that contain nice visualisations of CNNs:
- A cheat sheet Links to an external site. for various aspects of CNNs.
- Explanations and illustrations of padding, strides, transposed convolutions, etc: PDF Links to an external site. and webpage Links to an external site..
- A compilation of online demonstrations: https://p.migdal.pl/interactive-machine-learning-list/ (Links to an external site.)
- If you want to stay tuned with recent developments in deep learning, you can for instance use
- Twitter list with machine learning researchers and profiles (see Members/Medlemmar): twitter.com/DL_ML_Loop/lists/deep-learning-loop (Links to an external site.)
- arXiv alerts
- Quality discussion forum (subreddit) for machine learners: reddit.com/r/MachineLearning (Links to an external site.)
- sign up for a weekly newsletter such as https://www.deeplearningweekly.com/ Links to an external site. or https://us13.campaign-archive.com/home/?u=67bd06787e84d73db24fb0aa5&id=6c9d98ff2c (Links to an external site.) (Links to an external site.)
- Chris Olah has written several illuminating posts on the fundamentals of neural networks, backprop, LSTMs, etc. http://colah.github.io (Links to an external site.)
- Another popular free book on deep learning: neuralnetworksanddeeplearning.com (Links to an external site.)
- Vector, Matrix, and Tensor Derivatives: compsci697l.github.io/docs/vecDerivs.pdf (Links to an external site.)
- Sebastian Ruder's blog contains several well written texts, e.g., a summary of various methods for optimization in deep learning, http://ruder.io/optimizing-gradient-descent/. (Links to an external site.)