Deep Learning
Background and objectives
The objective is that a student that has worked through this material has an understanding of the core concepts of (deep) neural networks and how they are trained. You should also be able to use a contemporary tool for deep learning. We will make use of Keras Links to an external site. which is a python API that can sit ontop of, for example, TensorFlow Links to an external site.. If you want to use another tool, you are free to do so, but we will not be able to offer any support. However, it is likely that one of your colleagues will be able to assist you.
You should also be familiar with the area of Natural Language Processing and have worked with sequential data.
Textbook material
Given the interest in deep learning it is not surprising that there are a number of textbooks (physical or e-versions) on the subject. The bible is Goodfellow, Bengio, and Courville’s Deep Learning Links to an external site. book which is available online. However, it is a brick and quite theoretical. It fits well for those of you that want the details and to be able to go deep. For the assignment in this course, the book Deep learning with Python Links to an external site. by François Chollet fits very well as it takes you from the basics to the type of problems that you will be working on. A list of books is available here Links to an external site..
Introduction to deep learning
If you are completely new to neural networks, take a look the first chapter of Neural Networks and Deep Learning Links to an external site. by Michael Nielsen. Here you will be introduced the neuron model, how they can be combined into networks, how to define a basic cost function and how to learn the parameters using gradient descent.
Chapter 3 in the same book Links to an external site. discusses the cross-entropy cost function, regularisation methods, ways to initialise the weights in the network and ways to choose good hyper parameter. All seems like familiar concepts to you? Skip over this, but if not take a look. The important point here is to get familiar with the concepts and ideas not to know every detail. Deep learning is very much an engineering approach that takes lots of experience to do well, but having a fundamental theoretical understanding helps this process.
An alternative book is Deep learning with Python Links to an external site. by François Chollet which gives a slightly more applied approach. So, if you are more of a learning by doing type, we recommend the first three chapters which are available for download here. You find the corresponding set of Jupyter notebooks here https://github.com/fchollet/deep-learning-with-python-notebooks Links to an external site..
Assuming that you already have set up the environment as described in Computer Tools for Learning and Knowledge you should be able to do
cd ~/tensorflowenv/deep-learning-with-python-notebooks source ~/tensorflowenv/bin/activate
jupyter notebook
To start with, select the notebook 2.1-a-first-look-at-a-neural-network.ipynb and step through it and map what you find there to the general concepts you learned about above. NOTE: The training will take quite some time so let it simmer in the background and do not be afraid to interrupt it when you have got what you want from it. You can see everything that happens and why in Chapter two in Deep learning with Python Links to an external site. (available online).
Natural Language Processing
https://en.wikipedia.org/wiki/Word2vec Links to an external site.
http://cs224d.stanford.edu/syllabus.html Links to an external site.
Slides from talk: DL for NLP, Socher Links to an external site.
Slides from Sochers ML summer school Links to an external site.
https://www.linkedin.com/pulse/best-ai-algorithms-sentiment-analysis-muktabh-mayank/
Deep Learning for Natural Language Processing: Tutorials with Jupyter Notebooks Links to an external site., by John Kron. Signup required, 10 day free trial.
ELMo, said to be state-of-the-art for word embeddings Links to an external site.
Additional material Links to an external site.
- Experiment with this nice demo to get some intuition of what the different layers in a deep network do: ConvNetJS CIFAR-10 Links to an external site.
- Deep Learning for Speech Recognition, Talk by Adam Coates
Links to an external site.
- http://colah.github.io/posts/2015-08-Understanding-LSTMs/ Links to an external site.
- http://colah.github.io/ Links to an external site.
- Feature visualization https://distill.pub/2017/feature-visualization/ Links to an external site.
- Super resolution by ML:
- Keras solution to the Netflix challenge, Links to an external site.
-
Natural Language Processing
Links to an external site.
- Music generation Links to an external site.
- Talk giving some more advanced intuition to how Deep networks learn. Warning: You need some information theory background to read the paper: Opening the black box of Deep Neural Networks via Information
Links to an external site.
- Real-time Style Transfer Links to an external site.
- Some applications of Deep Learning
Links to an external site.
- Some more applications of Deep Learning
Links to an external site.
- More "Two Minute Papers Links to an external site."