Components of CNNs (part 4)

We are almost ready to put things together. However, as you know, neurons generally combine a linear (affine) mapping with an activation function. Without these our neural network would just be a linear (affine) function combined with pooling and would be severely limited in what they can do. 

Brandon here makes use of a ReLU activation function and refers to it as normalization (he also downplays the importance of activation functions, but remember that they are essential). Other activation functions are of course also possible. We generally view the nonlinear activation function as part of the convolutional layer (it is not illustrated as a separated function as done here). 

Another side note is that the term "normalization" is actually commonly used in deep learning, but it then refers to some version of the operations that we learned about in lecture 4.