Deep Learning (AING421)
A brief review of automatic differentiation and essential deep learning frameworks. Basic elements of linear regression, loss functions and their optimization. Introduction to linear neural networks. Minibatch-based training based on cross-entropy loss function. Multi-layer perceptrons (MLPs) and the concept of activation functions. Forward and backward propagation. Parameter initialization. Regularization methods, avoiding vanishing and exploding gradients using batch normalization. Dropout methods. Introduction to convolution layers, blocks, multi-channel inputs and cross-correlation operations in convolutional neural network architectures. Input padding, convolution window striding, maximum and average pooling. Implementation of convolutional neural networks and discussions on widely used architectures such as AlexNet, GoogLeNet and ResNet. Modelling sequential data using recurrent neural networks (RNN). Modern recurrent neural networks employing gated recurrent units and long short-term memory (LSTM), and their applications. Unsupervised learning using autoencoder architectures. Discussions on various types such as simple and sparse autoencoders and their implementations.