Deep Learning — Spring 2019
Tue: 5:30 pm — 7:00 pm, Thur: 7:15 pm — 8:30 pm
Office Hours and Contact Info.
Instructor: Mohsen Ali
Enthusiasm, Energy and Imagination
We are going to take “get your hands dirty” approach, you will be given assignments and projects to implement ideas discussed in the class. Projects and assignments will contain miniature versions of real life existing applications and problems (e.g can you train your computer to generate dialogues in Shakespeare style or convert your image into painting as done by Monet, sentiment analysis etc.. ).
Course will concentrate in developing both mathematical knowledge and implementation capabilities. We will start from training a single perceptron, move to training a deep neural network, study why training large networks is a problem and what could be its possible solutions. After dipping our toes in deep belief networks and recurrent neural network we will start looking into applications of deep learning in three different areas, text-analysis, speech processing and computer vision. Objective of this approach is to make you comfortable enough that you can understand various research problems and, if interested, can implement deep learning based applications.
In last few years machine learning has matured from the science fiction to reality. We are living in a world where we have already seen industry bringing to reality self-driving cars, face-recognizers that work on massive scale (facebook), speech translation systems that can translate from one language to many other simultaneously and in real-time, and more interestingly we have machines that can learn to play atari games in a similar fashion like we do.
A lot of these victories have come from the exciting field of Deep Learning; a learning methodology based on the concept that human mind captures details at multiple levels or at multiple abstract levels. One property of deep learning is removing the responsibility of humans to design features, instead Deep Learning is given a task to find the appropriate representation.
- 50% Assignments
- 15% Final Project
- 15% Quizzes
- 10% Midterm Exam
- 10% Final Exam
All cases of academic misconduct will be forwarded to the disciplinary committee. All assignments are group based unless explicitly specified by the instructor. In the words of Efros,let’s not embarrass ourselves.
Tentative and Rough Course Outline
|1||Introduction to Deep Learning
Difference between Machine Learning and Deep Learning
Basic Machine Learning: Linear & Logistic Regression,
|2||Supervised Learning with Neural Networks
Deep Learning, Single and Multi-Layer Neural Networks, Perceptron Rule, Gradient Descent, Backpropagation, Loss Functions
Tutorial 1: Python/Numpy Tutorial
|1||Hyperparameters tuning, Regularization and Optimization
Parameters vs Hyperparameters, Why regularization reduces overfitting? Data Augmentation, Vanishing/Exploding gradients, Weight Initialization Methods, Optimizers
Tutorial 2: Building a Linear Classifier
|2||Convolutional Neural Networks
Convolutional Filters, Pooling Layers, Classic CNNs: AlexNet, VGG, GoogleNet, ResNet, DenseNet. Transfer Learning
Tutorial 4: CNN Visualization
|2||Deep Learning for Vision Problems
Object Localization & Detection, Bounding box predictions, Anchor boxes, Region Proposal Networks, Detection Algorithms: RCNN, Faster RCNN, Yolo, SSD.
Tutorial 5: Caffe & Object Detection
Recurrent Neural Networks (RNN), Gate Recurrent Unit (GRU), Long Short Term Memory (LSTM), Bidirectional RNN, Backpropagation through time. Image Caption Generation, Machine Translation, Text Generation & Summarization
Tutorial 6: Image Captioning & Text Generation
|1||Auto-Encoders & Generative Models
Variational Auto-Encoders, Stacked Auto-Encoders, Denoising Auto-Encoders, Concept of Generative Adversarial Networks (GANs)
Capsule Networks, Convolutional LSTM, Attention Networks, Restricted Boltzmann Machine, One Shot Learning, Siamese Networks, Triplet Loss, Graph CNN, Approximate and Energy Efficient Design for Deep CNN (Dr. Rehan Hafiz)
|Date||Topic||Primary Reading||Optional Reading|
|Feb 7, 2018||Introduction to Deep Learning|
|Feb 8, 2018||Linear Regression
Error Function in Linear Regression
|Video Lecture by Nando de Freitas Slides|
- Text Book: Deep Learning by Ian Goodfellow Link
There is not assigned textbook, however following are recommended for reading.
- Book: http://neuralnetworksanddeeplearning.com/
- Will be published soon.
Some Interesting Links
- Linear algebra review / primer by Martial Hebert
- Some of the research groups working with commercial entities
- Vision related links on AAAI.org page
- MIT Nightmare Machine
- Major Advancements in Deep learning in 2016
- Google’s AI Chief Geoffrey Hinton: How Neural Networks really work?
- How Important is Weight Symmetry in Backpropagation?
List of Projects can be found here