Deep Learning — Spring 2019
Class HoursTue: 5:30 pm — 7:00 pm, Thur: 7:15 pm — 8:30 pm Office Hours and Contact Info.Instructor: Mohsen Ali |
Course BasicsCore Course PrerequisiteEnthusiasm, Energy and Imagination |
Course Overview
We are going to take “get your hands dirty” approach, you will be given assignments and projects to implement ideas discussed in the class. Projects and assignments will contain miniature versions of real life existing applications and problems (e.g can you train your computer to generate dialogues in Shakespeare style or convert your image into painting as done by Monet, sentiment analysis etc.. ).
Course will concentrate in developing both mathematical knowledge and implementation capabilities. We will start from training a single perceptron, move to training a deep neural network, study why training large networks is a problem and what could be its possible solutions. After dipping our toes in deep belief networks and recurrent neural network we will start looking into applications of deep learning in three different areas, text-analysis, speech processing and computer vision. Objective of this approach is to make you comfortable enough that you can understand various research problems and, if interested, can implement deep learning based applications.
Course Objectives
In last few years machine learning has matured from the science fiction to reality. We are living in a world where we have already seen industry bringing to reality self-driving cars, face-recognizers that work on massive scale (facebook), speech translation systems that can translate from one language to many other simultaneously and in real-time, and more interestingly we have machines that can learn to play atari games in a similar fashion like we do.
A lot of these victories have come from the exciting field of Deep Learning; a learning methodology based on the concept that human mind captures details at multiple levels or at multiple abstract levels. One property of deep learning is removing the responsibility of humans to design features, instead Deep Learning is given a task to find the appropriate representation.
Grading Policy
- 50% Assignments
- 15% Final Project
- 15% Quizzes
- 10% Midterm Exam
- 10% Final Exam
Honor Code
All cases of academic misconduct will be forwarded to the disciplinary committee. All assignments are group based unless explicitly specified by the instructor. In the words of Efros,let’s not embarrass ourselves.
Tentative and Rough Course Outline
| Weeks | Topics | Evaluations |
| 1 | Introduction to Deep Learning
Difference between Machine Learning and Deep Learning Basic Machine Learning: Linear & Logistic Regression, |
|
| 2 | Supervised Learning with Neural Networks
Deep Learning, Single and Multi-Layer Neural Networks, Perceptron Rule, Gradient Descent, Backpropagation, Loss Functions Tutorial 1: Python/Numpy Tutorial |
Assignment 1 |
| 1 | Hyperparameters tuning, Regularization and Optimization
Parameters vs Hyperparameters, Why regularization reduces overfitting? Data Augmentation, Vanishing/Exploding gradients, Weight Initialization Methods, Optimizers Tutorial 2: Building a Linear Classifier |
Assignment 2 |
| 2 | Convolutional Neural Networks
Convolutional Filters, Pooling Layers, Classic CNNs: AlexNet, VGG, GoogleNet, ResNet, DenseNet. Transfer Learning Tutorial 4: CNN Visualization |
Assignment 3 |
| 2 | Deep Learning for Vision Problems
Object Localization & Detection, Bounding box predictions, Anchor boxes, Region Proposal Networks, Detection Algorithms: RCNN, Faster RCNN, Yolo, SSD. Tutorial 5: Caffe & Object Detection |
|
| 2 | Sequence Models
Recurrent Neural Networks (RNN), Gate Recurrent Unit (GRU), Long Short Term Memory (LSTM), Bidirectional RNN, Backpropagation through time. Image Caption Generation, Machine Translation, Text Generation & Summarization Tutorial 6: Image Captioning & Text Generation |
Assignment 4 |
| 1 | Auto-Encoders & Generative Models
Variational Auto-Encoders, Stacked Auto-Encoders, Denoising Auto-Encoders, Concept of Generative Adversarial Networks (GANs) |
Assignment 5 |
| 3 | Miscellaneous
Capsule Networks, Convolutional LSTM, Attention Networks, Restricted Boltzmann Machine, One Shot Learning, Siamese Networks, Triplet Loss, Graph CNN, Approximate and Energy Efficient Design for Deep CNN (Dr. Rehan Hafiz) |
Course Content
| Date | Topic | Primary Reading | Optional Reading |
| Feb 7, 2018 | Introduction to Deep Learning | ||
| Feb 8, 2018 | Linear Regression Error Function in Linear Regression Logistic Regression
|
Video Lecture by Nando de Freitas Slides |
Text Book
- Text Book: Deep Learning by Ian Goodfellow Link
Recommended Readings
There is not assigned textbook, however following are recommended for reading.
Assignments
- Will be published soon.
Toolkits
| Caffe | TensorFlow |
| Torch | Keras |
Some Interesting Links
- Linear algebra review / primer by Martial Hebert
- Some of the research groups working with commercial entities
- Machine Learning Group – Geoffrey Hinton
- New York University – Yann Lecun
- Stanford University – Andrew Ng, Fei-fei Li‘s groups
- Microsoft Research
- Google DeepMind – Alex Graves
- Amazon Research
- Vision related links on AAAI.org page
- MIT Nightmare Machine
- Major Advancements in Deep learning in 2016
- Google’s AI Chief Geoffrey Hinton: How Neural Networks really work?
- How Important is Weight Symmetry in Backpropagation?
Students Projects
List of Projects can be found here