Deep Learning – Spring 2021
Class HoursTue: 5:30 pm — 7:00 pm, Thur: 7:15 pm — 8:45 pm Office Hours and Contact Info.Instructor: Mohsen Ali Teaching Assistant 1: Obaid Ullah Ahmad Teaching Assistant 2: Hafiz Muhammad Abdullah Zia Teaching Assistant 3: Muhammad Hamad Akram |
Course BasicsCore Course PrerequisiteEnthusiasm, Energy, and Imagination |
Course Overview
We are going to take the “get your hands dirty” approach, you will be given assignments and projects to implement ideas discussed in the class. Projects and assignments will contain miniature versions of real-life existing applications and problems (e.g can you train your computer to generate dialogues in Shakespeare style or convert your image into painting as done by Monet, sentiment analysis, etc.. ).
The course will concentrate on developing both mathematical knowledge and implementation capabilities. We will start from training a single perceptron, move to training a deep neural network, study why training large networks is a problem and what could be its possible solutions. After dipping our toes in deep belief networks and recurrent neural network we will start looking into applications of deep learning in three different areas, text-analysis, speech processing, and computer vision. The objective of this approach is to make you comfortable enough that you can understand various research problems and, if interested, can implement deep learning-based applications.
Course Objectives
In the last few years, machine learning has matured from science fiction to reality. We are living in a world where we have already seen industry bringing to reality self-driving cars, face-recognizers that work on a massive scale (Facebook), speech translation systems that can translate from one language to many other simultaneously and in real-time, and more interestingly we have machines that can learn to play atari games in a similar fashion as we do.
A lot of these victories have come from the exciting field of Deep Learning; a learning methodology based on the concept that the human mind captures details at multiple levels or at multiple abstract levels. One property of deep learning is removing the responsibility of humans to design features, instead, Deep Learning is given a task to find the appropriate representation.
Grading Policy
- 45% Assignments
- 5% Class participation and Creating Notes
- 20% Final Project
- 10% Quizzes
- 10% Midterm Exam
- 10% Final Exam
Honor Code
All cases of academic misconduct will be forwarded to the disciplinary committee. All assignments are group-based unless explicitly specified by the instructor. In the words of Efros, let’s not embarrass ourselves.
Tentative and Rough Course Outline
Weeks | Topics | Evaluations |
1 | Introduction to Deep Learning
Difference between Machine Learning and Deep Learning Basic Machine Learning: Linear & Logistic Regression, |
|
2 | Supervised Learning with Neural Networks
Deep Learning, Single and Multi-Layer Neural Networks, Perceptron Rule, Gradient Descent, Backpropagation, Loss Functions Tutorial 1: Python/Numpy Tutorial/Anaconda |
|
3 | Hyperparameters tuning, Regularization and Optimization
Parameters vs Hyperparameters, Why regularization reduces overfitting? Data Augmentation, Vanishing/Exploding gradients, Weight Initialization Methods, Optimizers Tutorial 2: Building a Linear Classifier |
|
4 | Convolutional Neural Networks
Convolutional Filters, Pooling Layers, Classic CNNs: AlexNet, VGG, GoogleNet, ResNet, DenseNet. Transfer Learning Tutorial 4: CNN Visualization |
|
5 | Deep Learning for Vision Problems
Object Localization & Detection, Bounding box predictions, Anchor boxes, Region Proposal Networks, Detection Algorithms: RCNN, Faster RCNN, Yolo, SSD. Tutorial 5: Caffe & Object Detection |
|
6 | Sequence Models
Recurrent Neural Networks (RNN), Gate Recurrent Unit (GRU), Long Short Term Memory (LSTM), Bidirectional RNN, Backpropagation through time. Image Caption Generation, Machine Translation, Text Generation & Summarization and Transformers Tutorial 6: Image Captioning & Text Generation |
|
7 | Auto-Encoders & Generative Models
Variational Auto-Encoders, Stacked Auto-Encoders, Denoising Auto-Encoders, Concept of Generative Adversarial Networks (GANs) |
|
8 | Miscellaneous
Deep Networks Generalization, Adversarial Examples and Attacks, Reinforcement Learning, Capsule Networks, Convolutional LSTM, Attention Networks, Restricted Boltzmann Machine, One-Shot Learning, Siamese Networks, Triplet Loss, Graph CNN, Approximate and Energy-Efficient Design for Deep CNN (Dr. Rehan Hafiz) |
Course Notes
Topics | Notes / Reading Material / Comments | News | |
09th Mar 2021 | Introduction | 45% Assignments
20% Final Project 5% Class participation and Creating Notes 10% Quizzes 10% Midterm Exam 10% Final Exam |
|
Recommended Resources |
|
||
11th Mar 2021 | Un-Supervised and Supervised Learning |
Assigned Readings:
Recommended Readings:
Refresh: Concepts of local minima, local maxima, convex functions, concave functions, critical points, chain rule, saddle point |
|
16th Mar 2021 | No Class | Makeup class will be announced soon | |
18th Mar 2021 | No Class | Makeup class(es) will take place on Saturday 27th March, 2021 | |
23rd Mar 2021 | No Class on account of Pakistan Resolution Day | Makeup class will be announced soon | |
25th Mar 2021 | Linear Regression |
|
Assignment 1 |
27th Mar 2021 | Optimization, Gradient Descent and Logistic Regression |
|
|
30th Mar 2021 | Logistic Regression, Classification, Loss Functions |
Take home task
Assigned Readings:
Recommended Reading: Softmax and Cross Entropy Assigned Readings (Linear Algebra): |
|
1st Apr 2021 | Multi Class Classification, Optimization and Gradient Descent |
Recommended Readings: Recommended Readings: Gradient Descent: Video Lecture from Coursera, Andrew NG |
|
2nd Apr 2021 | Neural Networks |
Assigned Readings:
Upto complete section 6.3. Recommended Readings: Perceptron Rule |
|
3rd Apr 2021 | Backpropagation |
Assigned Readings:
Recommended Readings
Optional: How to do backpropagation in a brain by Hinton Video Lecture: Lecture 4: Backpropagation; Dhruv Batra |
|
6th Apr 2021 | Neural Network Training |
Assigned Readings:
|
Assignment 2 |
8th Apr 2021 | Weight Initialization and Batch Normalization |
Assigned Readings Recommended Reading: |
|
13th Apr 2021 | Regularization and Dropout |
Home Work:
Assigned Readings: Video Lectures
Recommended Readings: |
|
15th Apr 2021 | Texture and Convolution filters |
|
|
20th Apr 2021 | Filters and Convolutional Neural Networks |
NOTES Assigned Readings Recommended Readings |
|
22th Apr 2021 | CNNs |
Assigned Readings |
|
27th Apr 2021 | Back Propagation in CNNs |
Reference
Video Lecture: |
Assignment 3 Deliverable 2 |
29th Apr 2021 | Transfer Learning |
|
Assignment 3 Deliverable 1 |
4th May 2021 | Transfer Learning and CNN Architectures |
Assigned Reading
Recommended Readings
|
|
6th May 2021 | Semantic Segmentation |
Assigned Reading Recommended Readings |
|
18th May 2021 | Localization and Object Detection |
Recommended Readings |
Assignment 4 |
20th May 2021 | Object Detection, Recurrent Neural Networks |
Assigned Reading Sequence Modeling: Recurrent and Recursive Nets, Chapter 10 from textbook. Recommended Readings |
|
1st June 2021 | Language Models |
Recommended Readings
|
|
3rd June 2021 | Unsupervised Learning |
Readings |
|
8th June 2021 | Autoencoders |
|
|
10th June 2021 | Generative Adversarial Networks |
|
|
15th June 2021 | Do deep networks Generalize? |
Assigned Readings: Assigned Video Content: Recommended Readings: Wasserstein GAN Blog Post (From GAN to WGAN) |
|
17th June 2021 | Adversarial Attacks |
|
Assignment 5 |
24th June 2021 | Reinforcement Learning |
|
Text Book
- Text Book: Deep Learning by Ian Goodfellow Link
- Reference Book: Dive into Deep Learning by Aston Zhang and co Link
Recommended Readings
Following are recommended for reading.
Toolkits
PyTorch |
Top Conferences to Follow
- International Conference on Machine Learning (ICML)
- Conference on Neural Information Processing Systems (NIPS)
- International Joint Conference on Artificial Intelligence (ICAI)
- Conference on Computer Vision and Pattern Recognition (CVPR)
- International Conference on Computer Vision (ICCV)
- British Machine Vision Conference (BMVC)