Deep Learning — Spring 2019

Class Hours

Tue:  5:30 pm — 7:00 pm,  Thur:  7:15 pm — 8:30 pm
Location LT:5

Office Hours and Contact Info.

Instructor: Mohsen Ali
Office Hours; to be announced
Email: mohsen.ali@itu.edu.pk
Teaching Assistant 1: Salman Ahmed
Office Hours; Wed 5:00 PM to 7:00 PM
Email: mscs17052@itu.edu.pk
Teaching Assistant 2: Izza Rabbani
Office Hours; to be announced
Email: mscs15037@itu.edu.pk

Course Basics

Core Course
Credit Hours: 3
Being offered to both MSDS, MSCS and BS students
Practical and hands on approach
5 to 6 programming assignments

Prerequisite

Enthusiasm, Energy and Imagination
Data Structures, Probability & Statistics, Linear Algebra and basic Calculus
Programming skills and desire to read & implement.

 

Course Overview

We are going to take “get your hands dirty” approach, you will be given assignments and projects to implement ideas discussed in the class. Projects and assignments will contain miniature versions of real life existing applications and problems (e.g can you train your computer to generate dialogues in Shakespeare style or convert your image into painting as done by Monet, sentiment analysis  etc.. ).
Course will concentrate in developing both mathematical knowledge and implementation capabilities. We will start from training a single perceptron, move to training a deep neural network, study why training large networks is a problem and what could be its possible solutions. After dipping our toes in deep belief networks and recurrent neural network we will start looking into applications of deep learning in three different areas, text-analysis, speech processing and computer vision. Objective of this approach is to make you comfortable enough that you can understand various research problems and, if interested, can implement deep learning based applications.

Course Objectives

In last few years machine learning has matured from the science fiction to reality. We are living in a world where we have already seen industry bringing to reality self-driving cars, face-recognizers that work on massive scale (facebook), speech translation systems that can translate from one language to many other simultaneously and in real-time, and more interestingly we have machines that can learn to play atari games in a similar fashion like we do.
A lot of these victories have come from the exciting field of Deep Learning; a learning methodology based on the concept that human mind captures details at multiple levels or at multiple abstract levels. One property of deep learning is removing the responsibility of humans to design features, instead Deep Learning is given a task to find the appropriate representation.

Grading Policy

  • 50% Assignments
  • 15% Final Project
  • 15% Quizzes
  • 10% Midterm Exam
  • 10% Final Exam

Honor Code

All cases of academic misconduct will be forwarded to the disciplinary committee. All assignments are group based unless explicitly specified by the instructor. In the words of Efros,let’s not embarrass ourselves.

Tentative and Rough Course Outline

 

Weeks Topics Evaluations
1 Introduction to Deep Learning

Difference between Machine Learning and Deep Learning

Basic Machine Learning: Linear & Logistic Regression,

2 Supervised Learning with Neural Networks

Deep Learning, Single and Multi-Layer Neural Networks, Perceptron Rule, Gradient Descent, Backpropagation, Loss Functions

Tutorial 1: Python/Numpy Tutorial

Assignment 1
1 Hyperparameters tuning, Regularization and Optimization

Parameters vs Hyperparameters, Why regularization reduces overfitting? Data Augmentation, Vanishing/Exploding gradients, Weight Initialization Methods, Optimizers

Tutorial 2: Building a Linear Classifier

Assignment 2
2 Convolutional Neural Networks

Convolutional Filters, Pooling Layers, Classic CNNs: AlexNet, VGG, GoogleNet, ResNet, DenseNet. Transfer Learning

Tutorial 4: CNN Visualization

Assignment 3
2 Deep Learning for Vision Problems

Object Localization & Detection, Bounding box predictions, Anchor boxes, Region Proposal Networks, Detection Algorithms: RCNN, Faster RCNN, Yolo, SSD.

Tutorial 5: Caffe & Object Detection

2 Sequence Models

Recurrent Neural Networks (RNN), Gate Recurrent Unit (GRU), Long Short Term Memory (LSTM), Bidirectional RNN, Backpropagation through time. Image Caption Generation, Machine Translation, Text Generation & Summarization

Tutorial 6: Image Captioning & Text Generation

Assignment 4
1 Auto-Encoders & Generative Models

Variational Auto-Encoders, Stacked Auto-Encoders, Denoising Auto-Encoders, Concept of Generative Adversarial Networks (GANs)

Assignment 5
3 Miscellaneous

Capsule Networks, Convolutional LSTM, Attention Networks, Restricted Boltzmann Machine, One Shot Learning, Siamese Networks, Triplet Loss, Graph CNN, Approximate and Energy Efficient Design for Deep CNN (Dr. Rehan Hafiz)

Course Content

Date Topic Primary Reading Optional Reading
Feb 7, 2018 Introduction to Deep Learning
Feb 8, 2018 Linear Regression
Error Function in Linear Regression
Logistic Regression

  • Squashing function
  • Sigmoid
  • Cross Entropy Loss Function
Video Lecture by Nando de Freitas Slides

Text Book

  • Text Book: Deep Learning by Ian Goodfellow Link

Recommended Readings

There is not assigned textbook, however following are recommended for reading.

Assignments

  • Will be published soon.

Toolkits

Caffe

●      Toolkit Page

●      Online Caffe Help

●      Caffe Pretrained Models

TensorFlow

●      Toolkit Page

●      Online Discussion and Help Forum

●      Tensoflow Pretrained Models

Torch

●      Toolkit Page

●      Online Help Forum

●      Pretrained Models

Keras

●      Toolkit Page

●      Online Help Forum

●      Pretrained Models

 

Some Interesting Links

Students Projects

List of Projects can be found here