πŸ•ΈοΈ
Deep Learning
  • πŸ’«Deep Learning Notes
  • πŸ’ΌPractical Tools
  • πŸ’ŽConcepts of Neural Networks
    • 🌱Introduction
    • πŸ”ŽThe Problem in General
    • πŸ‘·β€β™€οΈ Implementation Notes
    • πŸ“šCommon Concepts
    • πŸ’₯Activation Functions
    • 🎈Practical Aspects
    • πŸ‘©β€πŸ”§ NN Regularization
    • ✨Optimization Algorithms
    • 🎨Softmax Regression
    • πŸƒβ€β™€οΈ Introduction to Tensorflow
    • πŸ‘©β€πŸ’» Python Code Snippets
  • πŸ™‹β€β™€οΈ Hello World of Deep Learning with Neural Networks
    • 🌱Introduction
    • 🌐CNNs In Browser
  • πŸšͺIntroduction to Computer Vision
    • 🌱Introduction
  • 🚩Concepts of Convolutional Neural Networks
    • 🌱Introduction
    • πŸ“ŒCommon Concepts
    • 🌟Advanced Concepts
    • πŸ‘€Visualization
    • πŸ‘΅Classic Networks
    • ✨Other Approaches
    • πŸ•ΈοΈCommon Applications
  • πŸ‘©β€πŸ’» Works and Notes on CNNs
    • 🌱Introduction
  • πŸ’„Popular Strategies of Deep Learning
    • 🌱Introduction
    • πŸš™Transfer Learning
    • πŸ“šOther Strategies
  • 🀑Image Augmentation
    • 🌱Introduction
  • πŸ€Έβ€β™€οΈ Notes on Applied Machine Learning
    • 🌱Introduction
    • πŸ‘©β€πŸ”§ Notes on Structuring Machine Learning Projects
    • πŸ‘©β€πŸ« Implementation Guidelines
  • πŸ•΅οΈβ€β™€οΈ Basics of Object Detection
    • 🌱Introduction
    • β­•Region-Based CNNs
    • 🀳SSD and YOLO
    • πŸ€–TensorFlow Object Detection API
    • 🐞Model Debugging
  • ➰Sequence Models In Deep Learning
    • 🌱Introduction
    • πŸ“šGeneral Concepts
    • πŸ”„Recurrent Neural Networks
    • 🌌Vanishing Gradients with RNNs
    • 🌚Word Representation
    • πŸ’¬Mixed Info On NLP
  • πŸ’¬NLP
    • 🌱Introduction
  • πŸ’¬Applied NLP
    • πŸ™ŒπŸ» Handling texts
    • 🧩Regex
  • πŸ‘€Quick Visual Info
  • πŸ“šPDFs that I found and recommend
Powered by GitBook
On this page
  • πŸ”„ Residual Networks
  • πŸ™„ Problem
  • πŸ™Œ Plain Nets vs ResNets
  • πŸ‘€ Visualization
  • πŸ€— Advantages
  • 1️⃣ One By One Convolutions
  • Propblem (Or motivation πŸ€”)
  • πŸ€Έβ€β™€οΈ Solution
  • 🧐 Read More

Was this helpful?

Export as PDF
  1. Concepts of Convolutional Neural Networks

Other Approaches

PreviousClassic NetworksNextCommon Applications

Last updated 4 years ago

Was this helpful?

Approach

Description

Residual Networks

An approach to avoid vanishing gradient issue in deep NNs

One By One Convolution

Applying filters on color channels

πŸ”„ Residual Networks

πŸ™„ Problem

During each iteration of training a neural network, all weights receive an update proportional to the partial derivative of the error function with respect to the current weight. If the gradient is very small then the weights will not be change effectively and it may completely stop the neural network from further training πŸ™„πŸ˜ͺ. The phenomenon is called vanishing gradients πŸ™

Simply πŸ˜…: we can say that the data is disappearing through the layers of the deep neural network due to very slow gradient descent

The core idea of ResNet is introducing a so-called identity shortcut connection that skips one or more layers, like the following

πŸ™Œ Plain Nets vs ResNets

πŸ‘€ Visualization

πŸ€— Advantages

  • Easy for one of the blocks to learn an identity function

  • Can go deeper without hurting the performance

    • In the Plain NNs, because of the vanishing and exploding gradients problems the performance of the network suffers as it goes deeper.

1️⃣ One By One Convolutions

Propblem (Or motivation πŸ€”)

We can reduce the size of inputs by applying pooling and various convolution, these filteres can reduce the height and the width of the input image, what about color channels 🌈, in other words; what about the depth?

πŸ€Έβ€β™€οΈ Solution

We know that the depth of the output of a CNN is equal to the number of filters that we applied on the input;

In the example above, we applied 2 filters, so the output depth is 2

How can we use this info to improve our CNNs? πŸ™„

🧐 Read More

Let's say that we have a 28x28x192 dimensional input, if we apply 32 filters at 1x1x192 dimension and padding our output will become 28x28x32 ✨

🚩
✨
SAME
Detailed ResNets