πŸ•ΈοΈ
Deep Learning
  • πŸ’«Deep Learning Notes
  • πŸ’ΌPractical Tools
  • πŸ’ŽConcepts of Neural Networks
    • 🌱Introduction
    • πŸ”ŽThe Problem in General
    • πŸ‘·β€β™€οΈ Implementation Notes
    • πŸ“šCommon Concepts
    • πŸ’₯Activation Functions
    • 🎈Practical Aspects
    • πŸ‘©β€πŸ”§ NN Regularization
    • ✨Optimization Algorithms
    • 🎨Softmax Regression
    • πŸƒβ€β™€οΈ Introduction to Tensorflow
    • πŸ‘©β€πŸ’» Python Code Snippets
  • πŸ™‹β€β™€οΈ Hello World of Deep Learning with Neural Networks
    • 🌱Introduction
    • 🌐CNNs In Browser
  • πŸšͺIntroduction to Computer Vision
    • 🌱Introduction
  • 🚩Concepts of Convolutional Neural Networks
    • 🌱Introduction
    • πŸ“ŒCommon Concepts
    • 🌟Advanced Concepts
    • πŸ‘€Visualization
    • πŸ‘΅Classic Networks
    • ✨Other Approaches
    • πŸ•ΈοΈCommon Applications
  • πŸ‘©β€πŸ’» Works and Notes on CNNs
    • 🌱Introduction
  • πŸ’„Popular Strategies of Deep Learning
    • 🌱Introduction
    • πŸš™Transfer Learning
    • πŸ“šOther Strategies
  • 🀑Image Augmentation
    • 🌱Introduction
  • πŸ€Έβ€β™€οΈ Notes on Applied Machine Learning
    • 🌱Introduction
    • πŸ‘©β€πŸ”§ Notes on Structuring Machine Learning Projects
    • πŸ‘©β€πŸ« Implementation Guidelines
  • πŸ•΅οΈβ€β™€οΈ Basics of Object Detection
    • 🌱Introduction
    • β­•Region-Based CNNs
    • 🀳SSD and YOLO
    • πŸ€–TensorFlow Object Detection API
    • 🐞Model Debugging
  • ➰Sequence Models In Deep Learning
    • 🌱Introduction
    • πŸ“šGeneral Concepts
    • πŸ”„Recurrent Neural Networks
    • 🌌Vanishing Gradients with RNNs
    • 🌚Word Representation
    • πŸ’¬Mixed Info On NLP
  • πŸ’¬NLP
    • 🌱Introduction
  • πŸ’¬Applied NLP
    • πŸ™ŒπŸ» Handling texts
    • 🧩Regex
  • πŸ‘€Quick Visual Info
  • πŸ“šPDFs that I found and recommend
Powered by GitBook
On this page
  • ❓ What is Transfer Learning?
  • πŸ’« Traditional ML vs Transfer Learning
  • πŸ™„ Problem
  • πŸ”ƒ Comparison
  • πŸ€” When is it practical?
  • 🧐 References

Was this helpful?

Export as PDF
  1. Popular Strategies of Deep Learning

Transfer Learning

Applying a knowledge to separate tasks

PreviousIntroductionNextOther Strategies

Last updated 4 years ago

Was this helpful?

In short: Learning from one task and applying knowledge to separate tasks πŸ›°πŸš™

❓ What is Transfer Learning?

  • πŸ•΅οΈβ€β™€οΈ Transfer learning is a machine learning technique where a model trained on one task is re-purposed on a second related task.

  • 🌟 In addition, it is an optimization method that allows rapid progress or improved performance when modeling the second task.

  • πŸ€Έβ€β™€οΈ Transfer learning only works in deep learning if the model features learned from the first task are general.

Long story short: Rather than training a neural network from scratch we can instead download an open-source model that someone else has already trained on a huge dataset maybe for weeks and use these parameters as a starting point to train our model just a little bit more with the smaller dataset that we have ✨

πŸ’« Traditional ML vs Transfer Learning

πŸ™„ Problem

Layers in a neural network can sometimes end up having similar weights and possible impact each other leading to over-fitting. With a big complex model it's a risk. So if you can imagine the dense layers can look a little bit like this.

We can drop out some neurons that has similar weights with neighbors, so that overfitting is being removed.

πŸ”ƒ Comparison

πŸ€Έβ€β™€οΈ An NN before and after dropout

✨ Accuracy before and after dropout

πŸ€” When is it practical?

It is practical when we have a lot of data for problem that we are transferring from and usually relatively less data for the problem we are transferring to πŸ•΅οΈβ€

More accurately:

For task A to task B, it is sensible to do transfer learning from A to B when:

  • 🚩 Task A and task B have the same output x

  • ⭐ We have a lot more data for task A than task B

  • πŸ”Ž Low level features from task A could be helpful for learning task B

🧐 References

πŸ’„
πŸš™
More about transfer learning in Tensorflow
Understanding Dropout