šŸ•øļø
Deep Learning
  • šŸ’«Deep Learning Notes
  • šŸ’¼Practical Tools
  • šŸ’ŽConcepts of Neural Networks
    • 🌱Introduction
    • šŸ”ŽThe Problem in General
    • šŸ‘·ā€ā™€ļø Implementation Notes
    • šŸ“šCommon Concepts
    • šŸ’„Activation Functions
    • šŸŽˆPractical Aspects
    • šŸ‘©ā€šŸ”§ NN Regularization
    • ✨Optimization Algorithms
    • šŸŽØSoftmax Regression
    • šŸƒā€ā™€ļø Introduction to Tensorflow
    • šŸ‘©ā€šŸ’» Python Code Snippets
  • šŸ™‹ā€ā™€ļø Hello World of Deep Learning with Neural Networks
    • 🌱Introduction
    • 🌐CNNs In Browser
  • 🚪Introduction to Computer Vision
    • 🌱Introduction
  • 🚩Concepts of Convolutional Neural Networks
    • 🌱Introduction
    • šŸ“ŒCommon Concepts
    • 🌟Advanced Concepts
    • šŸ‘€Visualization
    • šŸ‘µClassic Networks
    • ✨Other Approaches
    • šŸ•øļøCommon Applications
  • šŸ‘©ā€šŸ’» Works and Notes on CNNs
    • 🌱Introduction
  • šŸ’„Popular Strategies of Deep Learning
    • 🌱Introduction
    • šŸš™Transfer Learning
    • šŸ“šOther Strategies
  • 🤔Image Augmentation
    • 🌱Introduction
  • šŸ¤øā€ā™€ļø Notes on Applied Machine Learning
    • 🌱Introduction
    • šŸ‘©ā€šŸ”§ Notes on Structuring Machine Learning Projects
    • šŸ‘©ā€šŸ« Implementation Guidelines
  • šŸ•µļøā€ā™€ļø Basics of Object Detection
    • 🌱Introduction
    • ā­•Region-Based CNNs
    • 🤳SSD and YOLO
    • šŸ¤–TensorFlow Object Detection API
    • šŸžModel Debugging
  • āž°Sequence Models In Deep Learning
    • 🌱Introduction
    • šŸ“šGeneral Concepts
    • šŸ”„Recurrent Neural Networks
    • 🌌Vanishing Gradients with RNNs
    • 🌚Word Representation
    • šŸ’¬Mixed Info On NLP
  • šŸ’¬NLP
    • 🌱Introduction
  • šŸ’¬Applied NLP
    • šŸ™ŒšŸ» Handling texts
    • 🧩Regex
  • šŸ‘€Quick Visual Info
  • šŸ“šPDFs that I found and recommend
Powered by GitBook
On this page
  • ✨ How to effectively set up evaluation metrics?
  • šŸ“š Types of Metrics
  • šŸš€ How to set up datasets to maximize the efficiency
  • šŸ‘©ā€šŸ« Guideline
  • šŸ¤” How to choose the size of sets
  • šŸ™„ When to change dev/test sets and metrics

Was this helpful?

Export as PDF
  1. šŸ¤øā€ā™€ļø Notes on Applied Machine Learning

šŸ‘©ā€šŸ”§ Notes on Structuring Machine Learning Projects

Make your training procedure more effective

✨ How to effectively set up evaluation metrics?

  • While looking to precesion P and recall R (for example) we may be not able to choose the best model correctly

    • So we have to create a new evaluation metric that makes a relation between P and R

    • Now we can choose the best model due to our new metric 🐣

    • For example: (as a popular associated metric) F1 Score is:

      • F1=21P+1RF1 = \frac{2}{\frac{1}{P}+\frac{1}{R}}F1=P1​+R1​2​

To summarize: we can construct our own metrics due to our models and values to be able to get the best choice šŸ‘©ā€šŸ«

šŸ“š Types of Metrics

For better evaluation we have to classify our metrics as the following:

Metric Type

Description

✨ Optimizing Metric

A metric that has to be in its best value

šŸ¤— Satisficing Metric

A metric that just has to be good enough

Technically, If we have N metrics we have to try to optimize 1 metric and to satisfice N-1 metrics šŸ™„

šŸ™Œ Clarification: we tune satisficing metrics due to a threshold that we determine

šŸš€ How to set up datasets to maximize the efficiency

  • It is recommended to choose the dev and test sets from the same distribution, so we have to shuffle the data randomly and then split it.

  • As a result, both test and dev sets have data from all categories ✨

šŸ‘©ā€šŸ« Guideline

We have to choose a dev set and test set - from same distribution - to reflect data we expect to get in te future and consider important to do well on

šŸ¤” How to choose the size of sets

  • If we have a small dataset (m < 10,000)

    • 60% training, 20% dev, 20% test will be good

  • If we have a huge dataset (1M for example)

    • 99% trainig, %1 dev, 1% test will be acceptable

      And so on, considering these two statuses we can choose the correct ratio šŸ‘®ā€

šŸ™„ When to change dev/test sets and metrics

Guideline: if doing well on metric + dev/test set and doesn't correspond to doing well in the real world application, we have to change our metric and/or dev/test set šŸ³

PreviousIntroductionNextšŸ‘©ā€šŸ« Implementation Guidelines

Last updated 4 years ago

Was this helpful?