👷‍♀️ Network Visualization Tool

• Visualize the graph of the network

• Netron ✨✨

💫 CNN Input / Output Visualization Tool

• Watch the inputs and outputs of each layer in your CNN

• Tensorspace 🎉

🖼️ OpenImages Downloading Tool

• 🚀 Download images by class

• 🔗 OID

🔗 Bulk Link Downloading Tool

• 💁‍♀️ Download bulk links by one click

• 👩‍💻 Google Chrome extension

• ⚓ Tab Save

👶The Growth of NLP

• Rule based systems

• Probabilistic systems

• End to end systems

🔎 Definition

A neural network is a type of machine learning which models itself after the human brain. This creates an artificial neural network that via an algorithm allows the computer to learn by incorporating new data.

Neural networks are able to perform what has been termed deep learning. While the basic unit of the brain is the neuron, the essential building block of an artificial neural network is a perceptron which accomplishes simple signal processing, and these are then connected into a large mesh network.

📑 Types of NNs

There are many types of neural networks, choosing a type is due to the problem that we are trying to solve, for example

🎨 Types of Data in Supervised Learning

• 🚧 Structured Data

  • Such as tables

  • We have input fields and an output field

• 🤹‍♂️ Unstructured Data

  • Such as images, audio and texts

  • We need to use feature extraction algorithms to build our model

🧐 References

• Introduction to Artificial Neural Networks (ANN)

📚 Popular Terms

👩‍💻 My Codes

• 🚙 Transfer Learning, Dog vs Cat 🐶🐱

👷‍♀️ Network Visualization Tool

Netron ✨✨

🎤 About

• 🕸 My notes about Artificial Neural Networks, Convolutional Neural Networks and Recurrent Neural Networks with theoretical details

• 🦋 I will share new details as I learn new concepts in this context

📑 Table of Contents

💉 Extensions

🚀 Other Version

• Turkish version of this project is

🙌 Quote

"Your learning algorithm has two main sources of knowledge; one is the data and other is whatever you hand design" 🤔🚀

⭐ Please..

• ✨ Help me to improve and to increase the content by opening a pull request

• 👓 Tell me your suggestions by sending me an or opening an issue

👜 Contact & Support

Find me on and feel free to mail me,

This folder contains theoric details about CNNs

📚 Important Terms

💫 Notes on performance

• Training speed of a CNN is too slower than plain NN because of its computational complexity 🐢

🧐 References

🙄 Problems that we can face while training custom object detection

1. Model is not doing well on test set

2. Model is doing well on test set but doing bad on real world images

In case that model is not doing well on test set you can try one or more from the followings:

• Add dropout to .config file

• Replace fixed_shape_resizer with keep_aspect_ratio_resizer, example:

👩‍🎓 Orthogonalisation

One of the challenges with building machine learning systems is that there are so many things we could try. Including, for example, so many hyperparameters we could tune. The art of knowing what parameter to tune to get what effect, is called orthogonalisation.

What should we pay attention to while evaluating an ML project? How to optimize it? How to speed up? Since there are a lot of parameters how to know where to fix and which parameter to tune? 🤔🤕

Before answering these questions let's take a look at the whole process 🧐

⛓ Chain of assumptions in ML

The model should:

Fit training set well on cost function (Human level performance ❌❌)

⬇

Fit dev set well on cost function

⬇

Fit test set well on cost function

⬇

Perform well in real world ✨

Figuring out what is exactly wrong can help us to choose a suitable solution and then to fix that part without affecting the whole project 👩‍🔧

🤸‍♀️ Applications

🔠 Neural Machine Translation

• A machine translation model is similar to a language model except it has an encoder network placed before.

• It is sometimes referred as a conditional language model.

🕵️‍♀️ Neural Machine Translation with Attention

• If you had to translate a book's paragraph from French to English, you would not read the whole paragraph, then close the book and translate 😅

• Even during the translation process, you would read/re-read and focus on the parts of the French paragraph corresponding to the parts of the English you are writing down 🤔

• The attention mechanism tells a Neural Machine Translation model where it should pay attention to at any step 👩‍🏫

🔊 Speech Recognition

• Converting an audio (x-input) to text (y-output)

  • By measuring air pressure 🙄

• Sequence-to-Sequence model

TODO: Add details

We can learn it by likening it to logistic regression: 😋

Recall that logistic regression produces a decimal between 0 and 1.0. For example, a logistic regression output of 0.8 from an email classifier suggests an 80% chance of an email being spam and a 20% chance of it being not spam. Clearly, the sum of the probabilities of an email being either spam or not spam is 1.0.

Softmax extends this idea into the MULTI-CLASS world. That is, Softmax assigns decimal probabilities to each class in a multi-class problem. Those decimal probabilities must add up to 1.0.

• Its other name is Maximum Entropy (MaxEnt) Classifier

We can say that softmax regression generalizes logistic regression

Logistic regression is a special status of softmax where C = 2 🤔

📚 Notation

C = number of classes = number of units of the output layer So, $\hat{y}_j$ is a (C, 1) dimensional vector.

🎨 Softmax Layer

Softmax is implemented through a neural network layer just before the output layer. The Softmax layer must have the same number of nodes as the output layer.

💥 Softmax Activation Function

$Softmax(x_i)\frac{exp(x_i)}{\sum_{j}exp(x_j)}$

🔨 Hard Max function

Takes the output of softmax layer and convert it into 1 vs 0 vector (as I called it 🤭) which will be our ŷ

For example:

t = 0.13  ==> ̂y = 0
    0.75          1
    0.01          0
    0.11          0

And so on 🐾

🔎 Loss Function

$L(\hat{y},y)=-\sum_{j=1}^{c}y_jlog(\hat{y}_j)$

Y and ŷ are (C,m) dimensional matrices 👩‍🔧

🧐 Read More

• Long story short from Google documentation

Briefly: A technique to prevent overfitting -and reduce variance-

🙄 Problem

In over-fitting situation, our model tries to learn too well the details and the noise from the training data, which ultimately results in poor performance on the unseen data (test set).

The following graph describes better:

👩‍🏫 Better Definition for Regularization

It is a technique which makes slight modifications to the learning algorithm such that the model generalizes better. This in turn improves the model’s performance on the unseen data as well.

🔨 Regularization Techniques

🔩 L2 Regularization (Weight decay)

The most common type of regularization, given by following formula:

$J=Loss+\frac{\lambda}{2m}-\sum ||w||^{2}$

Here, lambda is the regularization parameter. It is the hyperparameter whose value is optimized for better results. L2 regularization is also known as weight decay as it forces the weights to decay towards zero (but not exactly zero)

🔩 Dropout

Another regularization method by eliminating some neurons in a specific ratio randomly

Simply: For each node of probability p, don’t update its input or output weights during backpropagation (Just drop it 😅)

Better visualiztion:

An NN before and after dropout

It is commonly used in computer vision, but its downside is that Cost function J is no longer well defined

🤡 Data Augmentation

The simplest way to reduce overfitting is to increase the size of the training data, it is not always possible since getting more data is too costly, but sometimes we can increase our data based on our data, for example:

• Doing transformations on images can maximize our data set

🛑 Early Stopping

It is a kind of cross-validation strategy where we keep one part of the training set as the validation set. When we see that the performance on the validation set is getting worse, we immediately stop the training on the model. This is known as early stopping.

🧐 Read More

• Long Story Short 😅: Overfitting and Regularization in Neural Networks

Visualization of concepts explained in P1 and P2 to wrap them up 👩‍🎓

💫 Convolution

Applying a filter to extract features 🤗

Problem 😰: Images are shrinking 😱

😏 Take A Look At Padding

Images Are Too Large, Performance is Down 😔

😉 Let's See Pooling

🙄 Well, I have an RGB image

Filters must have depth that is equal to number of color channels

🤡 Ok, now I want to apply n filters

Depth of the output will be equal to n

🤗 Check Your Understanding With A Full Example

🧐 References

• DeepLearning series: Convolutional Neural Networks (😍✨✨✨)

➰ Multi-Task Learning

In short: We start simultaneously trying to have one NN do several things at same time and then each of these tasks helps all of the other tasks 🚀

In other words: Let's say that we want to build a detector to detect 4 classes of objects, instead of building 4 NN for each class, we can build one NN to detect the four classes 🤔 (The output layer has 4 units)

🤔 When Is It Practical?

• 🤳 Training on a set of tasks that could benefit from having shared lower level features

• ⛱ Amount of data we have for each task is quite similar (sometimes) ⛱

• 🤗 Can train a big enough NN to do well on all the tasks (instead of building a separate network fır each task)

👓 Multi task learning is used much less than transfer learning

👀 Visualization

🏴 End to End Deep Learning

• Briefly, there have been some data processing systems or learning systems that requires multiple stages of processing,

• End to end learning can take all these multiple stages and replace it with just a single NN

👩‍🔧 Long Story Short: breaking the big task into sub smaller tasks with the same NN

➕ Pros:

• 🦸‍♀️ Shows the power of the data

• ✨ Less hand designing of components needed

➖ Cons:

• 💔 May need large amount of data

• 🔎 Excludes potentially useful hand designed components

🚩 Guideline to Make Decision to Use It

Key question: do you have sufficient data to learn a function of the complexity needed to map x to y?

🔃 End to End Learning vs Transfer Learning

Materials will be divided into different files (or categories) as they increase 👮‍

📚 Types of Machine Learning

👓 Supervised Learning vs Unsupervised Learning

🕶 Machine Learning vs Deep Learning

🧠 Machine Learning Mind Map

Good Sources That Must Be Followed

• Instagram AI Machine Learning

📚 Important Terms

🎀 Convolution Example

🤔 How did we find -7?

We did element wise product then we get the sum of the result matrix; so:

And so on for other elements 🙃

👼 Visualization of Calculation

🔎 Edge Detection

An application of convolution operation

🔎 Edge Detection Examples

Result: horizontal lines pop out

Result: vertical lines pop out

🙄 What About The Other Numbers

There are a lot of ways we can put number inside elements of the filter.

For example Sobel filter is like:

Scharr filter is like:

Prewitt filter is like:

So the point here is to pay attention to the middle row

And Roberts filter is like:

✨ Another Approach

We can tune these numbers by ML approach; we can say that the filter is a group of weights that:

By that we can get -learned- horizontal, vertical, angled, or any edge type automatically rather than getting them by hand.

🤸‍♀️ Computational Details

If we have an n*n image and we convolve it by f*f filter the the output image will be n-f+1*n-f+1

😐 Downsides

1. 🌀 If we apply many filters then our image shrinks.

2. 🤨 Pixels at corners aren't being touched enough, so we are throwing away a lot of information from the edges of the image .

💡 Solution

We can the image 💪

🧐 References

🔄 Residual Networks

🙄 Problem

During each iteration of training a neural network, all weights receive an update proportional to the partial derivative of the error function with respect to the current weight. If the gradient is very small then the weights will not be change effectively and it may completely stop the neural network from further training 🙄😪. The phenomenon is called vanishing gradients 🙁

Simply 😅: we can say that the data is disappearing through the layers of the deep neural network due to very slow gradient descent

The core idea of ResNet is introducing a so-called identity shortcut connection that skips one or more layers, like the following

🙌 Plain Nets vs ResNets

👀 Visualization

🤗 Advantages

• Easy for one of the blocks to learn an identity function

• Can go deeper without hurting the performance

  • In the Plain NNs, because of the vanishing and exploding gradients problems the performance of the network suffers as it goes deeper.

1️⃣ One By One Convolutions

Propblem (Or motivation 🤔)

We can reduce the size of inputs by applying pooling and various convolution, these filteres can reduce the height and the width of the input image, what about color channels 🌈, in other words; what about the depth?

🤸‍♀️ Solution

We know that the depth of the output of a CNN is equal to the number of filters that we applied on the input;

In the example above, we applied 2 filters, so the output depth is 2

How can we use this info to improve our CNNs? 🙄

Let's say that we have a 28x28x192 dimensional input, if we apply 32 filters at 1x1x192 dimension and padding our output will become 28x28x32 ✨

🧐 Read More

✨ How to effectively set up evaluation metrics?

• While looking to precesion P and recall R (for example) we may be not able to choose the best model correctly

  • So we have to create a new evaluation metric that makes a relation between P and R

  • Now we can choose the best model due to our new metric 🐣

  • For example: (as a popular associated metric) F1 Score is:

To summarize: we can construct our own metrics due to our models and values to be able to get the best choice 👩‍🏫

📚 Types of Metrics

For better evaluation we have to classify our metrics as the following:

Technically, If we have N metrics we have to try to optimize 1 metric and to satisfice N-1 metrics 🙄

🙌 Clarification: we tune satisficing metrics due to a threshold that we determine

🚀 How to set up datasets to maximize the efficiency

• It is recommended to choose the dev and test sets from the same distribution, so we have to shuffle the data randomly and then split it.

• As a result, both test and dev sets have data from all categories ✨

👩‍🏫 Guideline

We have to choose a dev set and test set - from same distribution - to reflect data we expect to get in te future and consider important to do well on

🤔 How to choose the size of sets

• If we have a small dataset (m < 10,000)

  • 60% training, 20% dev, 20% test will be good

• If we have a huge dataset (1M for example)

  • 99% trainig, %1 dev, 1% test will be acceptable

    And so on, considering these two statuses we can choose the correct ratio 👮‍

🙄 When to change dev/test sets and metrics

Guideline: if doing well on metric + dev/test set and doesn't correspond to doing well in the real world application, we have to change our metric and/or dev/test set 🏳

📚 Common Terms

📑 More Detailed

✨ Popular Detection CNNs

• R-CNN (Regional Based Convolutional Neural Networks)

• Fast R-CNN (Regional Based Convolutional Neural Networks)

• Faster R-CNN (Regional Based Convolutional Neural Networks)

• RFCN (Region Based Fully Connected Convolutional Neural Networks)

• YOLO (You Only Look Once)

  • YOLO V1

  • YOLO V2

  • YOLO V3

• SSD (Single Shot Detection)

🤸‍♀️ Object Detection Series

👩‍🏫 Notation

In the context of text processing (e.g: Natural Language Processing NLP)

🚀 One Hot Encoding

A way to represent words so we can treat with them easily

🔎 Example

Let's say that we have a dictionary that consists of 10 words (🤭) and the words of the dictionary are:

• Car, Pen, Girl, Berry, Apple, Likes, The, And, Boy, Book.

Our $$X^{(i)}$$ is: The Girl Likes Apple And Berry

So we can represent this sequence like the following 👀

By representing sequences in this way we can feed out data to neural networks ✨

🙄 Disadvantage

• If our dictionary consists of 10,000 words so each vector will be 10,000 dimensional 🤕

• This representation can not capture semantic features 💔

In short: Learning from one task and applying knowledge to separate tasks 🛰🚙

❓ What is Transfer Learning?

• 🕵️‍♀️ Transfer learning is a machine learning technique where a model trained on one task is re-purposed on a second related task.

• 🌟 In addition, it is an optimization method that allows rapid progress or improved performance when modeling the second task.

• 🤸‍♀️ Transfer learning only works in deep learning if the model features learned from the first task are general.

Long story short: Rather than training a neural network from scratch we can instead download an open-source model that someone else has already trained on a huge dataset maybe for weeks and use these parameters as a starting point to train our model just a little bit more with the smaller dataset that we have ✨

💫 Traditional ML vs Transfer Learning

🙄 Problem

Layers in a neural network can sometimes end up having similar weights and possible impact each other leading to over-fitting. With a big complex model it's a risk. So if you can imagine the dense layers can look a little bit like this.

We can drop out some neurons that has similar weights with neighbors, so that overfitting is being removed.

🔃 Comparison

🤸‍♀️ An NN before and after dropout

✨ Accuracy before and after dropout

🤔 When is it practical?

It is practical when we have a lot of data for problem that we are transferring from and usually relatively less data for the problem we are transferring to 🕵️‍

More accurately:

For task A to task B, it is sensible to do transfer learning from A to B when:

• 🚩 Task A and task B have the same output x

• ⭐ We have a lot more data for task A than task B

• 🔎 Low level features from task A could be helpful for learning task B

🧐 References

PDFs will be categorized as they increase 👩‍🔧

📚 PDFs that I found and recommend

📂 Table of Contents

👗 What is MNIST?

The MNIST database: (Modified National Institute of Standards and Technology database)

• 🔎 Fashion-MNIST is consisting of a training set of 60,000 examples and a test set of 10,000 examples

• 🎨 Types:

  • 🔢 MNIST: for handwritten digits

  • 👗 Fashion-MNIST: for fashion

• 📃 Properties:

  • 🌚 Grayscale

  • 28x28 px

  • 10 different categories

  • Repo

📚 Important Terms

The main purpose of activation function is to convert a input signal of a node in a NN to an output signal. That output signal now is used as a input in the next layer in the stack 💥

💫 Notes on performance

• Values in MNIST are between 0-255 but neural networks work better with normalized data, so we can divide every value by 255 so the values are between 0,1.

• There are multiple criterias to stop training process, we can specify number of epochs or a threshold or both

  • Epochs: number of iterations

  • Threshold: a threshold for accuracy or loss after each iteration

  • Threshold with maximum number of epochs

We can check the accuracy at the end of each epoch by Callbacks 💥

👩‍💻 My Codes

• 👗 Fashion MNIST

• 1️⃣ Digit MNIST

• 🎈 Main Workflow

• 🎨 Detailed Classification

🧐 References

• Official Documentation of Keras

• More About Activation Functions

To implement our CNN based works in the Browser we need to use Tensorflow.JS 🚀

👷‍♀️ Workflow

1. 🚙 Import Tensorflow.js

2. 👷‍♀️ Create models

3. 👩‍🏫 Train

4. 👩‍⚖️ Do inference

🚙 Importing Tensorflow.js

We can import Tensorflow.js in the way below

    <script 
        src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest">
    </script>

👷‍♀️ Creating The Model

😎 Same as we did in Python:

1. 🐣 Decalre a Sequential object

2. 👩‍🔧 Add layers

3. 🚀 Compile the model

4. 👩‍🎓 Train (fit)

5. 🐥 Use the model to predict

// create sequential 
const model = tf.sequential();

// add layer(s)
model.add(tf.layers.dense({units: 1, inputShape: [1]}));

// set compiling parameters and compile the model
model.compile({loss:'meanSquaredError', 
                optimizer:'sgd'});

// get summary of the mdoel
model.summary();

// create sample data set
const xs = tf.tensor2d([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], [6, 1]);
const ys = tf.tensor2d([-3.0, -1.0, 2.0, 3.0, 5.0, 7.0], [6, 1]);

// train
doTraining(model).then(() => {
    // after training
    predict = model.predict(tf.tensor2d([10], [1,1]));
    predict.print();
});

([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], [6, 1])

[-1.0, 0.0, 1.0, 2.0, 3.0, 4.0]: Data set values

[6, 1]: Shape of input

👁‍🗨 Attention

• 🐢 Training is a long process so that we have to do it in an asynchronous function

async function doTraining(model){
  const history = 
  await model.fit(xs, ys, 
      { epochs: 500,
          callbacks:{
              onEpochEnd: async(epoch, logs) =>{
                  console.log("Epoch:" 
                      + epoch 
                      + " Loss:" 
                      + logs.loss);

              }
          }
      });
}

👩‍💻 Full Code

• 🐾 Here

Like every first app we should start with something super simple that gives us an idea about the whole methodology.

✨ What is Keras?

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.

📚 Important Terms

👩‍🔬 The Simplest Neural Network

It contains one layer with one neuron.

👩‍💻 Code Example

# initialize the model
model = Sequential()

# add a layer with one unit and set the dimension of input 
model.add(Dense(units=1, input_shape=[1]))

# set functional properties and compile the model
model.compile(optimizer='sgd', loss='mean_squared_error'

After building out neural network we can feed it with our sample data 😋

👩‍💻 Code Example

xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)

Then we have to start training process 🚀

👩‍💻 Code Example

model.fit(xs, ys, epochs=500)

Every thing is done 😎 ! Now we can test our neural network with new data 🎉

👩‍💻 Code Example

print(model.predict([10.0]))

👩‍💻 My Code

• Full source code is here 🐾

• Tensorflow.js in browser here 🐾

🔃 Traditional Programming vs Machine Learning

🧐 References

• Official Documentation of Keras

• More About Sequential model

• More About Optimizers in Keras

• More About Loss Functions in Keras

Given a dataset like:

$[(x^{1},y^{1}), (x^{2},y^{2}), ...., (x^{m},y^{m})]$

We want:

$\hat{y}^{(i)} \approx y^{(i)}$

📚 Basic Concepts and Notations

In other words: The Cost Function measures how well our parameters w and b are doing on the training set, so the best w and b are the values that minimize 𝙹(w, b) as possible

📉 Gradient Descent

General Formula:

$w:=w-\alpha\frac{dJ(w,b)}{dw}$

$b:=b-\alpha\frac{dJ(w,b)}{dw}$

α (alpha) is the Learning Rate

🥽 Learning Rate

It is a positive scalar determining the size of the step of each iteration of gradient descent due to the corresponded estimated error each time the model weights are updated, so, it controls how quickly or slowly a neural network model learns a problem.

🎀 Good Learning Rate

💢 Bad Learning Rate

🧐 References

• Introduction to Artificial Neural Networks (ANN)

• More on Learning Rate

The main purpose of Activation Functions is to convert an input signal of a node in an ANN to an output signal by applying a transformation. That output signal now is used as a input in the next layer in the stack.

📃 Types of Activaiton Functions

📈 Linear Activation Function (Identity Function)

Formula:

Graph:

It can be used in regression problem in the output layer

🎩 Sigmoid Function

Formula:

Graph:

🎩 Tangent Function

Almost always strictly superior than sigmoid function

Formula:

Shifted version of the Sigmoid function 🤔

Graph:

Activation functions can be different for different layers, for example, we may use tanh for a hidden layer and sigmoid for the output layer

🙄 Downsides on Tanh and Sigmoid

If z is very large or very small then the derivative (or the slope) of these function becomes very small (ends up being close to 0), and so this can slow down gradient descent 🐢

🎩 Rectified Linear Activation Unit (Relu ✨)

Another and very popular choice

Formula:

Graph:

So the derivative is 1 when z is positive and 0 when z is negative

Disadvantage: derivative=0 when z is negative 😐

🎩 Leaky Relu

Formula:

Graph:

Or: 😛

🎀 Advantages of Relu's

• A lot of the space of z the derivative of the activation function is very different from 0

• NN will learn much faster than when using tanh or sigmoid

🤔 Why Do NNs Need non-linear Activation Functions

Well, if we use linear function then the NN is just outputting a linear function of the input, so no matter how many layers out NN has 🙄, all it is doing is just computing a linear function 😕

❗ Remember that the composition of two linear functions is itself a linear function

👩‍🏫 Rules For Choosing Activation Function

• If the output is 0 or 1 (binary classification) ➡ sigmoid is good for output layer

• For all other units ➡ Relu ✨

We can say that relu is the default choice for activation function

Note:

If you are not sure which one of these functions work best 😵, try them all 🤕 and evaluate on different validation set and see which one works better and go with that 🤓😇

🧐 Read More

😉 You Only Look Once

• 💥 The approach involves a single neural network trained end to end

  • It takes an image as input and predicts bounding boxes and class labels for each bounding box directly.

• 😕 The technique offers lower predictive accuracy (e.g. more localization errors) Compared with region based models

• ➗ YOLO divides the input image into an S×S grid. Each grid cell predicts only one object

👷‍♀️ Long Story Short: The system divides the input image into an S × S grid. If the center of an object falls into a grid cell, that grid cell is responsible for detecting that object.

🎀 Advantages

• 🚀 Speed

• 🤸‍♀️ Feasible for real time applications

🙄 Disadvantages

• 😕 Poor performance on small-sized objects

  • It tends to give imprecise object locations.

TODO: Compare versions of YOLO

🤸‍♀️ SSD

• 💥 Predicts objects in images using a single deep neural network.

• 🤓 The network generates scores for the presence of each object category using small convolutional filters applied to feature maps.

• ✌ This approach uses a feed-forward CNN that produces a collection of bounding boxes and scores for the presence of certain objects.

• ❗ In this model, each feature map cell is linked to a set of default bounding boxes

👩‍🏫 Details

• 🖼️ After going through a certain of convolutions for feature extraction, we obtain a feature layer of size m×n (number of locations) with p channels, such as 8×8 or 4×4 above.

  • And a 3×3 conv is applied on this m×n×p feature layer.

• 📍 For each location, we got k bounding boxes. These k bounding boxes have different sizes and aspect ratios.

  • The concept is, maybe a vertical rectangle is more fit for human, and a horizontal rectangle is more fit for car.

• 💫 For each of the bounding boxes, we will compute c class scores and 4 offsets relative to the original default bounding box shape.

🤓 Long Story Short

The SSD object detection algorithm is composed of 2 parts:

• Extract feature maps

• Apply convolution filters to detect objects.

🕵️‍♀️ Evaluation

• Better accuracy compared to YOLO

• Better speed compared to Region based algorithms

👀 Visualization

🚫 SSD vs YOLO

🧐 References

Important Terms

🙌 Padding

Adding an additional one border or more to the image so the image is n+2 x n+2 and after convolution we end up with n x n image which is the original size of the image

p = number of added borders

For convention: it is filled by 0

🤔 How much to pad?

For better understanding let's say that we have two concepts:

🕵️‍♀️ Valid Convolutions

It means no padding so:

n x n * f x f ➡ n-f+1 x n-f+1

🥽 Same Convolutions

Pad so that output size is the same as the input size.

So we want that 🧐:

n+2p-f+1 = n

Hence:

p = (f-1)/2

For convention f is chosen to be odd 👩‍🚀

👀 Visualization

🔢 Strided Convolution

Another approach of convolutions, we calculate the output by applying filter on regions by some value s.

👀 Visualization

🤗 To Generalize

For an n x n image and f x f filter, with p padding and stride s; the output image size can be calculated by the following formula

🚀 Convolutions Over Volume

To apply convolution operation on an RGB image; for example on 10x10 px RGB image, technically the image's dimension is 10x10x3 so we can apply for example a 3x3x3 filter or fxfx3 🤳

Filters can be applied on a special color channel 🎨

👀 Visualization

🤸‍♀️ Multiple Filters

🎨 Types of Layer In A Convolutional Network

👩‍🏫 Usually when people report number of layers in an NN they just report the number of layers that have weights and params

Convention: CONV1 + POOL1 = LAYER1

🤔 Why Convolotions?

• Better performance since they decrease the parameters that will be tuned 💫

🧐 References

📚 Common Terms

I did my best, my project is still doing bad, what shall I do? 😥

Well, in this stage we have a criteria, is your model doing worse than humans (Because humans are quite good at a lot of tasks 👩‍🎓)? If yes, you can:

• 👩‍🏫 Get labeled data from humans

• 👀 Gain insight from manual error analysis; (Why did a person get this right? 🙄)

• 🔎 Better analysis of bias / variance 🔍

🤔 Note: knowing how well humans can do on a task can help us to understand better how much we should try to reduce bias and variance

🧐 Is your model doing better than humans?

• Processes are less clear 😥

Suitable techniques will be added here

🤓 Study case

Let's assume that we have these two situations:

Even though training and dev errors are same we will apply different tactics for better performance

• In Case1, We have High Bias so we have to focus on bias reduction techniques 🤔, in other words we have to reduce the difference between training and human errors the avoidable error

  • Better algorithm, better NN structure, ......

• In Case2, We have High Variance so we have to focus on variance reduction techniques 🙄, in other words we have to reduce the difference between training and dev errors

  • Adding regularization, getting more data, ......

We call this procedure of analysis Error analysis 🕵️‍

👀 Error Types Visualization

In computer vision issues, human-level-error ≈ bayes-error because humans are good in vision tasks

🤗 Problems that ML surpasses human level performance

• Online advertising

• Product recommendations

• Logistics

• Loan approvals

• .....

✨ My Detailed Notes on Bias / Variance and Related Procedures

🤸‍♀️ It is recommended to

When we have a new project it is recommended to produce an initial model and then iterate over it until you get the best model, this is more practical than spending time building model theoretical and thinking about the best hyperparameter -which is almost impossible 🙄-

So, just don't overthink! (In both ML problems and life problems 🤗🙆‍)

🔢 LeNet-5

LeNet-5 is a very simple network - By modern standards -. It only has 7 layers;

• among which there are 3 convolutional layers (C1, C3 and C5)

• 2 sub-sampling (pooling) layers (S2 and S4)

• 1 fully connected layer (F6)

• Output layer

👀 Visualization of the network

🙌 Summary of the network

🛸 AlexNet

• Too similar to LeNet-5

• It has more filters per layer

• It uses ReLU instead of tanh

• SGD with momentum

• Uses dropout instead of regularaization

👀 Visualization of the network

🔎 More Detailed

🙌 Summary of the network

🌱 VGG-16

👀 Visualization of the network

🙌 Summary of the network

🔎 More Detailed

😐 Drawbacks

• It is painfully slow to train (It has 138 million parameters 🙄)

👩‍🔧 Implementation

🧐 Read More

⛓ Sequence Models In General

• Sequences are data structures where each example could be seen as a series of data points, for example 🧐:

• Since we have labeled data X and Y so all of these tasks are addressed as Supervised Learning 👩‍🏫

• Even in Sequence-to-Sequence tasks lengths of input and output can be different ❗

🤔 Why Do We Need Sequence Models?

• Machine learning algorithms typically require the text input to be represented as a fixed-length vector 🙄

• Thus, to model sequences, we need a specific learning framework able to:

  • ✔ Deal with variable-length sequences

  • ✔ Maintain sequence order

  • ✔ Keep track of long-term dependencies rather than cutting input data too short

  • ✔ Share parameters across the sequence (so not re-learn things across the sequence)

👩‍💻 My Codes

🚩 Main flow of programs in Tensorflow

1. Create Tensors (variables) that are not yet executed/evaluated.

2. Write operations between those Tensors.

3. Initialize your Tensors.

4. Create a Session.

5. Run the Session. This will run the operations you'd written above.

To summarize, remember to initialize your variables, create a session and run the operations inside the session. 👩‍🏫

👩‍💻 Code Example

To calculate the following formula:

$loss=L(\hat{y},y)=(\hat{y}^{(i)}-y^{(i)})^2$

# Creating tensors and writing operations between them 
y_hat = tf.constant(36, name='y_hat')
y = tf.constant(39, name='y')
loss = tf.Variable((y - y_hat)**2, name='loss')

# Initializing tensors
init = tf.global_variables_initializer()

# Creating session
with tf.Session() as session: 
    # Running the operations
    session.run(init) 

    # printing results
    print(session.run(loss))

When we created a variable for the loss, we simply defined the loss as a function of other quantities, but did not evaluate its value. To evaluate it, we had to use the initializer.

❗ Değişken Başlatma (initalization) Hakkında Not

For the following code:

a = tf.constant(2)
b = tf.constant(10)
c = tf.multiply(a,b)
print(c)

🤸‍♀️ The output is

Tensor("Mul:0", shape=(), dtype=int32)

As expected, we will not see 20 🤓! We got a tensor saying that the result is a tensor that does not have the shape attribute, and is of type "int32". All we did was put in the 'computation graph', but we have not run this computation yet.

📦 Placeholders in TF

• A placeholder is an object whose value you can specify only later. To specify values for a placeholder, we can pass in values by using a feed dictionary.

• Below, a placeholder has been created for x. This allows us to pass in a number later when we run the session.

x = tf.placeholder(tf.int64, name = 'x')
print(sess.run(2 * x, feed_dict = {x: 3}))
sess.close()

🎀 More examples

Computing sigmoid function with TF

def sigmoid(z):
    """
    Computes the sigmoid of z

    Arguments:
    z -- input value, scalar or vector

    Returns: 
    results -- the sigmoid of z
    """

    # Creating a placeholder for x. Naming it 'x'.
    x =  tf.placeholder(tf.float32, name = 'x')

    # computing sigmoid(x)
    sigmoid = tf.sigmoid(x)

    # Creating a session, and running it.
    with tf.Session() as sess:
        # Running session and call the output "result"
        result = sess.run(sigmoid, feed_dict = {x: z})

    return result

Computing cost function with TF

def cost(logits, labels):
    """
    Computes the cost using the sigmoid cross entropy

    Arguments:
    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)
    labels -- vector of labels y (1 or 0) 

    Returns:
    cost -- runs the session of the cost function
    """

    # Creating the placeholders for "logits" (z) and "labels" (y)
    z = tf.placeholder(tf.float32, name = 'z')
    y = tf.placeholder(tf.float32, name = 'y')

    # Using the loss function
    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits = z,  labels = y)

    # Creating a session
    sess = tf.Session()

    # Running the session 
    cost = sess.run(cost, feed_dict = {z: logits, y: labels})

    # Closing the session
    sess.close()

    return cost

This section will be filled by codes and notes gradually

👩‍💻 Codes

1. 👶 Basic CNNs

2. 👀 CNN Visualization

3. 👨‍👩‍👧‍👧 Human vs Horse Classifier with CNN

4. 🐱 Dog vs Cat Classifier with CNN

5. 🎨 Multi-Class Classification

6. 🌐 Tensorflow.js based hand written digit recognizer

   1. Classifier.js

   2. MNISTData.js

   3. index.html

✋ RPS Dataset

• Rock Paper Scissors is an available dataset containing 2,892 images of diverse hands in Rock/Paper/Scissors poses.

• Rock Paper Scissors contains images from a variety of different hands, from different races, ages and genders, posed into Rock / Paper or Scissors and labelled as such.

🔎 All of this data is posed against a white background. Each image is 300×300 pixels in 24-bit color

🐛 CNN Debugging

We can get info about our CNN by

model.summary()

And the output will be like:

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_18 (Conv2D)           (None, 26, 26, 64)        640       
_________________________________________________________________
max_pooling2d_18 (MaxPooling (None, 13, 13, 64)        0         
_________________________________________________________________
conv2d_19 (Conv2D)           (None, 11, 11, 64)        36928     
_________________________________________________________________
max_pooling2d_19 (MaxPooling (None, 5, 5, 64)          0         
_________________________________________________________________
flatten_9 (Flatten)          (None, 1600)              0         
_________________________________________________________________
dense_14 (Dense)             (None, 128)               204928    
_________________________________________________________________
dense_15 (Dense)             (None, 10)                1290      
=================================================================

👩‍💻 For code in the notebook:

Here 🐾

• 🔎 The original dimensions of the images were 28x28 px

• 1️⃣ 1st layer: The filter can not be applied on the pixels on the edges

  • The output of first layer has 26x26 px

• 2️⃣ 2nd layer: After applying 2x2 max pooling the dimensions will be divided by 2

  • The output of this layer has 13x13 px

• 3️⃣ 3rd layer: The filter can not be applied on the pixels on the edges

  • The output of this layer has 11x11 px

• 4️⃣ 4th layer: After applying 2x2 max pooling the dimensions will be divided by 2

  • The output of this layer has 5x5 px

• 5️⃣ 5th layer: The output of the previous layer will be flattened

  • This layer has 5x5x64=1600 units

• 6️⃣ 6th layer: We set it to contain 128 units

• 7️⃣ 7th layer: Since we have 10 categories it consists of 10 units

😵 😵

👀 Visualization

The visualization of the output of each layer is available here 🔎

👷‍♀️ Network Visualization Tool

Netron ✨✨

🧐 References

• Binary Cross-Entropy

• RMSProp Explained

• RMSProp in Tensorflow

• Binary Classification

• TensorFlow: an ML platform for solving impactful and challenging problems

• Rock Paper Scissors Dataset

📕 Notebooks

• 💠 Python built-in functions

• 🐼 String Processing with Pandas

💠 Python built-in functions

📏 Length of a string

🔢 Number of characters

text = "Beauty always reserved in details, don't let the big picture steal your attention!"
len(text)
# 82

🧾 Number of words

text = "Beauty always reserved in details, don't let the big picture steal your attention!"
words = text.split(' ')
len(words)
# 13

4️⃣ Getting words have length greater than 4

text = "Beauty always reserved in details, don't let the big picture steal your attention!"
words = text.split(' ')
moreThan4 = [w for w in words if len(w) > 4]
# ['Beauty', 'always', 'reserved', 'details,', "don't", 'picture', 'steal', 'attention!']

🎒 Words properties

🔠 Getting capitalized words

text = "Beauty Always reserved in details, Don't let the big picture steal your attention!"
words = text.split(' ')
capitalized = [w for w in words if w.istitle()]
# ['Beauty', 'Always']
# "Don't" is not found 🙄

🔚 Getting words end with specific end

• or specific start .startswith()

text = "You can hide whatever you want to hide but your eyes will always expose you, eyes never lie."
words = text.split(' ')
endsWithEr = [w for w in words if w.endswith('er')]
# ['whatever', 'never']

🐥 Upper and lower

"ESMA".isupper() # True
"Esma".isupper() # False
"esma".isupper() # False

"esma".islower() # True
"ESMA".islower() # False
"Esma".islower() # False

🤵 Membership test

'm' in 'esma' # True
'es' in 'esma' # True
'ed' in 'esma' # False

🕵️‍♀️ Unique Words

🔍 Case sensitive

text = "To be or not to be"
words = text.split(' ')
unique = set(words)
# {'be', 'To', 'not', 'or', 'to'}

✖️ 🔍 Ignore case

text = "To be or not to be"
words = text.split(' ')
unique = set(w.lower() for w in words)
# {'not', 'or', 'be', 'to'}

👮‍♀️ Checking Ops

Is Digit?

'17'.isdigit() # True
'17.7'.isdigit() # False

Is Alphabetic?

'esma'.isalpha() # True
'esma17'.isalpha() # False

Is alphabetic or number?

'17esma'.isalnum() # True
'17esma;'.isalnum() # False

🔤 String Ops

"Esma".lower() # esma
"Esma".upper() # ESMA
"EsmA".title() # Esma

🧵 Split & Join

Split due to specific character

text = "Beauty,Always,reserved,in,details,Don't,let,the,big,picture,steal,your,attention!"
words = text.split(',')
# ['Beauty', 'Always', 'reserved', 'in', 'details', "Don't", 'let', 'the', 'big', 'picture', 'steal', 'your', 'attention!']

Join by specific character

text = "Beauty,Always,reserved,in,details,Don't,let,the,big,picture,steal,your,attention!"
words = text.split(',')
joined = " ".join(words)
# Beauty Always reserved in details Don't let the big picture steal your attention!

📈 Data Normalization

It is a part of data preparation

• If we have a feature that is all positive or all negative, this will make learning harder for the nodes in the layer that follows. They will have to zigzag like the ones following a sigmoid activation function.

• If we transform our data so it has a mean close to zero, we will thereby make sure that there are both positive values and negative ones.

Formula:

$normalized=\frac{x_{i}-\mu }{\sigma}$

Benifit: It makes cost function J easier and faster to optimize 😋

🚩 Things to think well before implementing NN

Number of layers, number of hidden units, learning rates, activation functions...

It is too difficult to choose them all true at the first time so it is an iterative process

Idea ➡ Code ➡ Experiment ➡ Idea 🔁

So the point here is how to go efficiently around this cycle 🤔

👷‍♀️ Train / Dev / Test Splitting

For good evaluation it is good to split dataset like the following:

🤓 Training Set

The actual dataset that we use to train the model (weights and biases in the case of Neural Network).

The model sees and learns from this data 👶

😐 Validation (Development) Set

The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.

The model sees this data, but never learns from this 👨‍🚀

🧐 Test Set

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset. It provides the gold standard used to evaluate the model 🌟.

Implementation Note: Test set should contain carefully sampled data that spans the various classes that the model would face, when used in the real world 🚩🚩🚩❗❗❗

It is only used once a model is completely trained 👨‍🎓

😕 Bias / Variance

🕹 Bias

Bias is how far are the predicted values from the actual values. If the average predicted values are far off from the actual values then the bias is high.

Having high-bias implies that the model is too simple and does not capture the complexity of data thus underfitting the data 🤕

🕹 Variance

• Variance is the variability of model prediction for a given data point or a value which tells us spread of our data.

• Model with high variance fails to generalize on the data which it hasn’t seen before.

Having high-variance implies that algorithm models random noise present in the training data and it overfits the data 🤓

👀 Variance / Bias Visualization

↘ While implementing the model..

If we aren't able to get wanted performance we should ask these questions to improve our model:

We check the performance of the following solutions on dev set

1. Do we have high bias? If yes, it is a trainig problem, we may:

   • Try bigger network

   • Train longer

   • Try better optimization algorithm

   • Try another NN architecture

We can say that it is a structural problem 🤔

1. Do we have high variance? If yes, it is a dev set performance problem, we may:

   • Get more data

   • Do regularization

     • L2, dropout, data augmentation

We can say that maybe it is data or algorithmic problem 🤔

1. No high variance and no high bias?

TADAAA it is done 🤗🎉🎊

🧐 References

• About Train, Validation and Test Sets in Machine Learning

• Bias and Variance in Machine Learning

🌌 Vanishing Gradients with RNNs

• An RNN that process a sequence data with the size of 10,000 time steps, has 10,000 deep layers which is very hard to optimize 🙄

• Same in Deep Neural Networks, deeper networks are getting into the vanishing gradient problem.

• That also happens with RNNs with a long sequence size 🐛

🧙‍♀️ Solutions

• GRU Gated Recurrent Unit

• LSTM Long Short-Term Memory

🚪 Gated Recurrent Unit (GRU)

GRUs are improved version of standard recurrent neural network ✨, GRU uses update gate and reset gate .

• Basically, these are two vectors which decide what information should be passed to the output.

• The special thing about them is that they can be trained to keep information from long ago

  • Without washing it through time or removing information which is relevant to the prediction.

🔁 Update Gate

Given this gate the issue of the vanishing gradient is eliminated since the model on its own learn how much of the past information to pass to the future.

In short: How much past should matter now? 🙄

0️⃣ Reset Gate

This gate has the opposite functionality in comparison with the update gate since it is used by the model to decide how much of the past information to forget.

In short: Drop previous information? 🙄

💬 Current Memory Content

Memory content which will use the reset gate to store the relevant information from the past.

🎈 Final Memory at Current Time Step

A vector which holds information for the current unit and it will pass it further down to the network.

👀 Visualization

🎉 GRU Conclusion

• A solution to eliminate the vanishing gradient problem

• The model is not washing out the new input every single time but keeps the relevant information and passes it down to the next time steps of the network.

🤸‍♀️ Long Short-Term Memory

0️⃣ Forget Gate

• Let's assume we are reading words in a piece of text, and want use an LSTM to keep track of grammatical structures, such as whether the subject is singular or plural.

• If the subject changes from a singular word to a plural word, we need to find a way to get rid of our previously stored memory value of the singular/plural state.

• In an LSTM, the forget gate let us do this:

$$\Gamma ^{}_f = \sigma(W_f[a^{}, x^{}]+b_f)$$

• Here, $W_f$ are weights that govern the forget gate's behavior. We concatenate $$[a^{}, x^{}]$$ and multiply by $$W_f$$. The equation above results in a vector $$\Gamma_f^{}$$ with values between 0 and 1.

• This forget gate vector will be multiplied element-wise by the previous cell state $$c^{}$$.

• So if one of the values of $$\Gamma_f^{}$$ is 0 (or close to 0) then it means that the LSTM should remove that piece of information (e.g. the singular subject) in the corresponding component of $$c^{}$$ .

• If one of the values is 1, then it will keep the information.

🔄 Update Gate

Once we forget that the subject being discussed is singular, we need to find a way to update it to reflect that the new subject is now plural. Here is the formula for the update gate:

$$\Gamma ^{}_u = \sigma(W_u[a^{}, x^{}]+b_u)$$

Similar to the forget gate, here $$\Gamma_u^{}$$ is again a vector of values between 0 and 1. This will be multiplied element-wise with $$\tilde{c}^{}$$, in order to compute $$c^{⟨t⟩}$$.

👩‍🔧 Updating the Cell

To update the new subject we need to create a new vector of numbers that we can add to our previous cell state. The equation we use is:

$$\tilde{c}^{}=tanh(W_c[a^{}, x^{}]+b_c)$$

Finally, the new cell state is:

$$c^{}=\Gamma _f^{}c^{} + \Gamma _u^{}\tilde{c}^{}$$

🚪 Output Gate

To decide which outputs we will use, we will use the following two formulas:

$$\Gamma _o^{}=\sigma(W_o[a^{}, x^{}]+b_o)$$

$$a^{} = \Gamma _o^{}*tanh(c^{})$$

Where in first equation we decide what to output using a sigmoid function and in second equation we multiply that by the tanh of the previous state.

GRU is newer than LSTM, LSTM is more powerful but GRU is easier to implement 🚧

🧐 Read More

• What are RNNs and GRUs

• Understanding GRU Networks

• Detailed LSTM

🌚 Word Representation

This document may contain incorrect info 🙄‼ Please open a pull request to fix when you find a one 🌟

• One Hot Encoding

• Featurized Representation (Word Embedding)

• Word2Vec

• Skip Gram Model

• GloVe (Global Vectors for Word Representation)

🚀 One Hot Encoding

A way to represent words so we can treat with them easily

🔎 Example

Let's say that we have a dictionary that consists of 10 words (🤭) and the words of the dictionary are:

• Car, Pen, Girl, Berry, Apple, Likes, The, And, Boy, Book.

Our $$X^{(i)}$$ is: The Girl Likes Apple And Berry

So we can represent this sequence like the following 👀

Car   -0)  ⌈ 0 ⌉   ⌈ 0 ⌉   ⌈ 0 ⌉   ⌈ 0 ⌉  ⌈ 0 ⌉   ⌈ 0 ⌉ 
Pen   -1)  | 0 |  | 0 |  | 0 |  | 0 |  | 0 |  | 0 |
Girl  -2)  | 0 |  | 1 |  | 0 |  | 0 |  | 0 |  | 0 |
Berry -3)  | 0 |  | 0 |  | 0 |  | 0 |  | 0 |  | 1 |
Apple -4)  | 0 |  | 0 |  | 0 |  | 1 |  | 0 |  | 0 |
Likes -5)  | 0 |  | 0 |  | 1 |  | 0 |  | 0 |  | 0 |
The   -6)  | 1 |  | 0 |  | 0 |  | 0 |  | 0 |  | 0 |
And   -7)  | 0 |  | 0 |  | 0 |  | 0 |  | 1 |  | 0 |
Boy   -8)  | 0 |  | 0 |  | 0 |  | 0 |  | 0 |  | 0 |
Book  -9)  ⌊ 0 ⌋   ⌊ 0 ⌋   ⌊ 0 ⌋   ⌊ 0 ⌋  ⌊ 0 ⌋   ⌊ 0 ⌋

By representing sequences in this way we can feed our data to neural networks✨

🙄 Disadvantage

• If our dictionary consists of 10,000 words so each vector will be 10,000 dimensional 🤕

• This representation can not capture semantic features 💔

🎎 Featurized Representation (Word Embedding)

• Representing words by associating them with features such as gender, age, royal, food, cost, size.... and so on

• Every feature is represented as a range between [-1, 1]

• Thus, every word can be represented as a vector of these features

  • The dimension of each vector is related to the number of features that we pick

🔢 Embedded Matrix

For a given word w, the embedding matrix E is a matrix that maps its 1-hot representation $$o_w$$ to its embedding $$e_w$$ as follows:

$$e_w=Eo_w$$

🎀 Advantages

• Words that have the similar meaning have a similar representation.

• This model can capture semantic features ✨

• Vectors are smaller than vectors in one hot representation.

TODO: Subtracting vectors of oppsite words

🔄 Word2Vec

• Word2vec is a strategy to learn word embeddings by estimating the likelihood that a given word is surrounded by other words.

• This is done by making context and target word pairs which further depends on the window size we take.

  • Window size: a parameter that looks to the left and right of the context word for as many as window_size words

Creating Context to Target pairs with window size = 2 🙌

Skip Gram Model

The skip-gram word2vec model is a supervised learning task that learns word embeddings by assessing the likelihood of any given target word t happening with a context word c. By noting $$θ_{t}$$ a parameter associated with t, the probability P(t|c) is given by:

$$P(t|c)=\frac{exp(\theta^T_te_c)}{\sum_{j=1}^{|V|}exp(\theta^T_je_c)}$$

Remark: summing over the whole vocabulary in the denominator of the softmax part makes this model computationally expensive

🚀 One Hot Rep. vs Word Embedding

🧤 GloVe

The GloVe model, short for global vectors for word representation, is a word embedding technique that uses a co-occurence matrix X where each $$X_{ij}$$ denotes the number of times that a target i occurred with a context j. Its cost function J is as follows:

$$J(\theta)=\frac{1}{2}\sum_{i,j=1}^{|V|}f(X_{ij})(\theta^T_ie_j+b_i+b'j-log(X{ij}))^2$$

where f is a weighting function such that $$X_{ij}=0$$ ⟹ $$f(X_{ij})$$ = 0. Given the symmetry that e and θ play in this model, the final word embedding e $$e^{(final)}_w$$ is given by:

$$e^{(final)}_w=\frac{e_w+\theta_w}{2}$$

👩‍🏫 Conclusion of Word Embeddings

• If this is your first try, you should try to download a pre-trained model that has been made and actually works best.

• If you have enough data, you can try to implement one of the available algorithms.

• Because word embeddings are very computationally expensive to train, most ML practitioners will load a pre-trained set of embeddings.

🧐 References

• Recurrent Neural Networks Cheatsheet ✨

• NLP — Word Embedding & GloVe

🔄 Recurrent Neural Networks

🔎 Definition

A class of neural networks that allow previous outputs to be used as inputs to the next layers

They remember things they learned during training ✨

🧱 Architecture

🔶 The Whole RNN Architecture

🧩 An RNN Cell

Basic RNN cell. Takes as input $$x^{⟨t⟩}$$ (current input) and $$a^{⟨t−1⟩}$$ (previous hidden state containing information from the past), and outputs $$a^{⟨t⟩}$$ which is given to the next RNN cell and also used to predict $$y^{⟨t⟩}$$

⏩ Forward Propagation

To find $a^{}$ :

To find $\hat{y}^{}$ :

$$\hat{y}^{} = g(W_{ya}a^{}+b_y)$$

👀 Visualization

⏪ Back Propagation

Loss Function is defined like the following

$$L^{}(\hat{y}^{}, y^{})=-y^{}log(\hat{y})-(1-y^{})log(1-\hat{y}^{})$$

$$L(\hat{y},y)=\sum_{t=1}^{T_y}L^{}(\hat{y}^{}, y^{})$$

🎨 Types of RNNs

• 1️⃣ ➡ 1️⃣ One-to-One (Traditional ANN)

• 1️⃣ ➡ 🔢 One-to-Many (Music Generation)

• 🔢 ➡ 1️⃣ Many-to-One (Semantic Analysis)

• 🔢 ➡ 🔢 Many-to-Many $$T_x = T_y$$ (Speech Recognition)

• 🔢 ➡ 🔢 Many-to-Many $$T_x \neq T_y$$ (Machine Translation)

🔥 Advanced Recurrent Neural Networks

🔄 Bidirectional RNNs (BRNN)

• In many applications we want to output a prediction of $$y^{(t)}$$ which may depend on the whole input sequence

• Bidirectional RNNs combine an RNN that moves forward through time beginning from the start of the sequence with another RNN that moves backward through time beginning from the end of the sequence ✨

💬 In Other Words

• Bidirectional recurrent neural networks(RNN) are really just putting two independent RNNs together.

• The input sequence is fed in normal time order for one network, and in reverse time order for another.

• The outputs of the two networks are usually concatenated at each time step.

• 🎉 This structure allows the networks to have both backward and forward information about the sequence at every time step.

👎 Disadvantages

We need the entire sequence of data before we can make prediction anywhere.

e.g: not suitable for real time speech recognition

👀 Visualization

🕸 Deep RNNs

The computation in most RNNs can be decomposed into three blocks of parameters and associated transformations: 1. From the input to the hidden state, $$x^{(t)}$$ ➡ $$a^{(t)}$$ 2. From the previous hidden state to the next hidden state, $$a^{(t-1)}$$ ➡ $$a^{(t)}$$ 3. From the hidden state to the output, $$a^{(t)}$$ ➡ $$y^{(t)}$$

We can use multiple layers for each of the above transformations, which results in deep recurrent networks 😋

👀 Visualization

❌ Problem: Vanishing Gradients with RNNs

• An RNN that processes a sequence data with the size of 10,000 time steps, has 10,000 deep layers which is very hard to optimize 🙄

• Same in Deep Neural Networks, deeper networks are getting into the vanishing gradient problem 🥽

• That also happens with RNNs with a long sequence size 🐛

🧙‍♀️ Solutions

• Read Part-2 for my notes on Vanishing Gradients with RNNs 🤸‍♀️

🧐 Read More

• Recurrent Neural Networks Cheatsheet ✨

• All About RNNs 🚀

Having fast and good optimization algorithms can speed up the efficiency of the whole work ✨

🔩 Batch Gradient Descent

In batch gradient we use the entire dataset to compute the gradient of the cost function for each iteration of the gradient descent and then update the weights.

• Since we use the entire dataset to compute the gradient convergence is slow.

🎩 Stochastic Gradient Descent (SGD)

In stochastic gradient descent we use a single datapoint or example to calculate the gradient and update the weights with every iteration, we first need to shuffle the dataset so that we get a completely randomized dataset.

Random sample helps to arrive at a global minima and avoids getting stuck at a local minima.

• Learning is much faster and convergence is quick for a very large dataset 🚀

🔩 Mini Batch Gradient Descent

• Mini-batch gradient is a variation of stochastic gradient descent where instead of single training example, mini-batch of samples is used.

• Mini batch gradient descent is widely used and converges faster and is more stable.

• Batch size can vary depending on the dataset.

1 ≤ batch-size ≤ m, batch-size is a hyperparameter ❗

🔃 Comparison

• Very large batch-size (m or close to m):

  • Too long per iteration

• Very small batch-size (1 or close to 1)

  • losing speed up of vectorization

• Not batch-size too large/small

  • We can do vectorization

  • Good speed per iteration

  • The fastest (best) learning 🤗✨

🚩 Guidelines for Choosing Batch-Size

• For a small (m ≤ 2000) dataset ➡ use batch gradient descent

• Typical mini batch-size: 64, 128, 256, 512, up to 1024

• Make sure mini batch-size fits in your CPU/GPU memory

It is better(faster) to choose mini batch size as a power of 2 (due to memory issues) 🧐

🔩 Gradient Descent with Momentum

Almost always, gradient descent with momentum converges faster ✨ than the standard gradient descent algorithm. In the standard gradient descent algorithm, we take larger steps in one direction and smaller steps in another direction which slows down the algorithm. 🤕

This is what momentum can improve, it restricts the oscillation in one direction so that our algorithm can converge faster. Also, since the number of steps taken in the y-direction is restricted, we can set a higher learning rate. 🤗

The following image describes better: 🧐

Formula:

For better understanding:

In gradient descent with momentum, while we are trying to speed up gradient descent we can say that:

• Derivatives are the accelerator

• v's are the velocity

• β is the friction

🔩 RMSprop Optimizer

The RMSprop optimizer is similar to the gradient descent algorithm with momentum. The RMSprop optimizer restricts the oscillations in the vertical direction. Therefore, we can increase our learning rate and our algorithm could take larger steps in the horizontal direction converging faster.

The difference between RMSprop and gradient descent is on how the gradients are calculated, RMSProp gradients are calculated by the following formula:

✨ Adam Optimizer

Adam stands for: ADAptive Moment estimation

Commonly used algorithm nowadays, Adam can be looked at as a combination of RMSprop and Stochastic Gradient Descent with momentum. It uses the squared gradients to scale the learning rate like RMSprop and it takes advantage of momentum by using moving average of the gradient instead of gradient itself like SGD with momentum.

To summarize: Adam = RMSProp + GD with momentum + bias correction

😵😵😵

👩‍🏫 Hyperparameters choice (recommended values)

• α: needs to be tuned

• β1: 0.9

• β2: 0.999

• ε:

🧐 References

• 💥 Basics of Image Augmentation which is a technique to avoid overfitting

• ⭐ When we have got a small dataset we are able to manipluate the dataset without changing the underlying images to open up whole scenarios for training and to be able to train by variuos techniques of image augmentation

Note: Image augmentation is needed for both training and test set 😅

🚩 Basic Concept of Image Augmentation

👩‍🏫 The concept is very simple though:

If we have limited data, then the chances of you having data to match potential future predictions is also limited, and logically, the less data we have, the less chance we have of getting accurate predictions for data that our model hasn't yet seen.

🙄 If we are training a model to spot cats, and our model has never seen what a cat looks like when lying down, it might not recognize that in future.

• Augmentation simply amends our images on-the-fly while training using transforms like rotation.

• So, it could 'simulate' an image of a cat lying down by rotating a 'standing' cat by 90 degrees.

• As such we get a cheap ✨ way of extending our dataset beyond what we have already.

🔎 Note: Doing image augmentation in runtime is preferred rather than to do it on memory to keep original data as it is 🤔

🤸‍♀️ Image Augmentation Techniques

✅ Mirroring

Flipping the image horizontally

🚀 Example

✂ Random Cropping

Picking an image and taking random crops

🚀 Example

🎨 Color Shifting

Adding and subtracting some values from color channels

🚀 Example

📐 Shearing Transformation

Shear transformation slants the shape of the image

🚀 Example

👩‍💻 Code Example

The following code is used to do image augmentation

Full code example is 👈

🧐 References