🎤 About

• 🕸 My notes about Artificial Neural Networks, Convolutional Neural Networks and Recurrent Neural Networks with theoretical details

• 🦋 I will share new details as I learn new concepts in this context

📑 Table of Contents

💉 Extensions

🚀 Other Version

🙌 Quote

"Your learning algorithm has two main sources of knowledge; one is the data and other is whatever you hand design" 🤔🚀

⭐ Please..

• ✨ Help me to improve and to increase the content by opening a pull request

👜 Contact & Support

📚 Common Terms

A scalar is considered to have rank zero ❗❕

🔩 Vectorization

Vectorization is used to speed up the Python (or Matlab) code without using loop. Using such a function can help in minimizing the running time of code efficiently. Various operations are being performed over vector such as dot product of vectors, outer products of vectors and element wise multiplication.

➕ Advantages

• Faster execution (allows parallel operations) 👨‍🔧

• Simpler and more readable code :sparkles:

👀 Simple Visualization

👩‍💻 Code Examples

Finding the dot product of two arrays:

import numpy as np
array1 = np.random.rand(1000)
array2 = np.random.rand(1000)

# not vectorized version
result=0
for i in range(len(array1)):
  result += array1[i] * array2[i]
# result: 244.4311

# vectorized version
v_result = np.dot(array1, array2)
# v_result: 244.4311

📈 Applying exponential operation on every element of an array (or matrix)

array = np.random.rand(1000)
exp = np.exp(array)

🚀 Vectorized version of sigmoid function

array = np.random.rand(1000)
sigmoid = 1 / (1 + np.exp(-array))

👩‍💻 Common Supported Operations in Numpy

🤸‍♀️ Common single array functions

• Taking the square root of each element in the array

  • np.sqrt(x)

• Taking the sum over all of the array's elements

  • np.sum(x)

• Taking the absolute value of each element in the array

  • np.abs(x)

• Applying trigonometric functions on each element in the array

  • np.sin(x), np.cos(x), np.tan(x)

• Applying logarithmic functions on each element in the array

  • np.log(x), np.log10(x), np.log2(x)

🤸‍♂️ Common multiple array functions

• Applying arithmetic operations on corresponded elements in the arrays

  • np.add(x, y), np.subtract(x, y), np.divide(x, y), np.multiply(x, y)

• Applying power operation on corresponded elements in the arrays

  • np.power(x, y)

➰ Common sequential functions

• Getting mean of an array

  • np.mean(x)

• Getting median of an array

  • np.median(x)

• Getting variance of an array

  • np.var(x)

• Getting standart deviation of an array

  • np.std(x)

• Getting maximum or minimum value of an array

  • np.max(x), np.min(x)

• Getting index of maximum or minimum value of an array

  • np.argmax(x), np.argmin(x)

💉 Broadcasting

The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.

Practically:

If you have a matrix A that is (m,n) and you want to add / subtract / multiply / divide with B matrix (1,n) matrix then B matrix will be copied m times into an (m,n) matrix and then wanted operation will be applied

Similarly: If you have a matrix A that is (m,n) and you want to add / subtract / multiply / divide with B matrix (m,1) matrix then B matrix will be copied n times into an (m,n) matrix and then wanted operation will be applied

Long story short: Arrays (or matrices) with different sizes can not be added, subtracted, or generally be used in arithmetic. So it is a way to make it possible by stretching shapes so they have compatible shapes :sparkles:

👀 Simple Visualization

👩‍💻 Code Examples:

➕ Adding a (1,n) row vector to a (2,n) matrix

a = np.array([[0, 1, 2], 
              [5, 6, 7]] )
b = np.array([1, 2, 3])
print(a + b)

# Output: [[ 1  3  5]
#          [ 6  8 10]]

➖ Subtracting 'a' scalar from a matrix

a = np.array( [[0, 1, 2], 
               [5, 6, 7]] )
c = 2
print(a - c)
# Output: [[-2 -1  0]
#          [ 3  4  5]]

1️⃣ Rank 1 Array

👩‍💻 Code Example

x = np.random.rand(5)
print('shape:', x.shape, 'rank:', x.ndim)

# Output: shape: (5,) rank: 1

y = np.random.rand(5, 1)
print('shape:', y.shape, 'rank:', y.ndim)

# Output: shape: (5, 1) rank: 2

z = np.random.rand(5, 2, 2)
print('shape:', z.shape, 'rank:', z.ndim)

# Output: shape: (5, 2, 2) rank: 3

It is recommended not to use rank 1 arrays

🤔 Why it is recommended not to use 1 rank arrays?

Rank 1 arrays may cause bugs that are difficult to find and fix, for example:

Dot operation on rank 1 arrays:

a = np.random.rand(4)
b = np.random.rand(4)
print(a)
print(a.T)
print(np.dot(a,b))

# Output
# [0.40464616 0.46423665 0.26137661 0.07694073]
# [0.40464616 0.46423665 0.26137661 0.07694073]
# 0.354194202098512

Dot operation on rank 2 arrays:

a = np.random.rand(4,1)
b = np.random.rand(4,1)
print(a)
print(np.dot(a,b))

# Output
# [[0.68418713]
# [0.53098868]
# [0.16929882]
# [0.62586001]]
# [[0.68418713 0.53098868 0.16929882 0.62586001]]
# ERROR: shapes (4,1) and (4,1) not aligned: 1 (dim 1) != 4 (dim 0)

Conclusion: We have to avoid using rank 1 arrays in order to make our codes more bug-free and easy to debug 🐛

🧐 References

👷‍♀️ Network Visualization Tool

• Visualize the graph of the network

💫 CNN Input / Output Visualization Tool

• Watch the inputs and outputs of each layer in your CNN

🖼️ OpenImages Downloading Tool

• 🚀 Download images by class

🔗 Bulk Link Downloading Tool

• 💁‍♀️ Download bulk links by one click

• 👩‍💻 Google Chrome extension

Given a dataset like:

We want:

📚 Basic Concepts and Notations

In other words: The Cost Function measures how well our parameters w and b are doing on the training set, so the best w and b are the values that minimize 𝙹(w, b) as possible

📉 Gradient Descent

General Formula:

α (alpha) is the Learning Rate

🥽 Learning Rate

It is a positive scalar determining the size of the step of each iteration of gradient descent due to the corresponded estimated error each time the model weights are updated, so, it controls how quickly or slowly a neural network model learns a problem.

🎀 Good Learning Rate

💢 Bad Learning Rate

🧐 References

🔎 Definition

A neural network is a type of machine learning which models itself after the human brain. This creates an artificial neural network that via an algorithm allows the computer to learn by incorporating new data.

Neural networks are able to perform what has been termed deep learning. While the basic unit of the brain is the neuron, the essential building block of an artificial neural network is a perceptron which accomplishes simple signal processing, and these are then connected into a large mesh network.

📑 Types of NNs

There are many types of neural networks, choosing a type is due to the problem that we are trying to solve, for example

🎨 Types of Data in Supervised Learning

• 🚧 Structured Data

  • Such as tables

  • We have input fields and an output field

• 🤹‍♂️ Unstructured Data

  • Such as images, audio and texts

  • We need to use feature extraction algorithms to build our model

🧐 References

The main purpose of Activation Functions is to convert an input signal of a node in an ANN to an output signal by applying a transformation. That output signal now is used as a input in the next layer in the stack.

📃 Types of Activaiton Functions

📈 Linear Activation Function (Identity Function)

Formula:

Graph:

It can be used in regression problem in the output layer

🎩 Sigmoid Function

Formula:

Graph:

🎩 Tangent Function

Almost always strictly superior than sigmoid function

Formula:

Shifted version of the Sigmoid function 🤔

Graph:

Activation functions can be different for different layers, for example, we may use tanh for a hidden layer and sigmoid for the output layer

🙄 Downsides on Tanh and Sigmoid

If z is very large or very small then the derivative (or the slope) of these function becomes very small (ends up being close to 0), and so this can slow down gradient descent 🐢

🎩 Rectified Linear Activation Unit (Relu ✨)

Another and very popular choice

Formula:

Graph:

So the derivative is 1 when z is positive and 0 when z is negative

Disadvantage: derivative=0 when z is negative 😐

🎩 Leaky Relu

Formula:

Graph:

Or: 😛

🎀 Advantages of Relu's

• A lot of the space of z the derivative of the activation function is very different from 0

• NN will learn much faster than when using tanh or sigmoid

🤔 Why Do NNs Need non-linear Activation Functions

Well, if we use linear function then the NN is just outputting a linear function of the input, so no matter how many layers out NN has 🙄, all it is doing is just computing a linear function 😕

❗ Remember that the composition of two linear functions is itself a linear function

👩‍🏫 Rules For Choosing Activation Function

• If the output is 0 or 1 (binary classification) ➡ sigmoid is good for output layer

• For all other units ➡ Relu ✨

We can say that relu is the default choice for activation function

Note:

If you are not sure which one of these functions work best 😵, try them all 🤕 and evaluate on different validation set and see which one works better and go with that 🤓😇

🧐 Read More

We can learn it by likening it to logistic regression: 😋

Recall that logistic regression produces a decimal between 0 and 1.0. For example, a logistic regression output of 0.8 from an email classifier suggests an 80% chance of an email being spam and a 20% chance of it being not spam. Clearly, the sum of the probabilities of an email being either spam or not spam is 1.0.

Softmax extends this idea into the MULTI-CLASS world. That is, Softmax assigns decimal probabilities to each class in a multi-class problem. Those decimal probabilities must add up to 1.0.

• Its other name is Maximum Entropy (MaxEnt) Classifier

We can say that softmax regression generalizes logistic regression

Logistic regression is a special status of softmax where C = 2 🤔

📚 Notation

🎨 Softmax Layer

Softmax is implemented through a neural network layer just before the output layer. The Softmax layer must have the same number of nodes as the output layer.

💥 Softmax Activation Function

🔨 Hard Max function

Takes the output of softmax layer and convert it into 1 vs 0 vector (as I called it 🤭) which will be our ŷ

For example:

And so on 🐾

🔎 Loss Function

Y and ŷ are (C,m) dimensional matrices 👩‍🔧

🧐 Read More

Basic Concepts of ANN

🍭 Basic Neural Network

Convention: The NN in the image called to be a 2-layers NN since input layer is not being counted 📢❗

📚 Common Terms

🧠 What does an artificial neuron do?

It calculates a weighted sum of its input, adds a bias and then decides whether it should be fired or not due to an activaiton function

👩‍🔧 Parameters Dimension Control

Making sure that these dimensions are true help us to write better and bug-free :bug: codes

🎈 Summary of Forward Propagation Process

👩‍🔧 Vectorized Equations

$Z^{[l]} =W^{[l]}A^{[l-1]}+b^{[l]}$ $A^{[l]} = g^{[l]}(Z^{[l]})$

🎈 Summary of Back Propagation Process

👩‍🔧 Vectorized Equations

$dZ^{[l]}=dA^{[l]} * {g^{[l]}}'(Z^{[l]})$

$dW^{[l]}=\frac{1}{m}dZ^{[l]}A^{[l-1]T}$

$db^{[l]}=\frac{1}{m}np.sum(dZ^{[l]}, axis=1, keepdims=True)$

$dA^{[l-1]}=W^{[l]T}dZ^{[l]}$

➰➰ To Put Forward Prop. and Back Prop. Together

😵🤕

✨ Parameters vs Hyperparameters

👩‍🏫 Parameters

• $W^{[1]}, W^{[2]}, W^{[3]}$

• $b^{[1]}, b^{[2]}$

• ......

👩‍🔧 Hyperparameters

• Learning rate

• Number of iterations

• Number of hidden layers

• Number of hidden units

• Choice of activation function

• ......

We can say that hyperparameters control parameters 🤔

📈 Data Normalization

It is a part of data preparation

• If we have a feature that is all positive or all negative, this will make learning harder for the nodes in the layer that follows. They will have to zigzag like the ones following a sigmoid activation function.

• If we transform our data so it has a mean close to zero, we will thereby make sure that there are both positive values and negative ones.

Formula:

Benifit: It makes cost function J easier and faster to optimize 😋

🚩 Things to think well before implementing NN

Number of layers, number of hidden units, learning rates, activation functions...

It is too difficult to choose them all true at the first time so it is an iterative process

Idea ➡ Code ➡ Experiment ➡ Idea 🔁

So the point here is how to go efficiently around this cycle 🤔

👷‍♀️ Train / Dev / Test Splitting

For good evaluation it is good to split dataset like the following:

🤓 Training Set

The actual dataset that we use to train the model (weights and biases in the case of Neural Network).

The model sees and learns from this data 👶

😐 Validation (Development) Set

The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.

The model sees this data, but never learns from this 👨‍🚀

🧐 Test Set

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset. It provides the gold standard used to evaluate the model 🌟.

Implementation Note: Test set should contain carefully sampled data that spans the various classes that the model would face, when used in the real world 🚩🚩🚩❗❗❗

It is only used once a model is completely trained 👨‍🎓

😕 Bias / Variance

🕹 Bias

Bias is how far are the predicted values from the actual values. If the average predicted values are far off from the actual values then the bias is high.

Having high-bias implies that the model is too simple and does not capture the complexity of data thus underfitting the data 🤕

🕹 Variance

• Variance is the variability of model prediction for a given data point or a value which tells us spread of our data.

• Model with high variance fails to generalize on the data which it hasn’t seen before.

Having high-variance implies that algorithm models random noise present in the training data and it overfits the data 🤓

👀 Variance / Bias Visualization

↘ While implementing the model..

If we aren't able to get wanted performance we should ask these questions to improve our model:

We check the performance of the following solutions on dev set

1. Do we have high bias? If yes, it is a trainig problem, we may:

   • Try bigger network

   • Train longer

   • Try better optimization algorithm

   • Try another NN architecture

We can say that it is a structural problem 🤔

1. Do we have high variance? If yes, it is a dev set performance problem, we may:

   • Get more data

   • Do regularization

     • L2, dropout, data augmentation

We can say that maybe it is data or algorithmic problem 🤔

1. No high variance and no high bias?

TADAAA it is done 🤗🎉🎊

🧐 References

🚩 Main flow of programs in Tensorflow

1. Create Tensors (variables) that are not yet executed/evaluated.

2. Write operations between those Tensors.

3. Initialize your Tensors.

4. Create a Session.

5. Run the Session. This will run the operations you'd written above.

To summarize, remember to initialize your variables, create a session and run the operations inside the session. 👩‍🏫

👩‍💻 Code Example

To calculate the following formula:

$loss=L(\hat{y},y)=(\hat{y}^{(i)}-y^{(i)})^2$

# Creating tensors and writing operations between them 
y_hat = tf.constant(36, name='y_hat')
y = tf.constant(39, name='y')
loss = tf.Variable((y - y_hat)**2, name='loss')

# Initializing tensors
init = tf.global_variables_initializer()

# Creating session
with tf.Session() as session: 
    # Running the operations
    session.run(init) 

    # printing results
    print(session.run(loss))

When we created a variable for the loss, we simply defined the loss as a function of other quantities, but did not evaluate its value. To evaluate it, we had to use the initializer.

❗ Değişken Başlatma (initalization) Hakkında Not

For the following code:

a = tf.constant(2)
b = tf.constant(10)
c = tf.multiply(a,b)
print(c)

🤸‍♀️ The output is

Tensor("Mul:0", shape=(), dtype=int32)

As expected, we will not see 20 🤓! We got a tensor saying that the result is a tensor that does not have the shape attribute, and is of type "int32". All we did was put in the 'computation graph', but we have not run this computation yet.

📦 Placeholders in TF

• A placeholder is an object whose value you can specify only later. To specify values for a placeholder, we can pass in values by using a feed dictionary.

• Below, a placeholder has been created for x. This allows us to pass in a number later when we run the session.

x = tf.placeholder(tf.int64, name = 'x')
print(sess.run(2 * x, feed_dict = {x: 3}))
sess.close()

🎀 More examples

Computing sigmoid function with TF

def sigmoid(z):
    """
    Computes the sigmoid of z

    Arguments:
    z -- input value, scalar or vector

    Returns: 
    results -- the sigmoid of z
    """

    # Creating a placeholder for x. Naming it 'x'.
    x =  tf.placeholder(tf.float32, name = 'x')

    # computing sigmoid(x)
    sigmoid = tf.sigmoid(x)

    # Creating a session, and running it.
    with tf.Session() as sess:
        # Running session and call the output "result"
        result = sess.run(sigmoid, feed_dict = {x: z})

    return result

Computing cost function with TF

def cost(logits, labels):
    """
    Computes the cost using the sigmoid cross entropy

    Arguments:
    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)
    labels -- vector of labels y (1 or 0) 

    Returns:
    cost -- runs the session of the cost function
    """

    # Creating the placeholders for "logits" (z) and "labels" (y)
    z = tf.placeholder(tf.float32, name = 'z')
    y = tf.placeholder(tf.float32, name = 'y')

    # Using the loss function
    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits = z,  labels = y)

    # Creating a session
    sess = tf.Session()

    # Running the session 
    cost = sess.run(cost, feed_dict = {z: logits, y: labels})

    # Closing the session
    sess.close()

    return cost

Briefly: A technique to prevent overfitting -and reduce variance-

🙄 Problem

In over-fitting situation, our model tries to learn too well the details and the noise from the training data, which ultimately results in poor performance on the unseen data (test set).

The following graph describes better:

👩‍🏫 Better Definition for Regularization

It is a technique which makes slight modifications to the learning algorithm such that the model generalizes better. This in turn improves the model’s performance on the unseen data as well.

🔨 Regularization Techniques

🔩 L2 Regularization (Weight decay)

The most common type of regularization, given by following formula:

$J=Loss+\frac{\lambda}{2m}-\sum ||w||^{2}$

Here, lambda is the regularization parameter. It is the hyperparameter whose value is optimized for better results. L2 regularization is also known as weight decay as it forces the weights to decay towards zero (but not exactly zero)

🔩 Dropout

Another regularization method by eliminating some neurons in a specific ratio randomly

Simply: For each node of probability p, don’t update its input or output weights during backpropagation (Just drop it 😅)

Better visualiztion:

An NN before and after dropout

It is commonly used in computer vision, but its downside is that Cost function J is no longer well defined

🤡 Data Augmentation

The simplest way to reduce overfitting is to increase the size of the training data, it is not always possible since getting more data is too costly, but sometimes we can increase our data based on our data, for example:

• Doing transformations on images can maximize our data set

🛑 Early Stopping

It is a kind of cross-validation strategy where we keep one part of the training set as the validation set. When we see that the performance on the validation set is getting worse, we immediately stop the training on the model. This is known as early stopping.

🧐 Read More

📚 General Code Snippets in ML

💥 Sigmoid Function

🚀 Sigmoid Gradient

👩‍🔧 Reshaping Arrays (or images)

💥 Normalizing Rows

🎨 Softmax Function

🤸‍♀️ L1 Loss Function

🤸‍♂️ L2 Loss Function

🏃‍♀️ Propagation Function

💫 Gradient Descent (Optimization)

🕸 Basic Code Snippets for Simple NN

Functions of 2-layer NN

Input layer, 1 hidden layer and output layer

🚀 Parameter Initialization

⏩ Forward Propagation

🚩 Cost Function

⏪ Back Propagation

🔃 Updating Parameters

👗 What is MNIST?

The MNIST database: (Modified National Institute of Standards and Technology database)

• 🔎 Fashion-MNIST is consisting of a training set of 60,000 examples and a test set of 10,000 examples

• 🎨 Types:

  • 🔢 MNIST: for handwritten digits

  • 👗 Fashion-MNIST: for fashion

• 📃 Properties:

  • 🌚 Grayscale

  • 28x28 px

  • 10 different categories

📚 Important Terms

The main purpose of activation function is to convert a input signal of a node in a NN to an output signal. That output signal now is used as a input in the next layer in the stack 💥

💫 Notes on performance

• Values in MNIST are between 0-255 but neural networks work better with normalized data, so we can divide every value by 255 so the values are between 0,1.

• There are multiple criterias to stop training process, we can specify number of epochs or a threshold or both

  • Epochs: number of iterations

  • Threshold: a threshold for accuracy or loss after each iteration

  • Threshold with maximum number of epochs

We can check the accuracy at the end of each epoch by Callbacks 💥

👩‍💻 My Codes

🧐 References

This folder contains theoric details about CNNs

📚 Important Terms

💫 Notes on performance

• Training speed of a CNN is too slower than plain NN because of its computational complexity 🐢

🧐 References

Like every first app we should start with something super simple that gives us an idea about the whole methodology.

✨ What is Keras?

Keras is a high-level neural networks API, written in Python and capable of running on top of TensorFlow, CNTK, or Theano.

📚 Important Terms

👩‍🔬 The Simplest Neural Network

It contains one layer with one neuron.

👩‍💻 Code Example

# initialize the model
model = Sequential()

# add a layer with one unit and set the dimension of input 
model.add(Dense(units=1, input_shape=[1]))

# set functional properties and compile the model
model.compile(optimizer='sgd', loss='mean_squared_error'

After building out neural network we can feed it with our sample data 😋

👩‍💻 Code Example

xs = np.array([-1.0,  0.0, 1.0, 2.0, 3.0, 4.0], dtype=float)
ys = np.array([-3.0, -1.0, 1.0, 3.0, 5.0, 7.0], dtype=float)

Then we have to start training process 🚀

👩‍💻 Code Example

model.fit(xs, ys, epochs=500)

Every thing is done 😎 ! Now we can test our neural network with new data 🎉

👩‍💻 Code Example

print(model.predict([10.0]))

👩‍💻 My Code

🔃 Traditional Programming vs Machine Learning

🧐 References

To implement our CNN based works in the Browser we need to use Tensorflow.JS 🚀

👷‍♀️ Workflow

2. 👷‍♀️ Create models

3. 👩‍🏫 Train

4. 👩‍⚖️ Do inference

🚙 Importing Tensorflow.js

We can import Tensorflow.js in the way below

    <script 
        src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest">
    </script>

👷‍♀️ Creating The Model

😎 Same as we did in Python:

1. 🐣 Decalre a Sequential object

2. 👩‍🔧 Add layers

3. 🚀 Compile the model

4. 👩‍🎓 Train (fit)

5. 🐥 Use the model to predict

// create sequential 
const model = tf.sequential();

// add layer(s)
model.add(tf.layers.dense({units: 1, inputShape: [1]}));

// set compiling parameters and compile the model
model.compile({loss:'meanSquaredError', 
                optimizer:'sgd'});

// get summary of the mdoel
model.summary();

// create sample data set
const xs = tf.tensor2d([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], [6, 1]);
const ys = tf.tensor2d([-3.0, -1.0, 2.0, 3.0, 5.0, 7.0], [6, 1]);

// train
doTraining(model).then(() => {
    // after training
    predict = model.predict(tf.tensor2d([10], [1,1]));
    predict.print();
});

([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], [6, 1])

[-1.0, 0.0, 1.0, 2.0, 3.0, 4.0]: Data set values

[6, 1]: Shape of input

👁‍🗨 Attention

• 🐢 Training is a long process so that we have to do it in an asynchronous function

async function doTraining(model){
  const history = 
  await model.fit(xs, ys, 
      { epochs: 500,
          callbacks:{
              onEpochEnd: async(epoch, logs) =>{
                  console.log("Epoch:" 
                      + epoch 
                      + " Loss:" 
                      + logs.loss);

              }
          }
      });
}

👩‍💻 Full Code

📚 Important Terms

🎀 Convolution Example

🤔 How did we find -7?

We did element wise product then we get the sum of the result matrix; so:

3*1 + 1*0 + 1*(-1)
+
1*1 + 0*0 + 7*(-1)
+
2*1 + 3*0 + 5*(-1)
=
-7

And so on for other elements 🙃

👼 Visualization of Calculation

🔎 Edge Detection

An application of convolution operation

🔎 Edge Detection Examples

Result: horizontal lines pop out

Result: vertical lines pop out

🙄 What About The Other Numbers

There are a lot of ways we can put number inside elements of the filter.

For example Sobel filter is like:

1   0   -1
2   0   -2
1   0   -1

Scharr filter is like:

3    0   -3
10   0   -10
3    0   -3

Prewitt filter is like:

-1   0   1
-1   0   1
-1   0   1

So the point here is to pay attention to the middle row

And Roberts filter is like:

1    0
0   -1

✨ Another Approach

We can tune these numbers by ML approach; we can say that the filter is a group of weights that:

w1    w2   w3
w4    w5   w6
w7    w8   w9

By that we can get -learned- horizontal, vertical, angled, or any edge type automatically rather than getting them by hand.

🤸‍♀️ Computational Details

If we have an n*n image and we convolve it by f*f filter the the output image will be n-f+1*n-f+1

😐 Downsides

1. 🌀 If we apply many filters then our image shrinks.

2. 🤨 Pixels at corners aren't being touched enough, so we are throwing away a lot of information from the edges of the image .

💡 Solution

We can pad the image 💪

🧐 References

Having fast and good optimization algorithms can speed up the efficiency of the whole work ✨

🔩 Batch Gradient Descent

In batch gradient we use the entire dataset to compute the gradient of the cost function for each iteration of the gradient descent and then update the weights.

• Since we use the entire dataset to compute the gradient convergence is slow.

🎩 Stochastic Gradient Descent (SGD)

In stochastic gradient descent we use a single datapoint or example to calculate the gradient and update the weights with every iteration, we first need to shuffle the dataset so that we get a completely randomized dataset.

Random sample helps to arrive at a global minima and avoids getting stuck at a local minima.

• Learning is much faster and convergence is quick for a very large dataset 🚀

🔩 Mini Batch Gradient Descent

• Mini-batch gradient is a variation of stochastic gradient descent where instead of single training example, mini-batch of samples is used.

• Mini batch gradient descent is widely used and converges faster and is more stable.

• Batch size can vary depending on the dataset.

1 ≤ batch-size ≤ m, batch-size is a hyperparameter ❗

🔃 Comparison

• Very large batch-size (m or close to m):

  • Too long per iteration

• Very small batch-size (1 or close to 1)

  • losing speed up of vectorization

• Not batch-size too large/small

  • We can do vectorization

  • Good speed per iteration

  • The fastest (best) learning 🤗✨

🚩 Guidelines for Choosing Batch-Size

• For a small (m ≤ 2000) dataset ➡ use batch gradient descent

• Typical mini batch-size: 64, 128, 256, 512, up to 1024

• Make sure mini batch-size fits in your CPU/GPU memory

It is better(faster) to choose mini batch size as a power of 2 (due to memory issues) 🧐

🔩 Gradient Descent with Momentum

Almost always, gradient descent with momentum converges faster ✨ than the standard gradient descent algorithm. In the standard gradient descent algorithm, we take larger steps in one direction and smaller steps in another direction which slows down the algorithm. 🤕

This is what momentum can improve, it restricts the oscillation in one direction so that our algorithm can converge faster. Also, since the number of steps taken in the y-direction is restricted, we can set a higher learning rate. 🤗

The following image describes better: 🧐

Formula:

$v_{dW} = \beta v_{dW }+ (1-\beta)dW$

$v_{db} = \beta v_{db }+ (1-\beta)db$

$W = W -\alpha v_{dW}$

$b = b -\alpha v_{db}$

For better understanding:

In gradient descent with momentum, while we are trying to speed up gradient descent we can say that:

• Derivatives are the accelerator

• v's are the velocity

• β is the friction

🔩 RMSprop Optimizer

The RMSprop optimizer is similar to the gradient descent algorithm with momentum. The RMSprop optimizer restricts the oscillations in the vertical direction. Therefore, we can increase our learning rate and our algorithm could take larger steps in the horizontal direction converging faster.

The difference between RMSprop and gradient descent is on how the gradients are calculated, RMSProp gradients are calculated by the following formula:

$S_{dW} = \beta S_{dW} + (1-\beta)dW^2$

$S_{db} = \beta S_{db} + (1-\beta)db^2$

$W = W -\alpha\frac{dW}{\sqrt{S_{dW}}}$

$b = b -\alpha\frac{db}{\sqrt{S_{db}}}$

✨ Adam Optimizer

Adam stands for: ADAptive Moment estimation

Commonly used algorithm nowadays, Adam can be looked at as a combination of RMSprop and Stochastic Gradient Descent with momentum. It uses the squared gradients to scale the learning rate like RMSprop and it takes advantage of momentum by using moving average of the gradient instead of gradient itself like SGD with momentum.

To summarize: Adam = RMSProp + GD with momentum + bias correction

$v_{dW}=\beta_1v_{dW}+ (1-\beta_1)dW$

$v_{db}=\beta_1v_{db}+ (1-\beta_1)db$

$S_{dW}=\beta_2S_{dW}+ (1-\beta_2)dW^2$

$S_{db}=\beta_2S_{db}+ (1-\beta_2)db^2$

$v^{corrected}_{dW}=\frac{v_{dW}}{1-\beta^t_1}$

$v^{corrected}_{db}=\frac{v_{dW}}{1-\beta^t_1}$

$S^{corrected}_{dW}=\frac{S_{dW}}{1-\beta^t_2}$

$S^{corrected}_{db}=\frac{S_{db}}{1-\beta^t_2}$

$W = W-\alpha \frac{v^{corrected}_{dW}}{\sqrt{S^{corrected}_{dW}}+\epsilon}$

$b = b-\alpha \frac{v^{corrected}_{db}}{\sqrt{S^{corrected}_{db}}+\epsilon}$

😵😵😵

👩‍🏫 Hyperparameters choice (recommended values)

• α: needs to be tuned

• β1: 0.9

• β2: 0.999

• ε: $10^{-8}$

🧐 References

🔢 LeNet-5

LeNet-5 is a very simple network - By modern standards -. It only has 7 layers;

• among which there are 3 convolutional layers (C1, C3 and C5)

• 2 sub-sampling (pooling) layers (S2 and S4)

• 1 fully connected layer (F6)

• Output layer

👀 Visualization of the network

🙌 Summary of the network

🛸 AlexNet

• Too similar to LeNet-5

• It has more filters per layer

• It uses ReLU instead of tanh

• SGD with momentum

• Uses dropout instead of regularaization

👀 Visualization of the network

🔎 More Detailed

🙌 Summary of the network

🌱 VGG-16

👀 Visualization of the network

🙌 Summary of the network

🔎 More Detailed

😐 Drawbacks

• It is painfully slow to train (It has 138 million parameters 🙄)

👩‍🔧 Implementation

🧐 Read More

🧒👧 Face Verification

🙌 Comparison

🤸‍♀️ Solving Approach

🤳 One Shot Learning

Learning from one example (that we have in the database) to recognize the person again

🖇 The Process

• Get input image

• Check if it belongs to the faces you have in the DB

👓 How to Check?

We have to calculate the similarity between the input image and the image in the database, so:

• ⭕ Use some function that

  • similarity(img_in, img_db) = some_val

• 👷‍♀️ Specifiy a threshold value

• 🕵️‍♀️ Check the threshold and specify the output

🤔 What can the similarity function be?

🔷 Siamese Network

A CNN which is used in face verification context, it recievs two images as input, after applying convolutions it calculates a feature vector from each image and, calculates the difference between them and then gives outputs decision.

In other words: it encodes the given images

👀 Visualization

Architecture:

👩‍🏫 How to Train?

We can train the network by taking an anchor (basic) image A and comparing it with both a positive sample P and a negative sample N. So that:

• 🚧 The dissimilarity between the anchor image and positive image must low

• 🚧 The dissimilarity between the anchor image and the negative image must be high

So:

Another variable called margin, which is a hyperparameter is added to the loss equation. Margin defines how far away the dissimilarities should be, i.e if margin = 0.2 and d(a,p) = 0.5 then d(a,n) should at least be equal to 0.7. Margin helps us distinguish the two images better 🤸‍♀️

Therefore, by using this loss function we:

• 👩‍🏫 Calculate the gradients and with the help of the gradients

• 👩‍🔧 We update the weights and biases of the Siamese network.

For training the network, we:

• 👩‍🏫 Take an anchor image and randomly sample positive and negative images and compute its loss function

• 🤹‍♂️ Update its gradients

🌠 Neural Style Transfer

Generating an image G by giving a content image C and a style image S

👀 Visualization

So to generate G, our NN has to learn features from S and apply suitable filters on C

👩‍🎓 Methodology

Usually we optimize the parameters -weights and biases- of the NN to get the wanted performance, here in Neural Style Transfer we start from a blank image composed of random pixel values, and we optimize a cost function by changing the pixel values of the image 🧐

In other words, we:

• ⭕ Start with a blank image consists of random pixels

• 👩‍🏫 Define some cost function J

• 👩‍🔧 Iteratively modify each pixel so as to minimize our cost function

Long story short: While training NNs we update our weights and biases, but in style transfer, we keep the weights and biases constant, and instead update our image itself 🙌

⌚ Cost Function

We can define J as

Which:

• α and β hyperparameters