Basic Concepts of ANN

🍭 Basic Neural Network

Convention: The NN in the image called to be a 2-layers NN since input layer is not being counted 📢❗

📚 Common Terms

🧠 What does an artificial neuron do?

It calculates a weighted sum of its input, adds a bias and then decides whether it should be fired or not due to an activaiton function

My detailed notes on activation functions are here 👩‍🏫

👩‍🔧 Parameters Dimension Control

Making sure that these dimensions are true help us to write better and bug-free :bug: codes

🎈 Summary of Forward Propagation Process

👩‍🔧 Vectorized Equations

🎈 Summary of Back Propagation Process

👩‍🔧 Vectorized Equations

➰➰ To Put Forward Prop. and Back Prop. Together

😵🤕

✨ Parameters vs Hyperparameters

👩‍🏫 Parameters

• ......

👩‍🔧 Hyperparameters

• Learning rate

• Number of iterations

• Number of hidden layers

• Number of hidden units

• Choice of activation function

• ......

We can say that hyperparameters control parameters 🤔

👷‍♀️ Network Visualization Tool

• Visualize the graph of the network

• Netron ✨✨

💫 CNN Input / Output Visualization Tool

• Watch the inputs and outputs of each layer in your CNN

• Tensorspace 🎉

🖼️ OpenImages Downloading Tool

• 🚀 Download images by class

• 🔗 OID

🔗 Bulk Link Downloading Tool

• 💁‍♀️ Download bulk links by one click

• 👩‍💻 Google Chrome extension

• ⚓ Tab Save

Briefly: A technique to prevent overfitting -and reduce variance-

🙄 Problem

In over-fitting situation, our model tries to learn too well the details and the noise from the training data, which ultimately results in poor performance on the unseen data (test set).

The following graph describes better:

👩‍🏫 Better Definition for Regularization

It is a technique which makes slight modifications to the learning algorithm such that the model generalizes better. This in turn improves the model’s performance on the unseen data as well.

🔨 Regularization Techniques

🔩 L2 Regularization (Weight decay)

The most common type of regularization, given by following formula:

$J=Loss+\frac{\lambda}{2m}-\sum ||w||^{2}$

Here, lambda is the regularization parameter. It is the hyperparameter whose value is optimized for better results. L2 regularization is also known as weight decay as it forces the weights to decay towards zero (but not exactly zero)

🔩 Dropout

Another regularization method by eliminating some neurons in a specific ratio randomly

Simply: For each node of probability p, don’t update its input or output weights during backpropagation (Just drop it 😅)

Better visualiztion:

An NN before and after dropout

It is commonly used in computer vision, but its downside is that Cost function J is no longer well defined

🤡 Data Augmentation

The simplest way to reduce overfitting is to increase the size of the training data, it is not always possible since getting more data is too costly, but sometimes we can increase our data based on our data, for example:

• Doing transformations on images can maximize our data set

🛑 Early Stopping

It is a kind of cross-validation strategy where we keep one part of the training set as the validation set. When we see that the performance on the validation set is getting worse, we immediately stop the training on the model. This is known as early stopping.

🧐 Read More

• Long Story Short 😅: Overfitting and Regularization in Neural Networks

🔎 Definition

A neural network is a type of machine learning which models itself after the human brain. This creates an artificial neural network that via an algorithm allows the computer to learn by incorporating new data.

Neural networks are able to perform what has been termed deep learning. While the basic unit of the brain is the neuron, the essential building block of an artificial neural network is a perceptron which accomplishes simple signal processing, and these are then connected into a large mesh network.

📑 Types of NNs

There are many types of neural networks, choosing a type is due to the problem that we are trying to solve, for example

🎨 Types of Data in Supervised Learning

• 🚧 Structured Data

  • Such as tables

  • We have input fields and an output field

• 🤹‍♂️ Unstructured Data

  • Such as images, audio and texts

  • We need to use feature extraction algorithms to build our model

🧐 References

• Introduction to Artificial Neural Networks (ANN)

📚 Common Terms

A scalar is considered to have rank zero ❗❕

🔩 Vectorization

Vectorization is used to speed up the Python (or Matlab) code without using loop. Using such a function can help in minimizing the running time of code efficiently. Various operations are being performed over vector such as dot product of vectors, outer products of vectors and element wise multiplication.

➕ Advantages

• Faster execution (allows parallel operations) 👨‍🔧

• Simpler and more readable code :sparkles:

👀 Simple Visualization

👩‍💻 Code Examples

Finding the dot product of two arrays:

import numpy as np
array1 = np.random.rand(1000)
array2 = np.random.rand(1000)

# not vectorized version
result=0
for i in range(len(array1)):
  result += array1[i] * array2[i]
# result: 244.4311

# vectorized version
v_result = np.dot(array1, array2)
# v_result: 244.4311

📈 Applying exponential operation on every element of an array (or matrix)

array = np.random.rand(1000)
exp = np.exp(array)

🚀 Vectorized version of sigmoid function

array = np.random.rand(1000)
sigmoid = 1 / (1 + np.exp(-array))

👩‍💻 Common Supported Operations in Numpy

🤸‍♀️ Common single array functions

• Taking the square root of each element in the array

  • np.sqrt(x)

• Taking the sum over all of the array's elements

  • np.sum(x)

• Taking the absolute value of each element in the array

  • np.abs(x)

• Applying trigonometric functions on each element in the array

  • np.sin(x), np.cos(x), np.tan(x)

• Applying logarithmic functions on each element in the array

  • np.log(x), np.log10(x), np.log2(x)

🤸‍♂️ Common multiple array functions

• Applying arithmetic operations on corresponded elements in the arrays

  • np.add(x, y), np.subtract(x, y), np.divide(x, y), np.multiply(x, y)

• Applying power operation on corresponded elements in the arrays

  • np.power(x, y)

➰ Common sequential functions

• Getting mean of an array

  • np.mean(x)

• Getting median of an array

  • np.median(x)

• Getting variance of an array

  • np.var(x)

• Getting standart deviation of an array

  • np.std(x)

• Getting maximum or minimum value of an array

  • np.max(x), np.min(x)

• Getting index of maximum or minimum value of an array

  • np.argmax(x), np.argmin(x)

💉 Broadcasting

The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes.

Practically:

If you have a matrix A that is (m,n) and you want to add / subtract / multiply / divide with B matrix (1,n) matrix then B matrix will be copied m times into an (m,n) matrix and then wanted operation will be applied

Similarly: If you have a matrix A that is (m,n) and you want to add / subtract / multiply / divide with B matrix (m,1) matrix then B matrix will be copied n times into an (m,n) matrix and then wanted operation will be applied

Long story short: Arrays (or matrices) with different sizes can not be added, subtracted, or generally be used in arithmetic. So it is a way to make it possible by stretching shapes so they have compatible shapes :sparkles:

👀 Simple Visualization

👩‍💻 Code Examples:

➕ Adding a (1,n) row vector to a (2,n) matrix

a = np.array([[0, 1, 2], 
              [5, 6, 7]] )
b = np.array([1, 2, 3])
print(a + b)

# Output: [[ 1  3  5]
#          [ 6  8 10]]

➖ Subtracting 'a' scalar from a matrix

a = np.array( [[0, 1, 2], 
               [5, 6, 7]] )
c = 2
print(a - c)
# Output: [[-2 -1  0]
#          [ 3  4  5]]

1️⃣ Rank 1 Array

👩‍💻 Code Example

x = np.random.rand(5)
print('shape:', x.shape, 'rank:', x.ndim)

# Output: shape: (5,) rank: 1

y = np.random.rand(5, 1)
print('shape:', y.shape, 'rank:', y.ndim)

# Output: shape: (5, 1) rank: 2

z = np.random.rand(5, 2, 2)
print('shape:', z.shape, 'rank:', z.ndim)

# Output: shape: (5, 2, 2) rank: 3

It is recommended not to use rank 1 arrays

🤔 Why it is recommended not to use 1 rank arrays?

Rank 1 arrays may cause bugs that are difficult to find and fix, for example:

Dot operation on rank 1 arrays:

a = np.random.rand(4)
b = np.random.rand(4)
print(a)
print(a.T)
print(np.dot(a,b))

# Output
# [0.40464616 0.46423665 0.26137661 0.07694073]
# [0.40464616 0.46423665 0.26137661 0.07694073]
# 0.354194202098512

Dot operation on rank 2 arrays:

a = np.random.rand(4,1)
b = np.random.rand(4,1)
print(a)
print(np.dot(a,b))

# Output
# [[0.68418713]
# [0.53098868]
# [0.16929882]
# [0.62586001]]
# [[0.68418713 0.53098868 0.16929882 0.62586001]]
# ERROR: shapes (4,1) and (4,1) not aligned: 1 (dim 1) != 4 (dim 0)

Conclusion: We have to avoid using rank 1 arrays in order to make our codes more bug-free and easy to debug 🐛

🧐 References

• Official Documentation of Broadcasting in NumPy

🎤 About

• 🕸 My notes about Artificial Neural Networks, Convolutional Neural Networks and Recurrent Neural Networks with theoretical details

• 🦋 I will share new details as I learn new concepts in this context

📑 Table of Contents

💉 Extensions

🚀 Other Version

• Turkish version of this project is here

🙌 Quote

"Your learning algorithm has two main sources of knowledge; one is the data and other is whatever you hand design" 🤔🚀

⭐ Please..

• ✨ Help me to improve and to increase the content by opening a pull request

• 👓 Tell me your suggestions by sending me an email or opening an issue

👜 Contact & Support

Find me on LinkedIn and feel free to mail me, Asmaa 🦋

Given a dataset like:

$[(x^{1},y^{1}), (x^{2},y^{2}), ...., (x^{m},y^{m})]$

We want:

$\hat{y}^{(i)} \approx y^{(i)}$

📚 Basic Concepts and Notations

In other words: The Cost Function measures how well our parameters w and b are doing on the training set, so the best w and b are the values that minimize 𝙹(w, b) as possible

📉 Gradient Descent

General Formula:

$w:=w-\alpha\frac{dJ(w,b)}{dw}$

$b:=b-\alpha\frac{dJ(w,b)}{dw}$

α (alpha) is the Learning Rate

🥽 Learning Rate

It is a positive scalar determining the size of the step of each iteration of gradient descent due to the corresponded estimated error each time the model weights are updated, so, it controls how quickly or slowly a neural network model learns a problem.

🎀 Good Learning Rate

💢 Bad Learning Rate

🧐 References

• Introduction to Artificial Neural Networks (ANN)

• More on Learning Rate

We can learn it by likening it to logistic regression: 😋

Recall that logistic regression produces a decimal between 0 and 1.0. For example, a logistic regression output of 0.8 from an email classifier suggests an 80% chance of an email being spam and a 20% chance of it being not spam. Clearly, the sum of the probabilities of an email being either spam or not spam is 1.0.

Softmax extends this idea into the MULTI-CLASS world. That is, Softmax assigns decimal probabilities to each class in a multi-class problem. Those decimal probabilities must add up to 1.0.

• Its other name is Maximum Entropy (MaxEnt) Classifier

We can say that softmax regression generalizes logistic regression

Logistic regression is a special status of softmax where C = 2 🤔

📚 Notation

🎨 Softmax Layer

Softmax is implemented through a neural network layer just before the output layer. The Softmax layer must have the same number of nodes as the output layer.

💥 Softmax Activation Function

🔨 Hard Max function

Takes the output of softmax layer and convert it into 1 vs 0 vector (as I called it 🤭) which will be our ŷ

For example:

And so on 🐾

🔎 Loss Function

Y and ŷ are (C,m) dimensional matrices 👩‍🔧

🧐 Read More

The main purpose of Activation Functions is to convert an input signal of a node in an ANN to an output signal by applying a transformation. That output signal now is used as a input in the next layer in the stack.

📃 Types of Activaiton Functions

📈 Linear Activation Function (Identity Function)

Formula:

Graph:

It can be used in regression problem in the output layer

🎩 Sigmoid Function

Formula:

Graph:

🎩 Tangent Function

Almost always strictly superior than sigmoid function

Formula:

Shifted version of the Sigmoid function 🤔

Graph:

Activation functions can be different for different layers, for example, we may use tanh for a hidden layer and sigmoid for the output layer

🙄 Downsides on Tanh and Sigmoid

If z is very large or very small then the derivative (or the slope) of these function becomes very small (ends up being close to 0), and so this can slow down gradient descent 🐢

🎩 Rectified Linear Activation Unit (Relu ✨)

Another and very popular choice

Formula:

Graph:

So the derivative is 1 when z is positive and 0 when z is negative

Disadvantage: derivative=0 when z is negative 😐

🎩 Leaky Relu

Formula:

Graph:

Or: 😛

🎀 Advantages of Relu's

• A lot of the space of z the derivative of the activation function is very different from 0

• NN will learn much faster than when using tanh or sigmoid

🤔 Why Do NNs Need non-linear Activation Functions

Well, if we use linear function then the NN is just outputting a linear function of the input, so no matter how many layers out NN has 🙄, all it is doing is just computing a linear function 😕

❗ Remember that the composition of two linear functions is itself a linear function

👩‍🏫 Rules For Choosing Activation Function

• If the output is 0 or 1 (binary classification) ➡ sigmoid is good for output layer

• For all other units ➡ Relu ✨

We can say that relu is the default choice for activation function

Note:

If you are not sure which one of these functions work best 😵, try them all 🤕 and evaluate on different validation set and see which one works better and go with that 🤓😇

🧐 Read More

🚩 Main flow of programs in Tensorflow

1. Create Tensors (variables) that are not yet executed/evaluated.

2. Write operations between those Tensors.

3. Initialize your Tensors.

4. Create a Session.

5. Run the Session. This will run the operations you'd written above.

To summarize, remember to initialize your variables, create a session and run the operations inside the session. 👩‍🏫

👩‍💻 Code Example

To calculate the following formula:

$loss=L(\hat{y},y)=(\hat{y}^{(i)}-y^{(i)})^2$

# Creating tensors and writing operations between them 
y_hat = tf.constant(36, name='y_hat')
y = tf.constant(39, name='y')
loss = tf.Variable((y - y_hat)**2, name='loss')

# Initializing tensors
init = tf.global_variables_initializer()

# Creating session
with tf.Session() as session: 
    # Running the operations
    session.run(init) 

    # printing results
    print(session.run(loss))

When we created a variable for the loss, we simply defined the loss as a function of other quantities, but did not evaluate its value. To evaluate it, we had to use the initializer.

❗ Değişken Başlatma (initalization) Hakkında Not

For the following code:

a = tf.constant(2)
b = tf.constant(10)
c = tf.multiply(a,b)
print(c)

🤸‍♀️ The output is

Tensor("Mul:0", shape=(), dtype=int32)

As expected, we will not see 20 🤓! We got a tensor saying that the result is a tensor that does not have the shape attribute, and is of type "int32". All we did was put in the 'computation graph', but we have not run this computation yet.

📦 Placeholders in TF

• A placeholder is an object whose value you can specify only later. To specify values for a placeholder, we can pass in values by using a feed dictionary.

• Below, a placeholder has been created for x. This allows us to pass in a number later when we run the session.

x = tf.placeholder(tf.int64, name = 'x')
print(sess.run(2 * x, feed_dict = {x: 3}))
sess.close()

🎀 More examples

Computing sigmoid function with TF

def sigmoid(z):
    """
    Computes the sigmoid of z

    Arguments:
    z -- input value, scalar or vector

    Returns: 
    results -- the sigmoid of z
    """

    # Creating a placeholder for x. Naming it 'x'.
    x =  tf.placeholder(tf.float32, name = 'x')

    # computing sigmoid(x)
    sigmoid = tf.sigmoid(x)

    # Creating a session, and running it.
    with tf.Session() as sess:
        # Running session and call the output "result"
        result = sess.run(sigmoid, feed_dict = {x: z})

    return result

Computing cost function with TF

def cost(logits, labels):
    """
    Computes the cost using the sigmoid cross entropy

    Arguments:
    logits -- vector containing z, output of the last linear unit (before the final sigmoid activation)
    labels -- vector of labels y (1 or 0) 

    Returns:
    cost -- runs the session of the cost function
    """

    # Creating the placeholders for "logits" (z) and "labels" (y)
    z = tf.placeholder(tf.float32, name = 'z')
    y = tf.placeholder(tf.float32, name = 'y')

    # Using the loss function
    cost = tf.nn.sigmoid_cross_entropy_with_logits(logits = z,  labels = y)

    # Creating a session
    sess = tf.Session()

    # Running the session 
    cost = sess.run(cost, feed_dict = {z: logits, y: labels})

    # Closing the session
    sess.close()

    return cost

📚 General Code Snippets in ML

💥 Sigmoid Function

def sigmoid(x):
    """
    Arguments:
    x -- A scalar, an array or a matrix

    Return:
    result -- sigmoid(x)
    """

    result = 1 /( 1 + np.exp(-x) )

    return result

🚀 Sigmoid Gradient

    def sigmoid_derivative(x):
    """
    Computes the gradient (also called the slope or derivative) of the sigmoid function with respect to its input x.

    Arguments:
    x -- A scalar or numpy array

    Return:
    ds -- Your computed gradient.
    """

    s = 1 / (1 + np.exp(-x))
    ds = s * (1 - s)

    return ds

👩‍🔧 Reshaping Arrays (or images)

    def arr2vec(arr, target):
     """
    Argument:
    image -- a numpy array of shape (length, height, depth)

    Returns:
    v -- a vector of shape (length*height*depth, 1)
    """

    v = image.reshape(image.shape[0] * image.shape[1] * image.shape[2], 1)

    return v

💥 Normalizing Rows

def normalizeRows(x):
    """
    Argument:
    x -- A numpy matrix of shape (n, m)

    Returns:
    x -- The normalized (by row) numpy matrix.
    """

    # Finding norms
    x_norm = np.linalg.norm(x, axis=1, keepdims=True)

    # Dividing x by its norm
    x = x / x_norm

    return x

🎨 Softmax Function

 def softmax(x):
    """Calculates the softmax for each row of the input x.

    Argument:
    x -- A numpy matrix of shape (n,m)

    Returns:
    s -- A numpy matrix equal to the softmax of x, of shape (n,m)
    """

    # Applying exp() element-wise to x
    x_exp = np.exp(x)

    # Creating a vector x_sum that sums each row of x_exp
    x_sum = np.sum(x_exp, axis=1, keepdims=True)

    # Computing softmax(x) by dividing x_exp by x_sum.
    # numpy broadcasting will be used automatically.
    s = x_exp / x_sum

    return s

🤸‍♀️ L1 Loss Function

def L1(yhat, y):
    """
    Arguments:
    yhat -- vector of size m (predicted labels)
    y -- vector of size m (true labels)

    Returns:
    loss -- the value of the L1 loss function defined previously
    """

    loss = np.sum(np.abs(y - yhat))

    return loss

🤸‍♂️ L2 Loss Function

def L2(yhat, y):
    """
    Arguments:
    yhat -- vector of size m (predicted labels)
    y -- vector of size m (true labels)

    Returns:
    loss -- the value of the L2 loss function defined above
    """

    loss = np.sum((y - yhat) ** 2)

    return loss

🏃‍♀️ Propagation Function

def propagate(w, b, X, Y):
    """
    Implementation of the cost function and its gradient for the propagation

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    """

    m = X.shape[1]

    # FORWARD PROPAGATION (FROM X TO COST)

    # computing activation
    A = sigmoid( np.dot(w.T, X) + b ) 

    # computing cost
    cost = - np.sum( Y * np.log(A) + (1-Y) * np.log(1 - A) ) / m 

    # BACKWARD PROPAGATION (TO FIND GRAD)

    dw = (np.dot(X,(A-Y).T))/m
    db = np.sum(A-Y)/m

    grads = {"dw": dw,
             "db": db}

    return grads, cost

💫 Gradient Descent (Optimization)

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    """
    This function optimizes w and b by running a gradient descent algorithm

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of shape (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat), of shape (1, number of examples)
    num_iterations -- number of iterations of the optimization loop
    learning_rate -- learning rate of the gradient descent update rule
    print_cost -- True to print the loss every 100 steps

    Returns:
    params -- dictionary containing the weights w and bias b
    grads -- dictionary containing the gradients of the weights and bias with respect to the cost function
    costs -- list of all the costs computed during the optimization, this will be used to plot the learning curve.
    """

    costs = []

    for i in range(num_iterations):


        # Cost and gradient calculation
        grads, cost = propagate(w, b, X, Y)

        # Retrieve derivatives from grads
        dw = grads["dw"]
        db = grads["db"]

        # update rule
        w = w - learning_rate*dw
        b = b - learning_rate*db

        # Record the costs
        if i % 100 == 0:
            costs.append(cost)

        # Print the cost every 100 training iterations (optional)
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))

    params = {"w": w,
              "b": b}

    grads = {"dw": dw,
             "db": db}

    return params, grads, costs

🕸 Basic Code Snippets for Simple NN

Functions of 2-layer NN

Input layer, 1 hidden layer and output layer

🚀 Parameter Initialization

def initialize_parameters(n_x, n_h, n_y):
    """
    Argument:
    n_x -- size of the input layer
    n_h -- size of the hidden layer
    n_y -- size of the output layer

    Returns:
    params -- python dictionary containing your parameters:
                    W1 -- weight matrix of shape (n_h, n_x)
                    b1 -- bias vector of shape (n_h, 1)
                    W2 -- weight matrix of shape (n_y, n_h)
                    b2 -- bias vector of shape (n_y, 1)
    """
    # multiplying with 0.01 to minimize values
    W1 = np.random.randn(n_h,n_x) * 0.01
    b1 = np.zeros((n_h,1))
    W2 = np.random.randn(n_y,n_h) * 0.01
    b2 = np.zeros((n_y,1))

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

    return parameters

⏩ Forward Propagation

def forward_propagation(X, parameters):
    """
    Argument:
    X -- input data of size (n_x, m)
    parameters -- python dictionary containing your parameters (output of initialization function)

    Returns:
    A2 -- The sigmoid output of the second activation
    cache -- a dictionary containing "Z1", "A1", "Z2" and "A2"
    """

    # Retrieving each parameter from the dictionary "parameters"
    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']

    Z1 = np.dot(W1, X) + b1
    A1 = np.tanh(Z1)
    Z2 = np.dot(W2, A1) + b2
    A2 = sigmoid(Z2)

    cache = {"Z1": Z1,
             "A1": A1,
             "Z2": Z2,
             "A2": A2}

    return A2, cache

🚩 Cost Function

def compute_cost(A2, Y):
    """
    Computes the cross-entropy cost given in the formula

    Arguments:
    A2 -- The sigmoid output of the second activation, of shape (1, number of examples)
    Y -- "true" labels vector of shape (1, number of examples)   

    Returns:
    cost -- cross-entropy cost given in the formula

    """

    # Number of examples
    m = Y.shape[1] 

    # Computing the cross-entropy cost
    logprobs = np.multiply(np.log(A2), Y) + (1 - Y) * np.log(1 - A2)
    cost = - np.sum(logprobs) / m
    cost = float(np.squeeze(cost))  

    return cost

⏪ Back Propagation

def backward_propagation(parameters, cache, X, Y):
    """
    Implement the backward propagation using the previously given instructions.

    Arguments:
    parameters -- python dictionary containing our parameters 
    cache -- a dictionary containing "Z1", "A1", "Z2" and "A2".
    X -- input data of shape (2, number of examples)
    Y -- "true" labels vector of shape (1, number of examples)

    Returns:
    grads -- python dictionary containing your gradients with respect to different parameters
    """
    m = X.shape[1]

    # Retrieving W1 and W2 from the dictionary "parameters".
    W1 = parameters['W1']
    W2 = parameters['W2']

    # Retrieving also A1 and A2 from dictionary "cache".
    A1 = cache['A1']
    A2 = cache['A2']

    # Backward propagation: calculating dW1, db1, dW2, db2. 
    dZ2 = A2 - Y
    dW2 = np.dot(dZ2, A1.T) / m
    db2 = np.sum(dZ2, axis = 1, keepdims = True) / m
    dZ1 = np.dot(W2.T, dZ2) * (1 - A1 ** 2)
    dW1 = np.dot(dZ1, X.T) / m
    db1 = np.sum(dZ1, axis = 1, keepdims = True) / m

    grads = {"dW1": dW1,
             "db1": db1,
             "dW2": dW2,
             "db2": db2}

    return grads

🔃 Updating Parameters

def update_parameters(parameters, grads, learning_rate = 1.2):
    """
    Updates parameters using the gradient descent update rule given previously

    Arguments:
    parameters -- python dictionary containing your parameters 
    grads -- python dictionary containing your gradients 

    Returns:
    parameters -- python dictionary containing your updated parameters 
    """
    # Retrieving each parameter from the dictionary "parameters"
    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']

    # Retrieving each gradient from the dictionary "grads"
    dW1 = grads['dW1']
    db1 = grads['db1']
    dW2 = grads['dW2']
    db2 = grads['db2']

    # Updating rule for each parameter
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2

    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}

    return parameters

📚 Important Terms

🎀 Convolution Example

🤔 How did we find -7?

We did element wise product then we get the sum of the result matrix; so:

3*1 + 1*0 + 1*(-1)
+
1*1 + 0*0 + 7*(-1)
+
2*1 + 3*0 + 5*(-1)
=
-7

And so on for other elements 🙃

👼 Visualization of Calculation

🔎 Edge Detection

An application of convolution operation

🔎 Edge Detection Examples

Result: horizontal lines pop out

Result: vertical lines pop out

🙄 What About The Other Numbers

There are a lot of ways we can put number inside elements of the filter.

For example Sobel filter is like:

1   0   -1
2   0   -2
1   0   -1

Scharr filter is like:

3    0   -3
10   0   -10
3    0   -3

Prewitt filter is like:

-1   0   1
-1   0   1
-1   0   1

So the point here is to pay attention to the middle row

And Roberts filter is like:

1    0
0   -1

✨ Another Approach

We can tune these numbers by ML approach; we can say that the filter is a group of weights that:

w1    w2   w3
w4    w5   w6
w7    w8   w9

By that we can get -learned- horizontal, vertical, angled, or any edge type automatically rather than getting them by hand.

🤸‍♀️ Computational Details

If we have an n*n image and we convolve it by f*f filter the the output image will be n-f+1*n-f+1

😐 Downsides

1. 🌀 If we apply many filters then our image shrinks.

2. 🤨 Pixels at corners aren't being touched enough, so we are throwing away a lot of information from the edges of the image .

💡 Solution

We can pad the image 💪

🧐 References

• More on Convolutional Neural Networks

📈 Data Normalization

It is a part of data preparation

• If we have a feature that is all positive or all negative, this will make learning harder for the nodes in the layer that follows. They will have to zigzag like the ones following a sigmoid activation function.

• If we transform our data so it has a mean close to zero, we will thereby make sure that there are both positive values and negative ones.

Formula:

Benifit: It makes cost function J easier and faster to optimize 😋

🚩 Things to think well before implementing NN

Number of layers, number of hidden units, learning rates, activation functions...

It is too difficult to choose them all true at the first time so it is an iterative process

Idea ➡ Code ➡ Experiment ➡ Idea 🔁

So the point here is how to go efficiently around this cycle 🤔

👷‍♀️ Train / Dev / Test Splitting

For good evaluation it is good to split dataset like the following:

🤓 Training Set

The actual dataset that we use to train the model (weights and biases in the case of Neural Network).

The model sees and learns from this data 👶

😐 Validation (Development) Set

The sample of data used to provide an unbiased evaluation of a model fit on the training dataset while tuning model hyperparameters. The evaluation becomes more biased as skill on the validation dataset is incorporated into the model configuration.

The model sees this data, but never learns from this 👨‍🚀

🧐 Test Set

The sample of data used to provide an unbiased evaluation of a final model fit on the training dataset. It provides the gold standard used to evaluate the model 🌟.

Implementation Note: Test set should contain carefully sampled data that spans the various classes that the model would face, when used in the real world 🚩🚩🚩❗❗❗

It is only used once a model is completely trained 👨‍🎓

😕 Bias / Variance

🕹 Bias

Bias is how far are the predicted values from the actual values. If the average predicted values are far off from the actual values then the bias is high.

Having high-bias implies that the model is too simple and does not capture the complexity of data thus underfitting the data 🤕

🕹 Variance

• Variance is the variability of model prediction for a given data point or a value which tells us spread of our data.

• Model with high variance fails to generalize on the data which it hasn’t seen before.

Having high-variance implies that algorithm models random noise present in the training data and it overfits the data 🤓

👀 Variance / Bias Visualization

↘ While implementing the model..

If we aren't able to get wanted performance we should ask these questions to improve our model:

We check the performance of the following solutions on dev set

1. Do we have high bias? If yes, it is a trainig problem, we may:

   • Try bigger network

   • Train longer

   • Try better optimization algorithm

   • Try another NN architecture

We can say that it is a structural problem 🤔

1. Do we have high variance? If yes, it is a dev set performance problem, we may:

   • Get more data

   • Do regularization

     • L2, dropout, data augmentation

We can say that maybe it is data or algorithmic problem 🤔

1. No high variance and no high bias?

TADAAA it is done 🤗🎉🎊

🧐 References

This folder contains theoric details about CNNs

📚 Important Terms

💫 Notes on performance

• Training speed of a CNN is too slower than plain NN because of its computational complexity 🐢

🧐 References

• More on Convolutional Neural Networks

To implement our CNN based works in the Browser we need to use Tensorflow.JS 🚀

👷‍♀️ Workflow

1. 🚙 Import Tensorflow.js

2. 👷‍♀️ Create models

3. 👩‍🏫 Train

4. 👩‍⚖️ Do inference

🚙 Importing Tensorflow.js

We can import Tensorflow.js in the way below

    <script 
        src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@latest">
    </script>

👷‍♀️ Creating The Model

😎 Same as we did in Python:

1. 🐣 Decalre a Sequential object

2. 👩‍🔧 Add layers

3. 🚀 Compile the model

4. 👩‍🎓 Train (fit)

5. 🐥 Use the model to predict

// create sequential 
const model = tf.sequential();

// add layer(s)
model.add(tf.layers.dense({units: 1, inputShape: [1]}));

// set compiling parameters and compile the model
model.compile({loss:'meanSquaredError', 
                optimizer:'sgd'});

// get summary of the mdoel
model.summary();

// create sample data set
const xs = tf.tensor2d([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], [6, 1]);
const ys = tf.tensor2d([-3.0, -1.0, 2.0, 3.0, 5.0, 7.0], [6, 1]);

// train
doTraining(model).then(() => {
    // after training
    predict = model.predict(tf.tensor2d([10], [1,1]));
    predict.print();
});

([-1.0, 0.0, 1.0, 2.0, 3.0, 4.0], [6, 1])

[-1.0, 0.0, 1.0, 2.0, 3.0, 4.0]: Data set values

[6, 1]: Shape of input

👁‍🗨 Attention

• 🐢 Training is a long process so that we have to do it in an asynchronous function

async function doTraining(model){
  const history = 
  await model.fit(xs, ys, 
      { epochs: 500,
          callbacks:{
              onEpochEnd: async(epoch, logs) =>{
                  console.log("Epoch:" 
                      + epoch 
                      + " Loss:" 
                      + logs.loss);

              }
          }
      });
}

👩‍💻 Full Code

• 🐾 Here

👗 What is MNIST?

The MNIST database: (Modified National Institute of Standards and Technology database)

• 🔎 Fashion-MNIST is consisting of a training set of 60,000 examples and a test set of 10,000 examples

• 🎨 Types:

  • 🔢 MNIST: for handwritten digits

  • 👗 Fashion-MNIST: for fashion

• 📃 Properties:

  • 🌚 Grayscale

  • 28x28 px

  • 10 different categories

  • Repo

📚 Important Terms

The main purpose of activation function is to convert a input signal of a node in a NN to an output signal. That output signal now is used as a input in the next layer in the stack 💥

💫 Notes on performance

• Values in MNIST are between 0-255 but neural networks work better with normalized data, so we can divide every value by 255 so the values are between 0,1.

• There are multiple criterias to stop training process, we can specify number of epochs or a threshold or both

  • Epochs: number of iterations

  • Threshold: a threshold for accuracy or loss after each iteration

  • Threshold with maximum number of epochs

We can check the accuracy at the end of each epoch by Callbacks 💥

👩‍💻 My Codes

• 👗 Fashion MNIST

• 1️⃣ Digit MNIST

• 🎈 Main Workflow

• 🎨 Detailed Classification

🧐 References

• Official Documentation of Keras

• More About Activation Functions

Having fast and good optimization algorithms can speed up the efficiency of the whole work ✨

🔩 Batch Gradient Descent

In batch gradient we use the entire dataset to compute the gradient of the cost function for each iteration of the gradient descent and then update the weights.

• Since we use the entire dataset to compute the gradient convergence is slow.

🎩 Stochastic Gradient Descent (SGD)

In stochastic gradient descent we use a single datapoint or example to calculate the gradient and update the weights with every iteration, we first need to shuffle the dataset so that we get a completely randomized dataset.

Random sample helps to arrive at a global minima and avoids getting stuck at a local minima.

• Learning is much faster and convergence is quick for a very large dataset 🚀

🔩 Mini Batch Gradient Descent

• Mini-batch gradient is a variation of stochastic gradient descent where instead of single training example, mini-batch of samples is used.

• Mini batch gradient descent is widely used and converges faster and is more stable.

• Batch size can vary depending on the dataset.

1 ≤ batch-size ≤ m, batch-size is a hyperparameter ❗

🔃 Comparison

• Very large batch-size (m or close to m):

  • Too long per iteration

• Very small batch-size (1 or close to 1)

  • losing speed up of vectorization

• Not batch-size too large/small

  • We can do vectorization

  • Good speed per iteration

  • The fastest (best) learning 🤗✨

🚩 Guidelines for Choosing Batch-Size

• For a small (m ≤ 2000) dataset ➡ use batch gradient descent

• Typical mini batch-size: 64, 128, 256, 512, up to 1024

• Make sure mini batch-size fits in your CPU/GPU memory

It is better(faster) to choose mini batch size as a power of 2 (due to memory issues) 🧐

🔩 Gradient Descent with Momentum

Almost always, gradient descent with momentum converges faster ✨ than the standard gradient descent algorithm. In the standard gradient descent algorithm, we take larger steps in one direction and smaller steps in another direction which slows down the algorithm. 🤕

This is what momentum can improve, it restricts the oscillation in one direction so that our algorithm can converge faster. Also, since the number of steps taken in the y-direction is restricted, we can set a higher learning rate. 🤗

The following image describes better: 🧐

Formula:

$v_{dW} = \beta v_{dW }+ (1-\beta)dW$

$v_{db} = \beta v_{db }+ (1-\beta)db$

$W = W -\alpha v_{dW}$

$b = b -\alpha v_{db}$

For better understanding:

In gradient descent with momentum, while we are trying to speed up gradient descent we can say that:

• Derivatives are the accelerator

• v's are the velocity

• β is the friction

🔩 RMSprop Optimizer

The RMSprop optimizer is similar to the gradient descent algorithm with momentum. The RMSprop optimizer restricts the oscillations in the vertical direction. Therefore, we can increase our learning rate and our algorithm could take larger steps in the horizontal direction converging faster.

The difference between RMSprop and gradient descent is on how the gradients are calculated, RMSProp gradients are calculated by the following formula:

$S_{dW} = \beta S_{dW} + (1-\beta)dW^2$

$S_{db} = \beta S_{db} + (1-\beta)db^2$

$W = W -\alpha\frac{dW}{\sqrt{S_{dW}}}$

$b = b -\alpha\frac{db}{\sqrt{S_{db}}}$

✨ Adam Optimizer

Adam stands for: ADAptive Moment estimation

Commonly used algorithm nowadays, Adam can be looked at as a combination of RMSprop and Stochastic Gradient Descent with momentum. It uses the squared gradients to scale the learning rate like RMSprop and it takes advantage of momentum by using moving average of the gradient instead of gradient itself like SGD with momentum.

To summarize: Adam = RMSProp + GD with momentum + bias correction

$v_{dW}=\beta_1v_{dW}+ (1-\beta_1)dW$

$v_{db}=\beta_1v_{db}+ (1-\beta_1)db$

$S_{dW}=\beta_2S_{dW}+ (1-\beta_2)dW^2$

$S_{db}=\beta_2S_{db}+ (1-\beta_2)db^2$

$v^{corrected}_{dW}=\frac{v_{dW}}{1-\beta^t_1}$

$v^{corrected}_{db}=\frac{v_{dW}}{1-\beta^t_1}$

$S^{corrected}_{dW}=\frac{S_{dW}}{1-\beta^t_2}$

$S^{corrected}_{db}=\frac{S_{db}}{1-\beta^t_2}$

$W = W-\alpha \frac{v^{corrected}_{dW}}{\sqrt{S^{corrected}_{dW}}+\epsilon}$

$b = b-\alpha \frac{v^{corrected}_{db}}{\sqrt{S^{corrected}_{db}}+\epsilon}$

😵😵😵

👩‍🏫 Hyperparameters choice (recommended values)

• α: needs to be tuned

• β1: 0.9

• β2: 0.999

• ε: $10^{-8}$

🧐 References

• Machine learning Gradient Descent

Visualization of concepts explained in P1 and P2 to wrap them up 👩‍🎓

💫 Convolution

Applying a filter to extract features 🤗

Problem 😰: Images are shrinking 😱

😏 Take A Look At Padding

Images Are Too Large, Performance is Down 😔

😉 Let's See Pooling

🙄 Well, I have an RGB image

Filters must have depth that is equal to number of color channels

🤡 Ok, now I want to apply n filters

Depth of the output will be equal to n

🤗 Check Your Understanding With A Full Example

🧐 References

• DeepLearning series: Convolutional Neural Networks (😍✨✨✨)

Important Terms

🙌 Padding

Adding an additional one border or more to the image so the image is n+2 x n+2 and after convolution we end up with n x n image which is the original size of the image

p = number of added borders

For convention: it is filled by 0

🤔 How much to pad?

For better understanding let's say that we have two concepts:

🕵️‍♀️ Valid Convolutions

It means no padding so:

n x n * f x f ➡ n-f+1 x n-f+1

🥽 Same Convolutions

Pad so that output size is the same as the input size.

So we want that 🧐:

n+2p-f+1 = n

Hence:

p = (f-1)/2

For convention f is chosen to be odd 👩‍🚀

👀 Visualization

🔢 Strided Convolution

Another approach of convolutions, we calculate the output by applying filter on regions by some value s.

👀 Visualization

🤗 To Generalize

For an n x n image and f x f filter, with p padding and stride s; the output image size can be calculated by the following formula

$\left \lfloor{\frac{n+2p-f}{s}+1}\right \rfloor \times \left \lfloor{\frac{n+2p-f}{s}+1}\right \rfloor$

🚀 Convolutions Over Volume

To apply convolution operation on an RGB image; for example on 10x10 px RGB image, technically the image's dimension is 10x10x3 so we can apply for example a 3x3x3 filter or fxfx3 🤳

Filters can be applied on a special color channel 🎨

👀 Visualization

🤸‍♀️ Multiple Filters

🎨 Types of Layer In A Convolutional Network

👩‍🏫 Usually when people report number of layers in an NN they just report the number of layers that have weights and params

Convention: CONV1 + POOL1 = LAYER1

🤔 Why Convolotions?

• Better performance since they decrease the parameters that will be tuned 💫

🧐 References

• Convolutional Neural Networks (CNN) Introduction (😍✨✨✨)

📚 Popular Terms

👩‍💻 My Codes

👷‍♀️ Network Visualization Tool

This section will be filled by codes and notes gradually

👩‍💻 Codes

1. 👶 Basic CNNs

2. 👀 CNN Visualization

3. 👨‍👩‍👧‍👧 Human vs Horse Classifier with CNN

4. 🐱 Dog vs Cat Classifier with CNN

5. 🎨 Multi-Class Classification

6. 🌐 Tensorflow.js based hand written digit recognizer

   1. Classifier.js

   2. MNISTData.js

   3. index.html

✋ RPS Dataset

• Rock Paper Scissors is an available dataset containing 2,892 images of diverse hands in Rock/Paper/Scissors poses.

• Rock Paper Scissors contains images from a variety of different hands, from different races, ages and genders, posed into Rock / Paper or Scissors and labelled as such.

🔎 All of this data is posed against a white background. Each image is 300×300 pixels in 24-bit color

🐛 CNN Debugging

We can get info about our CNN by

model.summary()

And the output will be like:

Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_18 (Conv2D)           (None, 26, 26, 64)        640       
_________________________________________________________________
max_pooling2d_18 (MaxPooling (None, 13, 13, 64)        0         
_________________________________________________________________
conv2d_19 (Conv2D)           (None, 11, 11, 64)        36928     
_________________________________________________________________
max_pooling2d_19 (MaxPooling (None, 5, 5, 64)          0         
_________________________________________________________________
flatten_9 (Flatten)          (None, 1600)              0         
_________________________________________________________________
dense_14 (Dense)             (None, 128)               204928    
_________________________________________________________________
dense_15 (Dense)             (None, 10)                1290      
=================================================================

👩‍💻 For code in the notebook:

Here 🐾

• 🔎 The original dimensions of the images were 28x28 px

• 1️⃣ 1st layer: The filter can not be applied on the pixels on the edges

  • The output of first layer has 26x26 px

• 2️⃣ 2nd layer: After applying 2x2 max pooling the dimensions will be divided by 2

  • The output of this layer has 13x13 px

• 3️⃣ 3rd layer: The filter can not be applied on the pixels on the edges

  • The output of this layer has 11x11 px

• 4️⃣ 4th layer: After applying 2x2 max pooling the dimensions will be divided by 2

  • The output of this layer has 5x5 px

• 5️⃣ 5th layer: The output of the previous layer will be flattened

  • This layer has 5x5x64=1600 units

• 6️⃣ 6th layer: We set it to contain 128 units

• 7️⃣ 7th layer: Since we have 10 categories it consists of 10 units

😵 😵

👀 Visualization

The visualization of the output of each layer is available here 🔎

👷‍♀️ Network Visualization Tool

Netron ✨✨

🧐 References

• Binary Cross-Entropy

• RMSProp Explained

• RMSProp in Tensorflow

• Binary Classification

• TensorFlow: an ML platform for solving impactful and challenging problems

• Rock Paper Scissors Dataset

👩‍🎓 Orthogonalisation

One of the challenges with building machine learning systems is that there are so many things we could try. Including, for example, so many hyperparameters we could tune. The art of knowing what parameter to tune to get what effect, is called orthogonalisation.

What should we pay attention to while evaluating an ML project? How to optimize it? How to speed up? Since there are a lot of parameters how to know where to fix and which parameter to tune? 🤔🤕

Before answering these questions let's take a look at the whole process 🧐

⛓ Chain of assumptions in ML

The model should:

Fit training set well on cost function (Human level performance ❌❌)

⬇

Fit dev set well on cost function

⬇

Fit test set well on cost function

⬇

Perform well in real world ✨

Figuring out what is exactly wrong can help us to choose a suitable solution and then to fix that part without affecting the whole project 👩‍🔧

In short: Learning from one task and applying knowledge to separate tasks 🛰🚙

❓ What is Transfer Learning?

• 🕵️‍♀️ Transfer learning is a machine learning technique where a model trained on one task is re-purposed on a second related task.

• 🌟 In addition, it is an optimization method that allows rapid progress or improved performance when modeling the second task.

• 🤸‍♀️ Transfer learning only works in deep learning if the model features learned from the first task are general.

Long story short: Rather than training a neural network from scratch we can instead download an open-source model that someone else has already trained on a huge dataset maybe for weeks and use these parameters as a starting point to train our model just a little bit more with the smaller dataset that we have ✨

💫 Traditional ML vs Transfer Learning

🙄 Problem

Layers in a neural network can sometimes end up having similar weights and possible impact each other leading to over-fitting. With a big complex model it's a risk. So if you can imagine the dense layers can look a little bit like this.

We can drop out some neurons that has similar weights with neighbors, so that overfitting is being removed.

🔃 Comparison

🤸‍♀️ An NN before and after dropout

✨ Accuracy before and after dropout

🤔 When is it practical?

It is practical when we have a lot of data for problem that we are transferring from and usually relatively less data for the problem we are transferring to 🕵️‍

More accurately:

For task A to task B, it is sensible to do transfer learning from A to B when:

• 🚩 Task A and task B have the same output x

• ⭐ We have a lot more data for task A than task B

• 🔎 Low level features from task A could be helpful for learning task B

🧐 References

• More about transfer learning in Tensorflow

• Understanding Dropout

🧒👧 Face Verification

🙌 Comparison

🤸‍♀️ Solving Approach

🤳 One Shot Learning

Learning from one example (that we have in the database) to recognize the person again

🖇 The Process

• Get input image

• Check if it belongs to the faces you have in the DB

👓 How to Check?

We have to calculate the similarity between the input image and the image in the database, so:

• ⭕ Use some function that

  • similarity(img_in, img_db) = some_val

• 👷‍♀️ Specifiy a threshold value

• 🕵️‍♀️ Check the threshold and specify the output

🤔 What can the similarity function be?

🔷 Siamese Network

A CNN which is used in face verification context, it recievs two images as input, after applying convolutions it calculates a feature vector from each image and, calculates the difference between them and then gives outputs decision.

In other words: it encodes the given images

👀 Visualization

Architecture:

👩‍🏫 How to Train?

We can train the network by taking an anchor (basic) image A and comparing it with both a positive sample P and a negative sample N. So that:

• 🚧 The dissimilarity between the anchor image and positive image must low

• 🚧 The dissimilarity between the anchor image and the negative image must be high

So:

$L=max(d(a,p)-d(a,n)+margin, 0)$

Another variable called margin, which is a hyperparameter is added to the loss equation. Margin defines how far away the dissimilarities should be, i.e if margin = 0.2 and d(a,p) = 0.5 then d(a,n) should at least be equal to 0.7. Margin helps us distinguish the two images better 🤸‍♀️

Therefore, by using this loss function we:

• 👩‍🏫 Calculate the gradients and with the help of the gradients

• 👩‍🔧 We update the weights and biases of the Siamese network.

For training the network, we:

• 👩‍🏫 Take an anchor image and randomly sample positive and negative images and compute its loss function

• 🤹‍♂️ Update its gradients

🌠 Neural Style Transfer

Generating an image G by giving a content image C and a style image S

👀 Visualization

So to generate G, our NN has to learn features from S and apply suitable filters on C

👩‍🎓 Methodology

Usually we optimize the parameters -weights and biases- of the NN to get the wanted performance, here in Neural Style Transfer we start from a blank image composed of random pixel values, and we optimize a cost function by changing the pixel values of the image 🧐

In other words, we:

• ⭕ Start with a blank image consists of random pixels

• 👩‍🏫 Define some cost function J

• 👩‍🔧 Iteratively modify each pixel so as to minimize our cost function

Long story short: While training NNs we update our weights and biases, but in style transfer, we keep the weights and biases constant, and instead update our image itself 🙌

⌚ Cost Function

We can define J as

$J(G)=\alpha J_{Content}(C,G)+\beta J_{Style}(S,G)$