πCommon Concepts

Basic Concepts of ANN

π­ Basic Neural Network

Convention: The NN in the image called to be a 2-layers NN since input layer is not being counted π’β

π Common Terms

 Term Description π Input Layer A layer that contains the inputs to the NN π Hidden Layer The layer(s) where computational operations are being done π Output Layer The final layer of the NN and it is responsible for generating the predicted value yΜ π§  Neuron A placeholder for a mathematical function, it applies a function on inputs and provides an output π₯ Activation Function A function that converts an input signal of a node to an output signal by applying some transformation πΆ Shallow NN NN with few number of hidden layers (one or two) πͺ Deep NN NN with large number of hidden layers $n^{[l]}$ Number of units in l layer

π§  What does an artificial neuron do?

It calculates a weighted sum of its input, adds a bias and then decides whether it should be fired or not due to an activaiton function

My detailed notes on activation functions are here π©βπ«

 Parameter Dimension $w^{[l]}$ $(n^{[l]},n^{[l-1]})$ $b^{[l]}$ $(n^{[l]},1)$ $dw^{[l]}$ $(n^{[l]},n^{[l-1]})$ $db^{[l]}$ $(n^{[l]},1)$

Making sure that these dimensions are true help us to write better and bug-free :bug: codes

π Summary of Forward Propagation Process

 Input: $a^{[l-1]}$ Output: $a^{[l]}, chache (z^{[l]})$

$Z^{[l]} =W^{[l]}A^{[l-1]}+b^{[l]}$ $A^{[l]} = g^{[l]}(Z^{[l]})$

π Summary of Back Propagation Process

 Input: $da^{[l]}$ Output: $da^{[l-1]}, dW^{[l]}, db^{[l]}$

$dZ^{[l]}=dA^{[l]} * {g^{[l]}}'(Z^{[l]})$

$dW^{[l]}=\frac{1}{m}dZ^{[l]}A^{[l-1]T}$

$db^{[l]}=\frac{1}{m}np.sum(dZ^{[l]}, axis=1, keepdims=True)$

$dA^{[l-1]}=W^{[l]T}dZ^{[l]}$

π΅π€

β¨ Parameters vs Hyperparameters

• $W^{[1]}, W^{[2]}, W^{[3]}$

• $b^{[1]}, b^{[2]}$

• ......

• Learning rate

• Number of iterations

• Number of hidden layers

• Number of hidden units

• Choice of activation function

• ......

We can say that hyperparameters control parameters π€

Last updated