Course Website

Types of Neural Network

Figure adapted from slides by Andrew NG, Deep Learning Specialization

A Little Bit of History – Perceptron

• First concept of a simplified brain cell (1943): McCullock-Pitts (MCP) neuron
• Frank Rosenblatt published the first concept of the perceptron learning rule based on MCP (1957)
• Classification of N points into 2 classes: -1 and +1 (i.e., two different colors in the picture below)

• Fun video: https://www.youtube.com/watch?v=cNxadbrN_aI

• Two features: (\(x_1\) and \(x_2\)); linear functions to separate classes; find (\(w_0\), \(w_1\), \(w_2\)) such that:

\[z^{(i)} = w_0 + w_1x_1^{(i)} + w_2x_2^{(i)}\]

\[pred^{(i)}=\begin{cases} \begin{array}{c} 1\\ -1 \end{array} & \begin{array}{c} if\ z^{(i)}>0\\ if\ z^{(i)}\le0 \end{array}\end{cases}\]

Perceptron Algorithm

Start with random weights. Set a maximum of epochs of M, for each epoch (permutation):

• For each data point \(i\):
  • Predict class label

  \[z^{(i)} = w_0 + w_1x_1^{(i)} + w_2x_2^{(i)}\]

  \[pred^{(i)}=\begin{cases} \begin{array}{c} 1\\ -1 \end{array} & \begin{array}{c} if\ z^{(i)}>0\\ if\ z^{(i)}\le0 \end{array}\end{cases}\]

  • For every data point, we update the weight based on the prediction correctness, learning rate and feature values.

\[w_0 = w_0 + \eta(actual^{(i)} - pred^{(i)})\]

\[w_1 = w_1 + \eta(actual^{(i)} - pred^{(i)})x_1^{(i)}\] \[w_2 = w_2 + \eta(actual^{(i)} - pred^{(i)})x_2^{(i)}\]

• Calculate accuracy for the entire dataset to see whether it meets the criteria after each epoch.

Perceptron Algorithm

• Perceptron algorithm is easy to implement in any modern programming language.
  • Perceptron R notebook

• It is a linear classification function, and the weight is updated after we feed each data point to the algorithm (a concept similar to stochastic gradient descent).

• The algorithm continues to update when we feed the same data set again and again (i.e., epochs)

• It does not solve non-linearly spreadable problems.

Logistic Regression as A Neural Network

• m training samples: \(\{(x^{(1)}, y^{(1)}),(x^{(2)}, y^{(2)}),...,(x^{(m)}, y^{(m)})\}\)

\[X=\left[\begin{array}{cccc} x_{1}^{(1)} & x_{1}^{(2)} & \dotsb & x_{1}^{(m)}\\ x_{2}^{(1)} & x_{2}^{(2)} & \dotsb & x_{2}^{(m)}\\ \vdots & \vdots & \vdots & \vdots\\ x_{n_{x}}^{(1)} & x_{n_{x}}^{(2)} & \dots & x_{n_{x}}^{(m)} \end{array}\right]\in\mathbb{R}^{n_{x}\times m}\]

\[y=[y^{(1)},y^{(2)},\dots,y^{(m)}] \in \mathbb{R}^{1 \times m}\]

\(\hat{y}^{(i)} = \sigma(w^Tx^{(i)} + b)\) where \(\sigma(z) = \frac{1}{1+e^{-z}}\)

Logistic Regression as A Neural Network

• m training samples: \(\{(x^{(1)}, y^{(1)}),(x^{(2)}, y^{(2)}),...,(x^{(m)}, y^{(m)})\}\)

\[X=\left[\begin{array}{cccc} x_{1}^{(1)} & x_{1}^{(2)} & \dotsb & x_{1}^{(m)}\\ x_{2}^{(1)} & x_{2}^{(2)} & \dotsb & x_{2}^{(m)}\\ \vdots & \vdots & \vdots & \vdots\\ x_{n_{x}}^{(1)} & x_{n_{x}}^{(2)} & \dots & x_{n_{x}}^{(m)} \end{array}\right]\in\mathbb{R}^{n_{x}\times m}\]

\[y=[y^{(1)},y^{(2)},\dots,y^{(m)}] \in \mathbb{R}^{1 \times m}\]

\(\hat{y}^{(i)} = \sigma(w^Tx^{(i)} + b)\) where \(\sigma(z) = \frac{1}{1+e^{-z}}\)

• Loss function: \(L(\hat{y},y) = -ylog(\hat{y})-(1-y)log(1-\hat{y})\)
• Cost function: \(J(w,b)=\frac{1}{m} \Sigma_{i=1}^m L(\hat{y}^{(i)}, y^{(i)}) = \frac{1}{m} \Sigma_{i=1}^{m} \{\ -y^{(i)}log(\hat{y}^{(i)})-(1-y^{(i)})log(1-\hat{y}^{(i)}) \}\)
• Goal: Find \(w,b\) that mininizes \(J(w,b)\)
  • \(w := w - \alpha \frac{\partial J}{\partial w}\)
  • \(b := b - \alpha \frac{\partial J}{\partial b}\)

Forward Propagation