Multilayer Networks & Forward Propagation

Source: Unit 2 §7

Why more than one layer

A single neuron cannot do complex tasks - XOR settled that. The brain's answer is to stack billions of neurons in layers, and artificial neurons do the same: information passes layer to layer, and the neurons are now called nodes or units.

The three layer types

The input layer only holds values; every bit of learning happens in the hidden layer, which is why the layer count ignores the inputs.

Layer	Role
Input layer	Where the inputs are fed in. The number of units equals the number of inputs, and there is no computation here.
Hidden layer(s)	Processes the input and derives the complex relationships and patterns. This is the main core of learning, and there can be any number of hidden layers.
Output layer	Receives the hidden layer's results and emits the output.

⚠GotchaCounting layers

Number of layers in an ANN = number of hidden layers + the output layer. The input layer is not counted, because it performs no computation. A network described as "2-layer" has one hidden layer and one output layer.

Activation functions, in one paragraph

An activation function (also called a transfer function) introduces non-linearity into the network. It is applied as $f(z)$ where

$$z = (\text{inputs} \times \text{weights}) + \text{bias}$$

⚠GotchaWithout an activation function there is no network

A neuron with no activation is just linear regression, and stacking linear layers produces another linear function. No amount of depth buys you a complex pattern until something non-linear sits between the layers. The sigmoid is the standard example; the full catalogue is on the Activation Functions page.

Forward propagation

Take a 2-layer network: 2 inputs $x_1, x_2$ → 4 hidden units → 1 output.

Initialise the weights randomly. In the real world we do not know which input matters more than another, so the weights and biases start at random values and training discovers the answer.

Input to hidden:

$$Z_1 = X \cdot W_{xh} + b_h \qquad a_1 = \sigma(Z_1)$$

Hidden to output:

$$Z_2 = a_1 \cdot W_{hy} + b_y \qquad \hat{y} = \sigma(Z_2)$$

where $W_{xh}, b_h$ are the weights and bias between the input and hidden layers, and $W_{hy}, b_y$ those between the hidden and output layers.

Forward propagation alternates two moves: a weighted sum with a bias, then a squash. Repeat once per layer and the last squash is the prediction.

#NumbersDimension alert

The weight-matrix dimension = (number of units in the current layer) × (number of units in the next layer). For the 2-4-1 network above, $W_{xh}$ is $2 \times 4$ and $W_{hy}$ is $4 \times 1$.

◎Exam cueWhat forward propagation does not tell you

Forward propagation produces an output - but nothing about whether that output is correct. That gap is filled by defining a cost function and then backpropagating the error, which is the next page.