Neural networks, one neuron at a time

One neuron does three things in order:

1. Weighted sum. Take the input vector $\mathbf{x}$, dot it with a weight vector $\mathbf{w}$, add a bias $b$.
2. Activation. Squash the result through a non-linear function $\sigma$ (sigmoid, ReLU, tanh, etc.).
3. Output. The squashed number is what the neuron passes on.

Tune $w_1,w_2,b$ below and see how the decision region shifts:

Stack $m$ neurons side by side. Each has its own weight vector. Pack their weight vectors as the rows of a matrix $W$, their biases as a vector $\mathbf{b}$. The whole layer is one matrix-vector multiply followed by an element-wise activation:

For a whole batch of $N$ inputs (one per row of $X$), we usually transpose the convention so we can multiply once and process everyone in parallel:

Without a non-linear activation, stacking layers does literally nothing useful. Two linear maps composed are still a single linear map. Watch:

Compare any pair of activations side-by-side (solid line = function, dashed = derivative):

Chain $L$ layers together. Each layer takes the previous layer's output as its input. We use a superscript in parentheses to label the layer index:

To train a network we need a single number that says "this prediction was bad." That number is the loss (or "cost"). We pick a loss function based on what kind of problem we're solving: