Linear algebra

A vector is an ordered list of numbers. Think of it as a column of values stacked vertically. We write it with bold lowercase letters like $\mathbf{x}$ so it's easy to spot:

Two operations are the bedrock of everything else: add two vectors, or multiply a vector by a regular (non-bold) number.

Drag the sliders below. The dashed green vector is $\mathbf{a}+\mathbf{b}$. Watch it change as you move $\mathbf{a}$ and $\mathbf{b}$.

The dot product takes two vectors and produces a single number. Multiply matched components, then sum:

There's a beautiful geometric fact about dot products you'll use forever:

A matrix is a rectangle of numbers. Capital letters like $A$ for matrices, two indices for each entry — first the row, then the column:

You can read a matrix two ways, both useful:

• As stacked rows. Each row is a vector. In a neural network, each row is one neuron's weight vector.
• As stacked columns. Each column is a vector. In that view, each column is one input feature's contribution to every neuron.

To multiply two matrices $C=AB$, the inner dimensions must match: if $A$ is $m\times p$ and $B$ is $p\times n$, then $C$ is $m\times n$. Each entry of $C$ is one dot product:

Step through the same kind of multiplication interactively:

The transpose of a matrix flips it diagonally — rows become columns and vice versa. We write the transpose with a little T in the upper-right:

Now the payoff. The forward pass of a single neural-network layer is just: