Neural Networks

Linear transformations alone can't learn nonlinear decision boundaries. Apply nonlinear functions element-wise to enable universal approximation.

How to efficiently compute gradients in multi-layer networks? Apply the chain rule layer-by-layer from output to input.