Gradient descent vs evolution

Source: Gradient Descent vs Evolution | How Neural Networks Learn, Emergent Garden, 23:46, uploaded 2025-03-01, category Science, playlist index 307.

Emergent Garden opens with a neural network already changing on screen. The video uses that small demonstration to explain what “learning” means in this setting: an optimisation process that searches for parameter values which make a network approximate a target function. It compares stochastic gradient descent with a simple evolutionary method called local search, then follows both methods down the same imagined Loss Mountain.

A network as a point in parameter space

The starting point is the idea that a neural network is a function. It takes numbers in and gives numbers out, so it can approximate another function from a sample of its data points. The shape of that function comes from its parameters, the weights and biases that can change during training. More parameters allow more complicated shapes. The video jokes that four parameters can fit an elephant and five can make it wiggle its trunk.

The web toy reduces this to a network with two parameters, A and B. A green curve is the network’s approximation of a blue sine wave, which supplies the target data. The network uses a tanh activation, and changing A or B moves a green point through a two-dimensional plane. Each point in that plane represents one network with this particular architecture. The plane is parameter space.

That picture gives optimisation a concrete shape. Some points produce a poor approximation and some produce a good one. A person can search the plane by hand, although training needs a procedure that can find useful parameters without checking every possibility. The problem becomes much harder once a network has hundreds or billions of parameters. The two-dimensional web toy keeps the geometry visible whilst the argument moves towards that larger problem.

Loss turns the search into a landscape

The network needs a score. The target function remains unknown, since the training data contains only a sample of its points. The video takes three points from the sine wave, asks the network to predict their outputs, and compares each prediction with the true value. Mean squared error turns those differences into a single loss. Lower loss means that the network fits the observed data more closely.

Loss becomes another function whose inputs are the network’s parameters and whose output is the loss for those parameters. If every point in the two-parameter plane receives a height based on its loss, parameter space becomes a loss landscape. High points represent poor networks. Low points represent good ones. Training means finding a low point, ideally the global minimum.

The landscape can be defined for a network with many parameters even when it cannot be drawn or calculated in full. A hundred parameters already create a hundred-dimensional space whose every combination cannot be generated in practice. The optimisation algorithm therefore needs to navigate the landscape with partial information. It needs a route down the mountain without a complete map.

The video gives the problem a person. Imagine a climber near the top of a rugged mountain range who cannot see. The climber can try a random step, keep it when the ground falls, and repeat. Taking several random steps and keeping the best one improves the guess. The other route is to feel the slope underfoot and step in the direction of the steepest descent. These are the two training methods that the rest of the video compares.

Local search and the blind climber

The evolutionary method begins with a random network. It copies that network, mutates some of the copy’s parameters by adding small random values, and creates a population of candidate steps. After calculating the loss for each candidate, it keeps the one with the lowest loss and repeats the process. The parameters act as the network’s genome. The loss acts as an inverse fitness score, since biological evolution tends to seek higher fitness whilst this procedure seeks lower error.

On the two-parameter web toy, local search quickly approaches the lowest point. The procedure can converge on the global minimum, although the video stresses that it is not guaranteed to do so. Each accepted step improves the approximation or leaves its loss unchanged. That strict rule makes the method easy to watch and also creates its main weakness: once it reaches a shallow valley, it cannot climb out to search elsewhere.

The next demonstration uses a real PyTorch network with 1,741 parameters. The loss landscape has too many dimensions to show, so the video displays the function that the network learns. Local search mutates a random subset of parameters, and the mutation rate falls over time so that the network can adjust finer details. The run uses 150 rounds with a population of 1,000, which means that the footage required 150,000 candidate networks.

The same method then approximates images. The network takes a pixel’s coordinates as two inputs and returns one pixel value. The target image supplies one data point for every pixel. A smiley face eventually takes shape because every accepted mutation improves the loss. A portrait of Charles Darwin exposes the limits of the method. The creator tunes the population size, number of rounds, and number of neurons by hand. These are hyperparameters, which are set before training and remain outside the learned parameter set. Even after tuning, the approximation gets stuck high on Loss Mountain.

The video treats that stall as a local minimum, a valley whose surrounding steps all make the result worse. A local search that can only descend cannot leave it. A more elaborate evolutionary method could preserve a diverse population, combine parameters through crossover, mutate the network architecture, or search its hyperparameters. Those additions might help, although the video leaves them for a future treatment. Its local search is a useful baseline because the failure remains visible.

Gradient descent and the slope

Gradient descent uses the information that local search lacks. The gradient is a vector whose entries give the slope of the loss with respect to each parameter. With two parameters, the two slopes define the direction of the steepest ascent. Negating that vector points downhill. The method can therefore choose a direction instead of guessing which parameter to mutate.

Calculating the gradient requires derivatives of the loss landscape with respect to every parameter. The video invokes the chain rule and points to backpropagation as the procedure that automates this calculation across a large network. A forward pass evaluates the current network and its loss. The backward pass moves through the network to calculate each parameter’s contribution to that loss. The optimiser then subtracts a scaled version of the gradient from the parameters.

The gradient supplies a direction, not a safe distance. A learning rate scales the step so that the network does not overshoot the useful part of the slope. The video describes the result as a smoother and more precise hill climber. It can still become jumpy on steep slopes, and individual steps can raise the loss, so the method does not guarantee steady improvement at every moment.

The stochastic part comes from random initialisation and from random batches. Instead of calculating one step from the whole data set, the network trains on a small subset, then continues through the other batches before shuffling them for the next pass. Each batch defines a slightly different loss landscape. Over many batches those variations average towards the landscape defined by the whole data set. In practice, the batch size and other hyperparameters still need tuning. Batches also spread the expensive backpropagation work across more computation.

The video runs the same image approximations with PyTorch’s SGD optimiser. The curves become smoother, and the network refines small details to reach a slightly lower loss. The comparison is visual rather than scientific because the creator is looking at the output rather than reporting a controlled measurement. Even so, the SGD run takes less compute time and reaches a lower loss in the simpler demonstrations. Darwin remains difficult. Gradient descent can get stuck in local minima too.

Momentum helps the final step roll through small valleys. The video then moves to Adam, a variant of SGD that uses the first and second moments of the gradient to adjust its step size. Adam trains the examples faster and produces more precise functions in the footage. Every result still uses the same neural architecture. The algorithms discover different values for its parameters, which gives the network a different function.

Scale and the gradient advantage

Adding parameters creates more dimensions in the loss landscape. A point that looks like a minimum in one dimension can be a saddle point when another dimension is available. The gradient can usually find a downhill direction out of that saddle. Evolutionary search can benefit from extra dimensions too, although it has to guess which parameters to mutate and in which direction.

That guess becomes less likely as the number of parameters grows. The video describes parameter space as growing exponentially, so the evolutionary method must generate many failed networks before it happens to mutate in a useful direction. Success remains possible. Given enough time, local improvements can accumulate. The source’s estimate for that time is deliberately absurd: billions of years.

Gradient descent gets its advantage from scale. The gradient can be calculated in linear time with respect to the number of parameters, and one step can use all of those slopes at once. That makes the method compatible with very large networks. The video calls this the blessing of dimensionality and connects it to the use of billions of parameters in large language models.

The comparison has a clear limit. The video places a heavily optimised gradient-descent implementation beside a homegrown local search algorithm. It says that state-of-the-art evolutionary methods may have advantages that this demonstration cannot show. The argument establishes why gradient descent fits large differentiable networks. It does not establish that every evolutionary algorithm is slower or less capable.

Where evolution still has a case

Gradient descent depends on continuity and differentiability. The loss function and the network need a surface whose slopes can carry information through the model. A jagged function with sharp breaks can block that route. The video gives a binary step function as an example and says that backpropagation cannot pass through it.

Evolution can mutate such structures because it only needs to evaluate the result. The creator evolves jagged and fractal-looking networks that produce strange images. They can be interesting and they can represent functions that gradient descent cannot train in the same way. The examples do not produce a better approximation of the target. They show that the evolutionary method has access to a different class of network shapes.

The final distinction reaches beyond these algorithms. If either programme ran for a billion years, the video says, it would keep improving its approximation. Biological evolution can move into new forms. It produces eyes, wings, brains, and pill bugs rather than converging on a single numerical minimum. That divergence gives biological evolution a property that the local search in the video lacks, although the source leaves the subject outside its scope.

For the problem shown here, stochastic gradient descent remains the most effective tool because it uses the slope and keeps working as the network grows. The closing joke turns back to the video itself: language models trained with gradient descent helped write its code and check its script, whilst the script and music remained human-made. The algorithm has trained a system that can help describe the process that trained it.

Limits

The figures for the 1,741 parameters, 150 rounds, population size, 150,000 generated networks, compute time, and relative loss come from the video’s demonstrations. The creator says that the visual comparison is an informal judgement rather than a scientific benchmark. The source also compares a simple local search implementation with mature gradient-based tools, so its result is an explanation of the gradient’s scaling advantage rather than a survey of evolutionary optimisation.

The captions and description name backpropagation, universal function approximation, and loss surfaces, and the description links papers for those subjects. I have kept those claims attached to Emergent Garden’s account here. The note does not treat the source’s statements about state-of-the-art methods, biological evolution, or large language models as independently verified research findings.

Further reading / references

• Hill Climbers, the interactive web toy used in the video.
• Neural networks as universal function approximators, linked under the video’s citations.
• Backpropagation, linked under the video’s citations.
• Loss surfaces of multilayer perceptrons, linked under the video’s citations.
• Webtoy source, animation source, and image approximators, linked in the description.
• Dawkins: Climbing Mount Improbable, linked in the description.