fabian s. klinkeFoundation: Are We Predictable?
Source: Foundation: Are We Predictable?, Oliver Lugg, 1:02:51, uploaded 2021-12-08, category Other / Unclear, playlist index 1165.
Oliver Lugg begins with Jurassic Park, where Ian Malcolm explains chaos theory and the butterfly effect with great confidence. Small changes can lead to large effects, yet Malcolm seems certain that the dinosaur park will fail. The joke opens a serious problem. Mathematics predicts the flight of a plane and the movement of planets with great precision, whilst other systems become impossible to forecast after a small change in their starting conditions. Isaac Asimov’s Foundation turns that problem into a fictional discipline. Psychohistory takes the current state of a galactic society, puts it into a mathematical model, and predicts the future. Lugg spends the rest of the video asking how far that idea survives contact with mathematics, history, machine learning, and fiction.
Asimov’s psychohistory
Isaac Asimov was born in the Soviet Union, although even he and his parents lacked an exact record of his birth date. He chose the latest plausible date himself, which later placed him just inside the age range for military service during the Second World War. His parents moved to the United States when he was three and opened a sweet shop that sold pulp science-fiction magazines. Those magazines gave the young Asimov a route into the field. At eighteen he began sending stories to John W. Campbell’s Astounding Science Fiction. Campbell rejected his first attempts whilst encouraging him to keep writing, and Asimov’s 1941 story Nightfall soon made him a leading science-fiction writer.
Lugg uses I, Robot to introduce Asimov’s habit of testing one idea from several directions. The stories examine the Three Laws of Robotics through situations that expose gaps in an apparently complete system. The Last Question returns to one problem across separate episodes in humanity’s future. Foundation applies the same interest in large ideas to history.
In 1941, whilst working at a Philadelphia dockyard, Asimov was reading Edward Gibbon’s six-volume History of the Decline and Fall of the Roman Empire. Gibbon described the factors that produced Rome’s collapse, and Asimov wondered whether someone could recognise those factors before the collapse took place. He pushed the idea through the scientific method until it became psychohistory, “that branch of mathematics which deals with the reactions of human conglomerates to fixed social and economic stimuli,” as the video quotes it.
Psychohistory does not predict an individual. It studies a large population, where the movement of institutions and incentives becomes more regular than the actions of each person. Asimov compares this to a gas. Its molecules collide in ways that are impossible to follow one by one, whilst the gas as a whole follows a pattern when external pressure changes. In the novels, mathematician Hari Seldon uses psychohistory to predict the fall of the Galactic Empire and the thirty-thousand-year dark age that will follow. He creates the Foundation at the edge of the galaxy, apparently to compile the Encyclopedia Galactica, whilst secretly placing it where it can alter the future at a sequence of Seldon Crises.
The first Foundation story appeared in 1942. Asimov later combined it with four other stories into Foundation in 1951, followed by Foundation and Empire in 1952 and Second Foundation in 1953. He returned to the series in the 1980s and 1990s with two sequels and two prequels. The format creates a problem for fiction. Characters appear briefly and then disappear across long stretches of time, whilst the social process receives more attention than a character’s inner life. One example makes the method clear. The Foundation has the military force to lose a war against the remaining Empire, yet it survives because any general who conquers it would become a threat to the Emperor. The Emperor therefore prevents the victory. The Foundation wins by waiting inside a political stalemate.
The series also tests its own premise. In Foundation and Empire, the Mule rises from the ruins of the Empire and conquers the Foundation. He carries a genetic mutation that lets him control other people’s emotions. The Mule began as Magnifico, an apparently harmless clown, and the protagonists gain their chance to resist him because one person shows him genuine affection. The Mule introduces a change in one person’s biology that Seldon’s model could not predict and that alters the course of the galaxy. He is Asimov’s version of the butterfly effect.
Mathematical models and their assumptions
Lugg then moves from the novels to mathematical modelling, which he defines as writing and solving equations that describe a real system. Mathematics predicts physical systems well because observation and theory can meet in a controlled model. Einstein’s general relativity predicted the apparent position of stars during a solar eclipse. Astronomers can forecast planetary motion far into the future. A projectile calculation can predict where a cannonball should land from its speed, angle, and the forces acting on it, although Lugg’s demonstration adds a small virtual gust that sends the first prediction astray.
The failure does not show that the equations have become bad at mathematics. It shows that the model left out a force. Every model makes choices about what belongs in the system. A heat-transfer equation can treat a material as continuous even though matter has a smallest scale. It can assume a fixed temperature at the boundary, uniform density, and physical laws that remain stable during the calculation. Those assumptions work when the scale and situation make them reasonable. The model becomes misleading when the omitted detail affects the result.
Asimov took the ideal gas law as a model for psychohistory. A gas contains too many molecules for a useful prediction of every collision, yet the collisions average into a macroscopic flow. Heat moves from hot regions towards cold ones even though the energy exchange between individual particles looks random. A differential equation can describe the larger movement with far fewer variables. Lugg compares this movement from particles to patterns with the proposed movement from individuals to social institutions.
The COVID-19 pandemic gives the model a direct political setting. Government restrictions in the United Kingdom followed projections of hospital admissions and deaths, and Lugg describes periods when he followed those models closely enough for his mood to depend on each new statistic. Biological models need more assumptions than physical ones because organisms reproduce, adapt, and respond to changing conditions. That helps explain why pandemic models often disagreed. Lugg’s point belongs to the modelling process: an equation can solve the problem it receives whilst the modeller has supplied the wrong problem.
Biological mathematics still gives the psychohistory idea some support. Population models describe predator-prey relations, sustainable fishing, ecological change, and aspects of evolution. Lugg suggests that Asimov, writing today, might base psychohistory on mathematical biology rather than the physical sciences. He uses Michael Crichton’s Jurassic Park as a bridge between the two. The novel treats biological prediction more seriously than the film, including a normal distribution as part of a plot about whether a population can remain under control. Crichton’s mathematician Ian Malcolm then provides a direct challenge to Asimov’s confidence in social prediction.
Lemmings and the onset of chaos
The video turns to the legend of suicidal lemmings. Disney’s 1958 documentary White Wilderness made the image famous, although the filmmakers pushed the animals from a cliff and presented the scene as voluntary mass suicide. The film amplified older folklore. Lemming populations do undergo sharp changes, with years of abundance followed by near extinction, and occasional accidents during migration offered a poor explanation for the whole pattern.
The logistic growth map gives Lugg a better one. The population in year n is xₙ, and the next value is calculated from xₙ, the reproductive rate r, and a term that represents environmental pressure. Low values of r lead the population towards extinction. Slightly larger values produce a stable population. As r increases, the population begins to alternate between two values, then four, and then more. Beyond roughly 3.57, the sequence becomes chaotic. It never settles into a repeating pattern, even though the equation remains deterministic.
This is the working meaning of chaos theory in the video. Edward Lorenz found it in 1961 whilst running weather calculations. He entered a stored value rounded to three decimal places instead of the original six, expecting the same forecast. The results diverged so far that a small numerical difference turned sunshine into storms. Lorenz described the finding in a paper whose title asked whether a butterfly’s wingbeat in Brazil could set off a tornado in Texas. Chaotic systems follow fixed laws, yet their future depends so sharply on their initial conditions that prediction requires a precision no practical measurement can supply.
Lugg connects the logistic map to lemmings through their rapid reproduction. Some estimates place their adjusted growth rate beyond the chaotic threshold, although he stresses that the point depends on the chosen model and that other models describe the animals differently. If lemming populations follow a suitable chaotic model, their fluctuations have a mathematical explanation that leaves no room for the old suicide story.
Chaos limits precise forecasts without destroying every form of prediction. Weather becomes hard to predict day by day, whilst its broad seasonal behaviour remains legible. A die roll is deterministic in the physical sense, yet a large set of rolls follows a stable probability distribution. Even the Lorenz attractor has an ordered shape: individual air trajectories vary, while the system remains within two large loops. The specific path can disappear into chaos as the aggregate pattern survives.
That distinction gives psychohistory a possible route. A single mutation can change history, as the Mule demonstrates, but a large population may still show regular movement. The video returns to rock-paper-scissors, where the three choices seem equal when people choose at random. World champions exist because human players display patterns, read small habits, and exploit the reasoning of their opponents. Geographic profiling uses the locations of repeated crimes to estimate where an offender may live. Crowd models use simple movement rules to design safer buildings, and Lugg’s favourite example places an obstacle in a large hall so that people flow around it like traffic at a roundabout.
Those examples show some capacity to predict individual behaviour. The Second Foundation in Asimov’s novels takes the idea further by developing psychohistory at a finer scale, although it does so through telepathy. Its members can read and alter individual minds, correct deviations from Seldon’s plan, and then conceal their intervention. The device gives the fiction a self-correcting version of psychohistory whilst keeping the real-world problem open. Human lives remain exposed to too many events for an individual forecast to become reliable.
Cliodynamics and the history of large populations
The real-world field closest to psychohistory is cliodynamics, which began in 2003 as an attempt to turn history into a quantitative study. Its researchers collect historical data and use statistical and mathematical models to test theories about social change. Peter Turchin leads much of the work. His background lies in beetle population dynamics, and he moved into human history after deciding that the same mathematical questions could apply at a larger scale.
Historians have proposed hundreds of explanations for the fall of Rome. The video refers to a list of 210 competing reasons and compares the problem with accounts of the Bronze Age Collapse, which range from invasions to earthquakes to a complex system failing under its own weight. Cliodynamics puts such hypotheses into models and compares them against databases that cover thousands of years. Its ambition is to find recurring laws of history in the way biology and physics contain recurring laws of their own.
One pattern is the secular cycle, a rhythm of expansion and crisis that Turchin connects to elite overproduction. A society can place more people into elite positions than it has positions to offer. The frustrated surplus then supplies people who can organise unrest. Turchin’s models can identify the accumulation of pressure, although he says they cannot identify the spark that turns it into a crash. He has also used the models to predict periods of unrest in the United States, including a forecast that the situation would become serious around 2020.
The forecast creates a problem that the video calls the uncertainty principle of the social sciences. A weather forecast does not change the weather. A public economic forecast can change the behaviour that produces the result. People can buy or sell in response to a prediction, which may bring about the crash or boom the forecast described. Turchin argues that cliodynamics can survive this problem because a single person’s reaction has little effect on a long historical trend, although the source leaves that claim open.
The objections are substantial. Critics accuse Turchin of choosing data that supports his conclusions and of turning historical predictions into policy advice. A successful doomsday forecast receives attention after the fact, whilst the many failed dates disappear from public memory. Reducing history to numbers can also erase the context that gives an event its meaning. The video compares this to Jared Diamond’s Guns, Germs, and Steel and to a study that claimed modern music was declining after measuring a few poorly chosen features without the surrounding musical context.
Turchin’s reply, as Lugg presents it, is that complexity makes mathematics more useful. Before weather forecasting, people relied on empirical signs from animals and the sky. Mathematical models did not remove uncertainty, yet they made forecasts far better than the earlier methods. A complicated society may require mathematics precisely because the human mind cannot hold all of its variables at once. An Imperial College talk included in the video’s source list gives the problem a more modest formulation: the mathematical ingredients for psychohistory may exist, whilst the recipe for combining them remains unknown.
Lugg adds a late warning about a 2020 reassessment of a 1970s study that predicts civilisational collapse in the 2040s. He says he has not had time to examine the research closely. The older forecasts of mass starvation were partly defeated by agricultural technology, and future technology may alter the newer trajectories as well. The example belongs in the note as a caution about public claims that “MIT predicts the end of the world”, rather than as evidence that collapse has been established.
Machine learning and the machine that sees too much
Machine learning offers another route towards the social scale of psychohistory. The 2006 Netflix Prize asked people to improve film recommendations from more than two million user-film ratings. The entries that performed best used algorithms that adjusted their internal parameters against large amounts of training data. They could identify useful groupings without being told what a genre was or why two films belonged together. Netflix could therefore predict a person’s future taste from the tastes of other users, even when the system could not explain the connection in human terms.
Lugg moves from recommendation to the Global Database of Events, Language, and Tone, or GDELT. The project collects news, social-media posts, and images, then turns them into quantitative measures of events and public emotion. Its creator, Kalev Leetaru, presents it as a way to find patterns across the world’s information and forecast what those patterns may lead to. The video says Leetaru claims that GDELT predicted the Arab Spring after it had happened, a qualification that matters more than the promotional force of the example.
Other projects aim at similar forms of prediction. Project Cassandra uses novels to anticipate future wars. GUARD applies the Alan Turing Institute’s work to military prediction in Britain. The United States military’s Global Information Dominance Experiment, or GIDE, explores whether artificial intelligence can identify developments several days in advance. Lugg treats the known projects as evidence that society may already live under machine-assisted governance, even when the people being measured have never agreed to a psychohistorical experiment.
Asimov had already imagined this turn. In Foundation and Earth, the characters discover that an ancient robot called R. Daneel Olivaw has manipulated human history for thousands of years and placed psychohistory in Seldon’s mind. I, Robot also ends with supercomputers running the global economy after humans decide that the machines understand it better than they do. These stories make artificial intelligence the hidden author of social prediction.
The promise collapses when the training data carries a bias. A machine-learning system designed to reconstruct pixelated faces produced a white face from a low-resolution image of Barack Obama. The source connects the failure to a data set that contained too few diverse faces. Robert Julian-Borchak Williams was arrested after facial-recognition software confused him with another Black man. Other examples include a turtle made from gun images that a vision system classified as a gun and a military tank detector that recognised sunny weather because every tank photograph in its training set had been taken on a sunny day. Netflix’s system also struggled with Napoleon Dynamite because its polarised ratings formed a pattern that looked unlike any other film in the data.
More data can repair some gaps. The No Free Lunch theorem places a harder boundary around the project. No learning algorithm can predict accurately in every possible setting, because a situation can always be constructed that defeats the algorithm. The difficult case may look artificial, yet the real world can contain strange cases without warning. A useful predictor can exist for a defined task. An infallible predictor cannot be guaranteed by adding more data or more computing power.
Prediction inside fiction
Lugg’s final section changes the object from society to storytelling. Fiction appears open-ended, although readers learn its structures early. A novel such as Georges Perec’s La Disparition can exclude the letter e, whilst more ordinary stories follow patterns associated with the hero’s journey, Save the Cat, or Dan Harmon’s story circle. Jokes depend on prediction too. Lugg’s recurring “wrong lever” joke works because the audience learns that one action will receive the same absurd response every time. His video essays use predictable paragraph lengths, musical cues, and comic reversals for clarity and rhythm.
Pure randomness makes a difficult story. A narrative usually needs some form of payoff, and audiences tend to distrust coincidences that solve a problem at the convenient moment. Excessive obedience to the rules produces a different failure. Lugg compares repetitive corporate elevator music with Arnold Schoenberg, then places good storytelling between total repetition and total disorder. A scene needs enough expectation for a deviation to register.
Foundation makes that balance unusually visible. Its history spans centuries and gives little space to the private lives of its characters. The novels contain few conventional heroes and villains, and their social processes can feel more important than their emotional lives. That structure makes the books difficult to adapt for television, where the audience expects personal drama to carry the long arc. Lugg watches Apple’s series and finds that the adaptation adds larger characters and spectacular settings whilst losing some of the dry, historical structure that made Foundation strange.
The adaptation problem was already inside the books. John W. Campbell worried that the early Foundation stories would become repetitive, so he asked Asimov to disrupt Seldon’s plan. Asimov created the Mule, a character who makes the prediction fail and then turns that failure into the next story. Science fiction can use this method to examine questions that science has not yet answered. A theory predicts the future, and the story invents the event that exposes the theory’s blind spot.
Limits
Lugg ends with a qualified answer: society appears predictable in limited models, whilst Asimov’s full psychohistory remains too distant. Mathematical models require assumptions about continuity, representative data, stable laws, and the variables that matter. Chaos can turn a small error into a large one, and nobody yet knows where history sits on the spectrum between a stable aggregate and a chaotic system. Cliodynamics may find recurring patterns whilst still losing the context that explains each case. Machine learning can find relations that a person cannot see, yet the same opacity lets it follow the wrong feature.
The historical anecdotes, figures, named studies, technical explanations, and machine-learning cases in this note follow Oliver Lugg’s video, captions, description, and supplied source list. I have not independently checked each underlying study or historical claim. The video itself marks some limits, especially the 2040s collapse forecast and the claims made for GDELT. Those qualifications belong to the evidence rather than to the conclusion.
Lugg’s conclusion reaches beyond the mathematical question. Humans use predictable forms to make sense of life and fiction, yet the value of both depends on events that escape the form. He prefers to leave the future open because a fully predicted life would leave no room for surprise. The video began as a defence of Foundation and ends as a defence of uncertainty, with mathematics still useful when it stays close to the human context it measures.