AI needs to feel pain

AI Needs to Feel Pain

Source: AI Needs to Feel Pain, Art of the Problem, 27:54, uploaded 2025-11-26, Watch Later position 278.

Art of the Problem opens with a sharp description of modern AI: a system that can produce language without caring about what the language causes. GPT-3 continued whatever pattern came before it, whether the pattern came from Shakespeare or from nonsense. Synthetic pain gives such a system a direction. Punishment makes mistakes costly, reward makes successful actions more likely, and the resulting value signal acts like a primitive emotion. The video uses this idea to tell the history of reinforcement learning, from a box of beads to robots that imagine the consequences of their movements.

Feeling as a value signal

The opening argument depends on a person whose emotional processing had been damaged. The man remained articulate and could solve small puzzles, yet ordinary decisions became painfully slow. Choosing a pair of socks could take hours. Art of the Problem presents this as evidence that intelligence alone cannot select a course of action. A value signal gives a situation some felt direction, and learning depends on that direction.

The machine version begins with a reward and punishment attached to action. A system receives a state of the world through sensors, chooses an action, and receives an occasional signal about the result. The state might be a game board, the angle of a pole, or the pixels on a screen. The policy maps a state to an action. Reinforcement learning searches for a policy that produces the largest future reward.

Michie’s matchboxes

In 1961, computer scientist Donald Michie built a tic-tac-toe player from roughly 300 matchboxes because he had no computer available to demonstrate the method. Each box represented a possible board position. Coloured beads inside the box represented the moves available from that position.

At the beginning, every move had roughly the same chance. The machine chose a box for the current board, drew a bead at random, and played the corresponding move. After the game, the result changed the contents of every box used during the match. A loss removed a bead of the chosen colour, which made that move less likely in the same situation. A win added a bead of that colour, which made the successful choice more likely next time. Michie’s player eventually learned to play tic-tac-toe perfectly from games rather than from a human-written strategy.

Michie then applied the same principle to the cart-and-pole problem. The boxes had to represent a physical state, so each one encoded the cart’s position and velocity after continuous measurements had been divided into ranges. This discretisation produced 162 possible combinations. Two bead colours represented moving left or right. When the pole fell past the horizontal, the system applied a reinforcement signal to the boxes along the failed path. Repeated play gradually favoured actions that kept the pole balanced.

The boxes made the basic division visible. An agent needs a way to perceive its environment, together with a way to choose what to do in the state it perceives. The first requirement is embodiment. The second is a policy. The method becomes expensive when every possible state needs its own box and every box needs many visits before its policy becomes reliable. Chess and walking contain so many possible states that a literal table would need more boxes than the universe can hold.

From hand-written value to learned value

Claude Shannon’s 1950 work on chess supplied the next step. Shannon treated exhaustive search as impossible because the number of future move sequences grows too quickly. He proposed an evaluation function that would score a board position between a likely loss and a likely win. The video calls this a value function: a way to estimate the quality of a state before the final outcome arrives.

Shannon’s function used features that a human could describe. The number of pieces mattered. A queen had greater value than a pawn. Mobility improved a position, whilst an exposed king weakened it. An equation assigned a weight to each feature and combined those weights into a score for the current board. The chess player then chose the move whose resulting position had the highest value, a greedy policy that selected the best-looking next state.

Art of the Problem connects this machine value to human feeling. A chess master can sense that a position is favourable without calculating every line that follows. The video suggests that evolution has tied human value functions to emotion, giving situations a felt quality that guides choice. Shannon’s system had a clear weakness: a person had supplied the features and their meaning, so the program could not learn from its own mistakes.

Arthur Samuel took up Shannon’s proposal with a checkers program. He kept human-designed features such as piece count, king count, control of the centre, and mobility, then let the program learn how much each feature mattered. The program played against itself with random weights. Each game changed the weights, which changed the policy, which produced new games for the next update. Samuel wrote that the program learned to play better checkers than its author after eight hours of machine play.

Samuel still had to tell the system which patterns to measure. The next problem was feature discovery. A name such as “mobility” describes a pattern in the arrangement of pieces, and a system that receives raw perceptions needs to discover such patterns for itself.

Tesauro and temporal difference

The breakthrough took more than 30 years. In the late 1980s, Yann LeCun’s neural networks learned patterns in handwritten digits, including curves and line counts. Gerald Tesauro saw that a network able to discover useful patterns in pixels might also discover useful patterns in a game position. He chose backgammon, which is harder than tic-tac-toe and checkers in the account given here.

Tesauro supplied a raw description of the board to a multi-layer neural network. A middle layer learned thousands of patterns related to winning. An output layer turned those patterns into a value estimate for the position. During training, the network adjusted the connection weights so that patterns which helped predict winning carried more influence.

He also changed the timing of the reward. Samuel’s system learned from wins and losses, which meant waiting until a game had finished before knowing whether the sequence of moves had succeeded. Tesauro used temporal-difference learning to create a signal after each move. If the current position looked like an 80 percent chance of winning and the next position looked like a 90 percent chance, the network adjusted its estimate towards the later value. Richard Sutton described the method as learning from guesses made one step apart in time.

The final win or loss still anchors the process. After a game reaches its end, the network improves its estimate of positions close to that result. It then uses those improved estimates to learn about positions one move earlier, working backwards from end games towards the opening. This bootstrapping lets a few hundred finished games teach the system about much earlier states. After roughly 300,000 self-play games, Tesauro’s backgammon network played better than a human player in the video’s account.

Tesauro found that the network’s strength lay in the middle game, where judgment matters more than exhaustive calculation. The machine had developed an intuition that resembled the judgement of a master. His result answered Samuel’s feature-discovery problem, although it did not make reinforcement learning easy. Researchers found that value learning could fail to start, slow down, or get worse in complex games and continuous robotic tasks.

Q functions and deep Q networks

The video locates one source of this instability in the kind of value being learned. A value function estimates the quality of a state. A system also needs to know the quality of a particular action taken in that state. In a 1989 PhD thesis, Christopher Watkins proposed learning this action value directly. The result was the Q function, which can make learning faster and more stable because it assigns a value to each available action.

Another change came from larger neural networks. AlexNet, introduced in 2012, showed how a large network could learn to classify complex images. DeepMind combined that kind of network with Q-learning and called the result a deep Q network. The goal was a single system that could play many Atari games from screen pixels and game rewards alone.

The network began with random weights, eight layers of neurons, and one output for each possible controller action. At first it pressed buttons at random. After each experience, it adjusted the Q function towards actions that led to more points or a win. Millions of game attempts gave it enough experience to improve. Art of the Problem says that after two hours, or about 300 games, it could play better than any human at one of the games.

The video uses Seaquest to show what the learned value means during play. The predicted value rises when an enemy appears, peaks as a torpedo approaches its target, and falls back once the event is over. The curve shows a system estimating the future reward of a sequence of actions rather than reacting only to the current image.

Q-learning still fits a small, discrete action space. A controller with a few buttons is easy to enumerate. A robot arm or a football-playing machine can choose from a continuous range of movements. RoboCup, which began in the mid-1990s, exposed this problem. Early systems divided movement into buckets such as moving forward, turning 45 degrees, or kicking. The result was brittle and awkward because the buckets discarded much of the available action space.

Policy gradients and the problem of reality

Policy gradients return to the idea behind Michie’s beads. A neural network receives a state and outputs a probability distribution across possible actions. A new policy begins with roughly equal probabilities. Rewards reshape the distribution so that actions associated with better outcomes become more likely. The system can therefore handle a continuous range of actions without assigning a separate value to every possible movement.

The trade-off is experience. The network learns a whole action distribution rather than the value of one action at a time, so direct policy learning needs many examples. That demand pushed early progress towards simulated robotics. A simulator can run years of experience in a few hours, while a physical robot moves at the pace and cost of the real world.

Simulation introduced its own failure. A robot trained in a clean virtual environment could fail when transferred to a physical machine because the real world contains details and variations the simulation omitted. Researchers tried stabilising the policy with rules such as limiting how sharply the action distribution could change. Proximal policy optimisation treats the policy as if an elastic restraint were attached to it, which keeps each update within a safer range.

The more important change came from making simulations deliberately imprecise. Domain randomisation varies the conditions of each simulated experience, including gravity, friction, object size, and lighting. The robot sees enough different versions of the world to learn behaviour that transfers across them. The simulation becomes a source of variation rather than a single world the robot has to imitate exactly.

OpenAI used this approach to train a robotic hand to rotate a cube. The reward measured the cube’s orientation. The system learned in simulation and then transferred its policy to a physical hand. The randomised training let it recover from disturbances and generalise to other shapes. The video presents these abilities as emergent physical intelligence because nobody wrote a separate rule for every recovery or manipulation.

Art of the Problem then moves to DeepMind’s 2024 work on robot soccer. Humanoid robots learned a direct policy in simulation, with scoring as the reward and domain randomisation across the environment. The trained network moved into a physical robot without further training. The researchers showed behaviours such as anticipating a shot and blocking it before the ball arrived. They represented the learned behaviour as pathways through action space and identified continued learning in the physical world as the next question: how much can the robot learn in simulation, and how much must it learn whilst living in the world?

Imagination and general physical intelligence

Every system in this history remains tied to a task. A checkers player, a chess player, an Atari player, and a soccer robot can learn impressive behaviour inside a reward structure that someone chose in advance. The video calls this narrow intelligence and asks how a robot might act across a much wider range of situations.

Researchers are now giving robots millions of hours of human movement as training data. Imitation alone remains brittle because the robot copies actions without understanding the consequences that made those actions useful. The proposed addition is imagination. A robot can simulate the next few seconds of movement before it acts, in a process the video compares with a language model considering continuations before producing a response.

A simulation becomes useful when the system values its outcome. Synthetic pain supplies that value. The robot can compare imagined consequences and learn to avoid a failure before the physical movement occurs. This brings the history back to Michie’s matchboxes. The early machine learned after a game ended, while the imagined robot learns from a forecast of what its action will cause.

The video ends by leaving the design of that mind open. It asks what an imagination looks like and where the line between actions and words lies. The final answer remains a research direction rather than a finished theory. Reinforcement learning gives machines a way to care about consequences in the narrow sense of assigning them different values. The step from that value signal to a general physical intelligence still depends on the quality of the imagined world, the reward it contains, and the experience a robot can gather outside simulation.

Limits

The video offers a compact history of reinforcement learning, with named researchers and examples that make the changing machinery easy to follow. It gives no bibliography or links to the underlying papers. The dates, performance figures, descriptions of the checkers and backgammon systems, and summaries of DeepMind and OpenAI robotics therefore remain claims made by Art of the Problem in this note. The captions misrecognise several names, including Michie, LeCun, and Tesauro. I have used the names supplied in the description and the established spellings of the researchers’ names.

The opening story about emotional brain damage appears as an unnamed anecdote. The video uses it to motivate the analogy between human feeling and a machine’s value function. That analogy explains the history’s organising idea, whilst it leaves open whether a reward signal amounts to an emotion or a feeling in the human sense.