The Trouble with Quantum Mechanics

The development of quantum mechanics in the first decades of the twentieth century came as a shock to many physicists. Today, despite the great successes of quantum mechanics, arguments continue about its meaning, and its future.

1.

The first shock came as a challenge to the clear categories to which physicists by 1900 had become accustomed. There were particles—atoms, and then electrons and atomic nuclei—and there were fields—conditions of space that pervade regions in which electric, magnetic, and gravitational forces are exerted. Light waves were clearly recognized as self-sustaining oscillations of electric and magnetic fields. But in order to understand the light emitted by heated bodies, Albert Einstein in 1905 found it necessary to describe light waves as streams of massless particles, later called photons.

Then in the 1920s, according to theories of Louis de Broglie and Erwin Schrödinger, it appeared that electrons, which had always been recognized as particles, under some circumstances behaved as waves. In order to account for the energies of the stable states of atoms, physicists had to give up the notion that electrons in atoms are little Newtonian planets in orbit around the atomic nucleus. Electrons in atoms are better described as waves, fitting around the nucleus like sound waves fitting into an organ pipe.¹ The world’s categories had become all muddled.

Worse yet, the electron waves are not waves of electronic matter, in the way that ocean waves are waves of water. Rather, as Max Born came to realize, the electron waves are waves of probability. That is, when a free electron collides with an atom, we cannot in principle say in what direction it will bounce off. The electron wave, after encountering the atom, spreads out in all directions, like an ocean wave after striking a reef. As Born recognized, this does not mean that the electron itself spreads out. Instead, the undivided electron goes in some one direction, but not a precisely predictable direction. It is more likely to go in a direction where the wave is more intense, but any direction is possible.

Probability was not unfamiliar to the physicists of the 1920s, but it had generally been thought to reflect an imperfect knowledge of whatever was under study, not an indeterminism in the underlying physical laws. Newton’s theories of motion and gravitation had set the standard of deterministic laws. When we have reasonably precise knowledge of the location and velocity of each body in the solar system at a given moment, Newton’s laws tell us with good accuracy where they will all be for a long time in the future. Probability enters Newtonian physics only when our knowledge is imperfect, as for example when we do not have precise knowledge of how a pair of dice is thrown. But with the new quantum mechanics, the moment-to-moment determinism of the laws of physics themselves seemed to be lost.

All very strange. In a 1926 letter to Born, Einstein complained:

Quantum mechanics is very impressive. But an inner voice tells me that it is not yet the real thing. The theory produces a good deal but hardly brings us closer to the secret of the Old One. I am at all events convinced that He does not play dice.²

As late as 1964, in his Messenger lectures at Cornell, Richard Feynman lamented, “I think I can safely say that no one understands quantum mechanics.”³ With quantum mechanics, the break with the past was so sharp that all earlier physical theories became known as “classical.”

The weirdness of quantum mechanics did not matter for most purposes. Physicists learned how to use it to do increasingly precise calculations of the energy levels of atoms, and of the probabilities that particles will scatter in one direction or another when they collide. Lawrence Krauss has labeled the quantum mechanical calculation of one effect in the spectrum of hydrogen “the best, most accurate prediction in all of science.”⁴ Beyond atomic physics, early applications of quantum mechanics listed by the physicist Gino Segrè included the binding of atoms in molecules, the radioactive decay of atomic nuclei, electrical conduction, magnetism, and electromagnetic radiation.⁵ Later applications spanned theories of semiconductivity and superconductivity, white dwarf stars and neutron stars, nuclear forces, and elementary particles. Even the most adventurous modern speculations, such as string theory, are based on the principles of quantum mechanics.

Many physicists came to think that the reaction of Einstein and Feynman and others to the unfamiliar aspects of quantum mechanics had been overblown. This used to be my view. After all, Newton’s theories too had been unpalatable to many of his contemporaries. Newton had introduced what his critics saw as an occult force, gravity, which was unrelated to any sort of tangible pushing and pulling, and which could not be explained on the basis of philosophy or pure mathematics. Also, his theories had renounced a chief aim of Ptolemy and Kepler, to calculate the sizes of planetary orbits from first principles. But in the end the opposition to Newtonianism faded away. Newton and his followers succeeded in accounting not only for the motions of planets and falling apples, but also for the movements of comets and moons and the shape of the earth and the change in direction of its axis of rotation. By the end of the eighteenth century this success had established Newton’s theories of motion and gravitation as correct, or at least as a marvelously accurate approximation. Evidently it is a mistake to demand too strictly that new physical theories should fit some preconceived philosophical standard.

In quantum mechanics the state of a system is not described by giving the position and velocity of every particle and the values and rates of change of various fields, as in classical physics. Instead, the state of any system at any moment is described by a wave function, essentially a list of numbers, one number for every possible configuration of the system.⁶ If the system is a single particle, then there is a number for every possible position in space that the particle may occupy. This is something like the description of a sound wave in classical physics, except that for a sound wave a number for each position in space gives the pressure of the air at that point, while for a particle in quantum mechanics the wave function’s number for a given position reflects the probability that the particle is at that position. What is so terrible about that? Certainly, it was a tragic mistake for Einstein and Schrödinger to step away from using quantum mechanics, isolating themselves in their later lives from the exciting progress made by others.

2.

Even so, I’m not as sure as I once was about the future of quantum mechanics. It is a bad sign that those physicists today who are most comfortable with quantum mechanics do not agree with one another about what it all means. The dispute arises chiefly regarding the nature of measurement in quantum mechanics. This issue can be illustrated by considering a simple example, measurement of the spin of an electron. (A particle’s spin in any direction is a measure of the amount of rotation of matter around a line pointing in that direction.)

All theories agree, and experiment confirms, that when one measures the amount of spin of an electron in any arbitrarily chosen direction there are only two possible results. One possible result will be equal to a positive number, a universal constant of nature. (This is the constant that Max Planck originally introduced in his 1900 theory of heat radiation, denoted h, divided by 4π.) The other possible result is its opposite, the negative of the first. These positive or negative values of the spin correspond to an electron that is spinning either clockwise or counter-clockwise in the chosen direction.

But it is only when a measurement is made that these are the sole two possibilities. An electron spin that has not been measured is like a musical chord, formed from a superposition of two notes that correspond to positive or negative spins, each note with its own amplitude. Just as a chord creates a sound distinct from each of its constituent notes, the state of an electron spin that has not yet been measured is a superposition of the two possible states of definite spin, the superposition differing qualitatively from either state. In this musical analogy, the act of measuring the spin somehow shifts all the intensity of the chord to one of the notes, which we then hear on its own.

This can be put in terms of the wave function. If we disregard everything about an electron but its spin, there is not much that is wavelike about its wave function. It is just a pair of numbers, one number for each sign of the spin in some chosen direction, analogous to the amplitudes of each of the two notes in a chord.⁷ The wave function of an electron whose spin has not been measured generally has nonzero values for spins of both signs.

There is a rule of quantum mechanics, known as the Born rule, that tells us how to use the wave function to calculate the probabilities of getting various possible results in experiments. For example, the Born rule tells us that the probabilities of finding either a positive or a negative result when the spin in some chosen direction is measured are proportional to the squares of the numbers in the wave function for those two states of the spin.⁸

The introduction of probability into the principles of physics was disturbing to past physicists, but the trouble with quantum mechanics is not that it involves probabilities. We can live with that. The trouble is that in quantum mechanics the way that wave functions change with time is governed by an equation, the Schrödinger equation, that does not involve probabilities. It is just as deterministic as Newton’s equations of motion and gravitation. That is, given the wave function at any moment, the Schrödinger equation will tell you precisely what the wave function will be at any future time. There is not even the possibility of chaos, the extreme sensitivity to initial conditions that is possible in Newtonian mechanics. So if we regard the whole process of measurement as being governed by the equations of quantum mechanics, and these equations are perfectly deterministic, how do probabilities get into quantum mechanics?

One common answer is that, in a measurement, the spin (or whatever else is measured) is put in an interaction with a macroscopic environment that jitters in an unpredictable way. For example, the environment might be the shower of photons in a beam of light that is used to observe the system, as unpredictable in practice as a shower of raindrops. Such an environment causes the superposition of different states in the wave function to break down, leading to an unpredictable result of the measurement. (This is called decoherence.) It is as if a noisy background somehow unpredictably left only one of the notes of a chord audible. But this begs the question. If the deterministic Schrödinger equation governs the changes through time not only of the spin but also of the measuring apparatus and the physicist using it, then the results of measurement should not in principle be unpredictable. So we still have to ask, how do probabilities get into quantum mechanics?

One response to this puzzle was given in the 1920s by Niels Bohr, in what came to be called the Copenhagen interpretation of quantum mechanics. According to Bohr, in a measurement the state of a system such as a spin collapses to one result or another in a way that cannot itself be described by quantum mechanics, and is truly unpredictable. This answer is now widely felt to be unacceptable. There seems no way to locate the boundary between the realms in which, according to Bohr, quantum mechanics does or does not apply. As it happens, I was a graduate student at Bohr’s institute in Copenhagen, but he was very great and I was very young, and I never had a chance to ask him about this.

Today there are two widely followed approaches to quantum mechanics, the “realist” and “instrumentalist” approaches, which view the origin of probability in measurement in two very different ways.⁹ For reasons I will explain, neither approach seems to me quite satisfactory.¹⁰

3.

The instrumentalist approach is a descendant of the Copenhagen interpretation, but instead of imagining a boundary beyond which reality is not described by quantum mechanics, it rejects quantum mechanics altogether as a description of reality. There is still a wave function, but it is not real like a particle or a field. Instead it is merely an instrument that provides predictions of the probabilities of various outcomes when measurements are made.

It seems to me that the trouble with this approach is not only that it gives up on an ancient aim of science: to say what is really going on out there. It is a surrender of a particularly unfortunate kind. In the instrumentalist approach, we have to assume, as fundamental laws of nature, the rules (such as the Born rule I mentioned earlier) for using the wave function to calculate the probabilities of various results when humans make measurements. Thus humans are brought into the laws of nature at the most fundamental level. According to Eugene Wigner, a pioneer of quantum mechanics, “it was not possible to formulate the laws of quantum mechanics in a fully consistent way without reference to the consciousness.”¹¹

Thus the instrumentalist approach turns its back on a vision that became possible after Darwin, of a world governed by impersonal physical laws that control human behavior along with everything else. It is not that we object to thinking about humans. Rather, we want to understand the relation of humans to nature, not just assuming the character of this relation by incorporating it in what we suppose are nature’s fundamental laws, but rather by deduction from laws that make no explicit reference to humans. We may in the end have to give up this goal, but I think not yet.

Some physicists who adopt an instrumentalist approach argue that the probabilities we infer from the wave function are objective probabilities, independent of whether humans are making a measurement. I don’t find this tenable. In quantum mechanics these probabilities do not exist until people choose what to measure, such as the spin in one or another direction. Unlike the case of classical physics, a choice must be made, because in quantum mechanics not everything can be simultaneously measured. As Werner Heisenberg realized, a particle cannot have, at the same time, both a definite position and a definite velocity. The measuring of one precludes the measuring of the other. Likewise, if we know the wave function that describes the spin of an electron we can calculate the probability that the electron would have a positive spin in the north direction if that were measured, or the probability that the electron would have a positive spin in the east direction if that were measured, but we cannot ask about the probability of the spins being found positive in both directions because there is no state in which an electron has a definite spin in two different directions.

4.

These problems are partly avoided in the realist—as opposed to the instrumentalist—approach to quantum mechanics. Here one takes the wave function and its deterministic evolution seriously as a description of reality. But this raises other problems.

The realist approach has a very strange implication, first worked out in the 1957 Princeton Ph.D. thesis of the late Hugh Everett. When a physicist measures the spin of an electron, say in the north direction, the wave function of the electron and the measuring apparatus and the physicist are supposed, in the realist approach, to evolve deterministically, as dictated by the Schrödinger equation; but in consequence of their interaction during the measurement, the wave function becomes a superposition of two terms, in one of which the electron spin is positive and everyone in the world who looks into it thinks it is positive, and in the other the spin is negative and everyone thinks it is negative. Since in each term of the wave function everyone shares a belief that the spin has one definite sign, the existence of the superposition is undetectable. In effect the history of the world has split into two streams, uncorrelated with each other.

This is strange enough, but the fission of history would not only occur when someone measures a spin. In the realist approach the history of the world is endlessly splitting; it does so every time a macroscopic body becomes tied in with a choice of quantum states. This inconceivably huge variety of histories has provided material for science fiction,¹² and it offers a rationale for a multiverse, in which the particular cosmic history in which we find ourselves is constrained by the requirement that it must be one of the histories in which conditions are sufficiently benign to allow conscious beings to exist. But the vista of all these parallel histories is deeply unsettling, and like many other physicists I would prefer a single history.

There is another thing that is unsatisfactory about the realist approach, beyond our parochial preferences. In this approach the wave function of the multiverse evolves deterministically. We can still talk of probabilities as the fractions of the time that various possible results are found when measurements are performed many times in any one history; but the rules that govern what probabilities are observed would have to follow from the deterministic evolution of the whole multiverse. If this were not the case, to predict probabilities we would need to make some additional assumption about what happens when humans make measurements, and we would be back with the shortcomings of the instrumentalist approach. Several attempts following the realist approach have come close to deducing rules like the Born rule that we know work well experimentally, but I think without final success.

The realist approach to quantum mechanics had already run into a different sort of trouble long before Everett wrote about multiple histories. It was emphasized in a 1935 paper by Einstein with his coworkers Boris Podolsky and Nathan Rosen, and arises in connection with the phenomenon of “entanglement.”¹³

We naturally tend to think that reality can be described locally. I can say what is happening in my laboratory, and you can say what is happening in yours, but we don’t have to talk about both at the same time. But in quantum mechanics it is possible for a system to be in an entangled state that involves correlations between parts of the system that are arbitrarily far apart, like the two ends of a very long rigid stick.

For instance, suppose we have a pair of electrons whose total spin in any direction is zero. In such a state, the wave function (ignoring everything but spin) is a sum of two terms: in one term, electron A has positive spin and electron B has negative spin in, say, the north direction, while in the other term in the wave function the positive and negative signs are reversed. The electron spins are said to be entangled. If nothing is done to interfere with these spins, this entangled state will persist even if the electrons fly apart to a great distance. However far apart they are, we can only talk about the wave function of the two electrons, not of each separately. Entanglement contributed to Einstein’s distrust of quantum mechanics as much or more than the appearance of probabilities.

Strange as it is, the entanglement entailed by quantum mechanics is actually observed experimentally. But how can something so nonlocal represent reality?

5.

What then must be done about the shortcomings of quantum mechanics? One reasonable response is contained in the legendary advice to inquiring students: “Shut up and calculate!” There is no argument about how to use quantum mechanics, only how to describe what it means, so perhaps the problem is merely one of words.

On the other hand, the problems of understanding measurement in the present form of quantum mechanics may be warning us that the theory needs modification. Quantum mechanics works so well for atoms that any new theory would have to be nearly indistinguishable from quantum mechanics when applied to such small things. But a new theory might be designed so that the superpositions of states of large things like physicists and their apparatus even in isolation suffer an actual rapid spontaneous collapse, in which probabilities evolve to give the results expected in quantum mechanics. The many histories of Everett would naturally collapse to a single history. The goal in inventing a new theory is to make this happen not by giving measurement any special status in the laws of physics, but as part of what in the post-quantum theory would be the ordinary processes of physics.

One difficulty in developing such a new theory is that we get no direction from experiment—all data so far agree with ordinary quantum mechanics. We do get some help, however, from some general principles, which turn out to provide surprisingly strict constraints on any new theory.

Obviously, probabilities must all be positive numbers, and add up to 100 percent. There is another requirement, satisfied in ordinary quantum mechanics, that in entangled states the evolution of probabilities during measurements cannot be used to send instantaneous signals, which would violate the theory of relativity. Special relativity requires that no signal can travel faster than the speed of light. When these requirements are put together, it turns out that the most general evolution of probabilities satisfies an equation of a class known as Lindblad equations.¹⁴ The class of Lindblad equations contains the Schrödinger equation of ordinary quantum mechanics as a special case, but in general these equations involve a variety of new quantities that represent a departure from quantum mechanics. These are quantities whose details of course we now don’t know. Though it has been scarcely noticed outside the theoretical community, there already is a line of interesting papers, going back to an influential 1986 article by Gian Carlo Ghirardi, Alberto Rimini, and Tullio Weber at Trieste, that use the Lindblad equations to generalize quantum mechanics in various ways.

Lately I have been thinking about a possible experimental search for signs of departure from ordinary quantum mechanics in atomic clocks. At the heart of any atomic clock is a device invented by the late Norman Ramsey for tuning the frequency of microwave or visible radiation to the known natural frequency at which the wave function of an atom oscillates when it is in a superposition of two states of different energy. This natural frequency equals the difference in the energies of the two atomic states used in the clock, divided by Planck’s constant. It is the same under all external conditions, and therefore serves as a fixed reference for frequency, in the way that a platinum-iridium cylinder at Sèvres serves as a fixed reference for mass.

Tuning the frequency of an electromagnetic wave to this reference frequency works a little like tuning the frequency of a metronome to match another metronome. If you start the two metronomes together and the beats still match after a thousand beats, you know that their frequencies are equal at least to about one part in a thousand. Quantum mechanical calculations show that in some atomic clocks the tuning should be precise to one part in a hundred million billion, and this precision is indeed realized. But if the corrections to quantum mechanics represented by the new terms in the Lindblad equations (expressed as energies) were as large as one part in a hundred million billion of the energy difference of the atomic states used in the clock, this precision would have been quite lost. The new terms must therefore be even smaller than this.

How significant is this limit? Unfortunately, these ideas about modifications of quantum mechanics are not only speculative but also vague, and we have no idea how big we should expect the corrections to quantum mechanics to be. Regarding not only this issue, but more generally the future of quantum mechanics, I have to echo Viola in Twelfth Night: “O time, thou must untangle this, not I.”