Everyone knows about the marshmallow test. Kids were given a marshmallow and told that they’d get a second one if they resisted eating the first one for a while. The experimenter then left the room and watched the kids endure marshmallow temptation. Years later, the kids who had been able to fight temptation were found to have higher SAT scores, better jobs, less addiction, and better physical fitness than those who succumbed. The meaning was clear; early self control, whether innate or taught, is key to later success. The test results and their interpretation were, scientifically speaking, too good to be true. And in most ways they weren’t true.

That wrinkle doesn’t stop the marshmallow test from being trotted out weekly on LinkedIn and social sites where experts and moralists opine. That trotting out comes with behavioral economics lessons, dripping with references to Kahnemann, Ariely and the like about our irrationality as we face intertemporal choices, as they’re known in the trade. When adults choose an offer of $1000 today over an offer for $1400 to be paid in one year, even when they have no pressing financial need, they are deemed irrational or lacking self control, like the marshmallow kids.

The famous marshmallow test was done by Walter Mischel in the 1960s through 1980s. Not only did subsequent marshmallow tests fail to show as much correlation between not waiting for the second marshmallow and a better life, but, more importantly, similar tests for at least twenty years have pointed to a more salient result, one which Mischel was aware of, but which got lost in popular retelling. Understanding the deeper implications of the marshmallow tests, along with a more charitable view of kids who grabbed the early treat, requires digging down into the design of experiments, Bayesian reasoning, and the concept of risk neutrality.

Intertemporal choice tests like the marshmallow test involve choices between options that involve different payoffs at different times. We face these choices often. And when we face them in the real world, our decision process is informed by memories and judgments about our past choices and their outcomes. In Bayesian terms, our priors incorporate this history. In real life, we are aware that all contracts, treaties, and promises for future payment come with a finite risk of default.

In intertemporal choice scenarios, the probability of the deferred payment actually occurring is always less than 100%. That probability is rarely known and is often unknowable. Consider choices A and B below. This is how the behavioral economists tend to frame the choices.

But this framing ignores an important feature of any real-world, non-hypothetical intertemporal choice situation: the probability of choice B is always less than 100%. In the above example, even risk-neutral choosers (those indifferent to all choices having the same expected value) would pick choice A over choice B if they judge the probability of non-default (actually getting the deferred payment) to be less than a certain amount.

As shown above, if choosers believe the deferred payment likelihood to be less than about 70%, they cannot be  called irrational for choosing choice A.

Lack of Self Control – or Rational Intuitive Bayes?

Now for the final, most interesting twist in tests like the marshmallow test, almost universally ignored by those who cite them. Unlike my example above where the wait time is one year, in the marshmallow tests, the time period during which the subject is tempted to eat the first marshmallow is unknown to the subject. Subjects come into the game with a certain prior – a certain belief about the probability of non-default. But, as intuitive Bayesians, these subjects update the probability they assign to non-default, during their wait, based on the amount of time they have been waiting. The speed at which they revise their probability downward depends on their judgment of the distribution of wait times experienced in their short lives.

If kids in the marshmallow tests have concluded, based on their experience, that adults are not dependable, choice A makes sense; they should immediately eat the first marshmallow, since the second one may never materialize. Kids who endure temptation for a few minutes only to give in and eat their first marshmallow are seen as both irrational and being incapable of self-control.

But if those kids adjust their probability judgments that the second marshmallow will appear based on a prior distribution that is not a normal distribution (i.e., if as intuitive Bayesians they model wait times imposed by adults as a power-law distribution), then their eating the first marshmallow after some test-wait period makes perfect sense. They rightly conclude, on the basis of available evidence, that wait times longer than some threshold period may be very long indeed. These kids aren’t irrational, and self-control is not their main problem. Their problem is that they have been raised by irresponsible adults who have both displayed a tendency to default on payments and who are late to fulfill promises by time durations obeying power-law distributions.

Subsequent marshmallow tests have verified this. In 2013, psychologist Laura Michaelson, after more sophisticated versions of the marshmallow test, concluded “implications of this work include the need to revise prominent theories of delay of gratification.” Actually, tests going back over 50 years have shown similar results (A.R. Mahrer, The role of expectancy in delayed reinforcement, 1956).

In three recent posts (first, second, third) I suggested that behavioral economists and business people who follow them are far too prone to seeing innate bias everywhere, when they are actually seeing rational behavior through their own bias. This is certainly the case with the common misuse of the marshmallow tests. Interpreting these tests as rational behavior in light of subjects’ experience is a better explanatory theory, one more consistent with the evidence, and one that coheres with other explanatory observations, such as humans’ capacity for intuitive Bayesian belief updates.

Charismatic pessimists about human rationality twist the situation so that their pessimism is framed as good news, in the sense that they have at least illuminated an inherent human bias. That pessimism, however cheerfully expressed, is both misguided and harmful. Their failure to mention the more nuanced interpretation of marshmallow tests is dishonest and self-serving. The problem we face is not innate, and it is mostly curable. Better parenting can fix it. The marshmallow tests measure parents more than they measure kids.

Walter Mischel died in 2018. I heard his 2016 talk at the Long Now Foundation in San Francisco. He acknowledged the relatively weak correlation between marshmallow test results and later success, and he mentioned that descriptions of his experiments in popular press were rife with errors. But his talk still focused almost solely on the self-control aspect of the experiments. He missed a great opportunity to help disseminate a better story about the role of trustworthiness and reliability of parents in delayed gratification of children.

A better description of the way we really work through intertemporal choices would require going deeper into risk neutrality and how, even for a single person, our departure from risk neutrality – specifically risk-appetite skewness – varies between situations and across time. I have enjoyed doing some professional work in that area. Getting it across in a blog post is probably beyond my current blog-writing skills.

William Storage – 8/1/2016
Visiting Scholar, UC Berkeley History of Science

Nearly everything relies on science. Having been the main vehicle of social change in the west, science deserves the special epistemic status that it acquired in the scientific revolution. By special epistemic status, I mean that science stands privileged as a way of knowing. Few but nihilists, new-agers, and postmodernist diehards would disagree.

That settled, many are surprised by claims that there is not really a scientific method, despite what you learned in 6^th grade. A recent New York Times piece by James Blachowicz on the absence of a specific scientific method raised the ire of scientists, Forbes science writer Ethan Siegel (Yes, New York Times, There Is A Scientific Method), and a cadre of Star Trek groupies.

Siegel is a prolific writer who does a fine job of making science interesting and understandable. But I’d like to show here why, on this particular issue, he is very far off the mark. I’m not defending Blachowicz, but am disputing Siegel’s claim that the work of Kepler and Galileo “provide extraordinary examples of showing exactly how… science is completely different than every other endeavor” or that it is even possible to identify a family of characteristics unique to science that would constitute a “scientific method.”

Siegel ties science’s special status to the scientific method. To defend its status, Siegel argues “[t]he point of Galileo’s is another deep illustration of how science actually works.” He praises Galileo for idealizing a worldly situation to formulate a theory of falling bodies, but doesn’t explain any associated scientific method.

Galileo’s pioneering work on mechanics of solids and kinematics in Two New Sciences secured his place as the father of modern physics. But there’s more to the story of Galileo if we’re to claim that he shows exactly how science is special.

A scholar of Siegel’s caliber almost certainly knows other facts about Galileo relevant to this discussion – facts that do damage to Siegel’s argument – yet he withheld them. Interestingly, Galileo used this ploy too. Arguing without addressing known counter-evidence is something that science, in theory, shouldn’t tolerate. Yet many modern scientists do it – for political or ideological reasons, or to secure wealth and status. Just like Galileo did. The parallel between Siegel’s tactics and Galileo’s approach in his support of Copernican world view is ironic. In using Galileo as an exemplar of scientific method, Siegel failed to mention that Galileo failed to mention significant problems with the Copernican model – problems that Galileo knew well.

In his support of a sun-centered astronomical model, Galileo faced hurdles. Copernicus’s model said that the sun was motionless and that the planets revolved around it in circular orbits with constant speed. The ancient Ptolemaic model, endorsed by the church, put earth at the center. Despite obvious disagreement with observational evidence (the retrograde motions of outer planets), Ptolemy faced no serious challenges in nearly 2000 years. To explain the inconsistencies with observation, Ptolemy’s model included layers of epicycles, which had planets moving in small circles around points on circular orbits around the sun. Copernicus thought his model would get rid of the epicycles; but it didn’t. So the Copernican model took on its own epicycles to fit astronomical data.

Let’s stop here and look at method. Copernicus (~1540) didn’t derive his theory from any new observations but from an ancient speculation by Aristarchus (~250 BC). Everything available to Copernicus had been around for a thousand years. His theory couldn’t be tested in any serious way. It was wrong about circular orbits and uniform planet speed. It still needed epicycles, and gave no better predictions than the existing Ptolemaic model. Copernicus acted simply on faith, or maybe he thought his model simpler or more beautiful. In any case, it’s hard to see that Copernicus, or his follower, Galileo, applied much method or had much scientific basis for their belief.

In Galileo’s early writings on the topic, he gave no new evidence for a moving earth and no new disconfirming evidence for a moving sun. Galileo praised Copernicus for advancing the theory in spite of its being inconsistent with observations. You can call Copernicus’s faith aspirational as opposed to religious faith; but it is hard to reconcile this quality with any popular account of scientific method. Yet it seems likely that faith, dogged adherence to a contrarian hunch, or something similar was exactly what was needed to advance science at that moment in history. Needed, yes, but hard to reconcile with any scientific method and hard to distance from the persuasive tools used by poets, priests and politicians.

In Dialogue Concerning the Two Chief World Systems, Galileo sets up a false choice between Copernicanism and Ptolemaic astronomy (the two world systems). The main arguments against Copernicanism were the lack of parallax in observations of stars and the absence of lateral displacement of a falling body from its drop point. Galileo guessed correctly on the first point; we don’t see parallax because stars are just too far away. On the latter point he (actually his character Salviati) gave a complex but nonsensical explanation. Galileo did, by this time, have new evidence. Venus shows a full set of phases, a fact that strongly contradicts Ptolemaic astronomy.

But Ptolemaic astronomy was a weak opponent compared to the third world system (4^th if we count Aristotle’s), the Tychonic system, which Galileo knew all too well. Tycho Brahe’s model solved the parallax problem, the falling body problem, and the phases of Venus. For Tycho, the earth holds still, the sun revolves around it, Mercury and Venus orbit the sun, and the distant planets orbit both the sun and the earth. Based on available facts at the time, Tycho’s model was most scientific – observational indistinguishable from Galileo’s model but without its flaws.

In addition to dodging Tycho, Galileo also ignored Kepler’s letters to him. Kepler had shown that orbits were not circular but elliptical, and that planets’ speeds varied during their orbits; but Galileo remained an orthodox Copernican all his life. As historian John Heilbron notes in Galileo, “Galileo could stick to an attractive theory in the face of overwhelming experimental refutation,” leaving modern readers to wonder whether Galileo was a quack or merely dishonest. Some of each, perhaps, and the father of modern physics. But can we fit his withholding evidence, mocking opponents, and baffling with bizzarria into a scientific method?

Nevertheless, Galileo was right about the sun-centered system, despite the counter-evidence; and we’re tempted to say he knew he was right. This isn’t easy to defend given that Galileo also fudged his data on pendulum periods, gave dishonest arguments on comet orbits, and wrote horoscopes even when not paid to do so. This brings up the thorny matter of theory choice in science. A dispute between competing scientific theories can rarely be resolved by evidence, experimentation, and deductive reasoning. All theories are under-determined by data. Within science, common criteria for theory choice are accuracy, consistency, scope, simplicity, and explanatory power. These are good values by which to test theories; but they compete with one another.

Galileo likely defended heliocentrism with such gusto because he found it simpler than the Tychonic system. That works only if you value simplicity above consistency and accuracy. And the desire for simplicity might be, to use Galileo’s words, just a metaphysical urge. If we promote simplicity to the top of the theory-choice criteria list, evolution, genetics and stellar nucleosynthesis would not fare well.

Whatever method you examine in a list of any proposed family of scientific methods will not be consistent with the way science has made progress. Competition between theories is how science advances; and it’s untidy, entailing polemical and persuasive tactics. Historian Paul Feyerabend argues that any conceivable set of rules, if followed, would have prevented at least one great scientific breakthrough. That is, if method is the distinguishing feature of science as Siegel says, it’s going to be tough to find a set of methods that let evolution, cosmology, and botany in while keeping astrology, cold fusion and parapsychology out.

This doesn’t justify epistemic relativism or mean that science isn’t special; but it does make the concept of scientific method extremely messy. About all we can say about method is that the history of  science reveals that its most accomplished practitioners aimed to be methodical but did not agree on a particular method. Looking at their work, we see different combinations of experimentation, induction, deduction and creativity as required by the theories they pursued. But that isn’t much of a definition of scientific method, which is probably why Siegel, for example, in hailing scientific method, fails to identify one.

–  –  –

[edit 8/4/16] For another take on this story, see “Getting Kepler Wrong” at The Renaissance Mathematicus. Also, Psybertron Asks (“More on the Myths of Science”) takes me to task for granting science special epistemic status from authority.

–  –  –

.

“There are many ways to produce scientific bullshit. One way is to assert that something has been ‘proven,’ ‘shown,’ or ‘found’ and then cite, in support of this assertion, a study that has actually been heavily critiqued … without acknowledging any of the published criticisms of the study or otherwise grappling with its inherent limitations.”- Brain D Earp, The Unbearable Asymmetry of Bullshit

“One can show the following: given any rule, however ‘fundamental’ or ‘necessary’ for science, there are always circumstances when it is advisable not only to ignore the rule, but to adopt its opposite.” – Paul Feyerabend

“Trying to understand the way nature works involves a most terrible test of human reasoning ability. It involves subtle trickery, beautiful tightropes of logic on which one has to walk in order not to make a mistake in predicting what will happen. The quantum mechanical and the relativity ideas are examples of this.” – Richard Feynman