Are you better than average as a driver? I know I am. I’ll bet 90 percent of you think you are, too, because this is the well-documented phenomenon known as the above-average effect, part of the psychology of optimism.

According to psychologist Daniel Kahneman, in his 2011 book Thinking, Fast and Slow, “people tend to be overly optimistic about their relative standing on any activity in which they do moderately well.” But optimism can slide dangerously into overoptimism. Research shows that chief financial officers, for example, “were grossly overconfident about their ability to forecast the market” when tested by Duke University professors who collected 11,600 CFO forecasts and matched them to market outcomes and found a correlation of less than zero. Such overconfidence can be costly. “The study of CFOs showed that those who were most confident and optimistic about the S&P index were also overconfident and optimistic about the prospects of their own firm, which went on to take more risk than others,” Kahneman notes.

Isn’t optimistic risk taking integral to building a successful business? Yes, to a point. “One of the benefits of an optimistic temperament is that it encourages persistence in the face of obstacles,” Kahneman explains. But “pervasive optimistic bias” can be detrimental: “Most of us view the world as more benign than it really is, our own attributes as more favorable than they truly are, and the goals we adopt as more achievable than they are likely to be.” For example, only 35 percent of small businesses survive in the U.S. When surveyed, however, 81 percent of entrepreneurs assessed their odds of success at 70 percent, and 33 percent of them went so far as to put their chances at 100 percent. So what? In a Canadian study Kahneman cites, 47 percent of inventors participating in the Inventor’s Assistance Program, in which they paid for objective evaluations of their invention on 37 criteria, “continued development efforts even after being told that their project was hopeless, and on average these persistent (or obstinate) individuals doubled their initial losses before giving up.” Failure may not be an option in the mind of an entrepreneur, but it is all too frequent in reality. High-risk-taking entrepreneurs override such loss aversion, a phenomenon most of us succumb to—in which losses hurt twice as much as gains feel good—that we developed in our evolutionary environment of scarcity and uncertainty.

This loss-aversion override by those with pervasive optimistic bias seems to work because of what I call biographical selection bias: the few entrepreneurs who succeed spectacularly have biographies (and autobiographies), whereas the many who fail do not.

Think Steve Jobs, whose pervasive optimistic bias was channeled through something a co-worker called Jobs’s “reality distortion field.” According to his biographer Walter Isaacson, “at the root of the reality distortion was Jobs’s belief that the rules didn’t apply to him.... He had the sense that he was special, a chosen one, an enlightened one.” Jobs’s optimism morphed into a reality-distorting will to power over rules that applied only to others and was reflected in numerous ways: legal (parking in handicapped spaces, driving without a license plate), moral (accusing Microsoft of ripping off Apple when both took from Xerox the idea of the mouse and the graphical user interface), personal (refusing to acknowledge his daughter Lisa even after an irrefutable paternity test), and practical (besting resource-heavy giant IBM in the computer market).

There was one reality Jobs’s distortion field optimism could not completely bend to his will: cancer. After he was diagnosed with a treatable form of pancreatic cancer, Jobs initially refused surgery. “I really didn’t want them to open up my body, so I tried to see if a few other things would work,” he admitted to Isaacson. Those other things included consuming large quantities of carrot and fruit juices, bowel cleansings, hydrotherapy, acupuncture and herbal remedies, a vegan diet, and, Isaacson says, “a few other treatments he found on the Internet or by consulting people around the country, including a psychic.” They didn’t work.

Out of this heroic tragedy a lesson emerges—reality must take precedence over willful optimism. Nature cannot be distorted.

The article was published in print as "Opting Out of Overoptimism ."