The market crash of 2008 that plunged the world into the economic recession from which it is still reeling had many causes. One of them was mathematics. Financial investment firms had developed such complex ways of investing their clients’ money that they came to rely on arcane formulas to judge the risks they were taking on. Yet as we learned so painfully three years ago, those formulas, or models, are only pale reflections of the real world, and sometimes they can be woefully misleading.

The financial world is not alone, of course, in depending on mathematical models that aren’t always reliable for decision-making guid­ance. Scientists struggle with models in many fields—including climate science, coastal erosion and nuclear safety—in which the phenomena they describe are very complex, or information is hard to come by, or, as is the case with financial models, both. But in no area of human activity is so much faith placed in such flimsy science as finance.

It was the supposed strength of risk models that gave investment firms the confidence to leverage their bets with massive sums of borrowed money. The models would tell them how risky these bets were and what other investments would offset that risk. Yet the huge uncertainties in the models gave them false con­fidence. “We just don’t know enough to get a good theoretical grasp of the financial risks we face,” comments David Colander, an economist at Middlebury College who has studied the 2008 crisis. “The idea that we have models that can account for all the uncertainty and unpredictable behavior we see in markets is just crazy. But that’s how the models have been used.”

Blaming the economic calamity on risk models would be an oversimplification. Other human factors—political and regulatory ones—certainly came into play. Still, the models were arguably a crucial link, perhaps even the sine qua non of economic disaster. With so much at stake, in the past three years financial firms have spent tens of millions of dollars in buttressing their models of investment risk in the hope that new ones will preclude anything like the 2008 collapse from happening again. But that may be a vain hope or a case of wishful thinking. Experts in financial models have serious doubts about whether risk models can be improved in any fundamental way. What this means is as obvious as it is scary: banks and investment firms are leading the global economy into a future that is at great risk of repeating the past.

The Rosy Future circa 2007
In a sense, the downfall of the risk models in 2007 and 2008 is simple to understand. The models were supposed to simulate the complex interactions of many market forces on one another, including fluctuations in markets, changing interest rates, prices of various stocks, bonds, options and other financial instruments. Even if they did that—that’s arguable—they failed to account for one important scenario: What happens when everybody wants to sell all their holdings at the same time? This is precisely what happened in those dark days of September 2008, when the U.S. government decided not to bail out Lehman Brothers, and the venerable institution defaulted on its creditors. The domino effect of collapse was averted only by massive infusions of money from the federal government.

Through 2007 the risk models indicated that the chance of any major institution defaulting was minimal. According to Marco Avellaneda, a New York University mathematician and expert on financial risk models, a big problem was that the models omitted a major variable affecting the health of a portfolio: liquidity, or the ability of a market to match buyers and sellers. A missing key variable is a big deal—an equation that predicts an airplane flight’s risk of arriving late will not be very reliable if it has no mathematical term representing weather delays. And liquidity may be the most important variable in assessing the risk of default in mortgage-backed securities, the various financial instruments woven around the explosion of home lending that had taken place over the previous decade, particularly to riskier, or “subprime,” borrowers. When housing prices began to fall in 2008, no one was sure just how much these instruments were worth, and as a result, trading in them ground to a halt—the instruments had become “illiquid.” That left the banks that were holding them with no way of cashing out, causing panic among investors. If financial models had properly identified illiquidity risk, Avellaneda says, banks could have dropped prices of the instruments sooner, so that buyers could put less money at risk.

Omitting a key variable seems egregious, but scientists do it all the time. Sometimes they are unaware that a variable plays a key role, or they do not know how to account for it. That is a problem in climate science, Colander says, where models often have no terms to account for the effects of clouds. “Clouds control 60 percent of the weather, and models usually ignore them,” he notes. “When you can’t model a factor that has that kind of influence on the outcome, you have to use a lot of judgment in whether to believe the results.” The problem crops up in many other situations. How do you account for the willingness of the public to get vaccines when modeling the spread of a new, dangerous form of flu? Or of the ability of emergency response teams to replace faulty parts and put out fires in overheating nuclear power plants?

Once an oversight in a model is clearly identified—typically the hard way—it may or may not be possible to remedy it. In the case of financial risk models, accounting for illiquidity isn’t easy, says Robert Jarrow, a Cornell University finance and economics professor who focuses on risk models, because illiquidity tends to be much more nonlinear than the normal behavior of prices. Markets go from high liquidity to no liquidity in the blink of an eye, so it is like the difference between modeling airflow around an airplane flying at ordinary speeds and around one cracking the sound barrier (a lot of aircraft got into trouble before aerospace modelers got that one right). Jarrow is working on adding illiquidity risk to models but cautions that the resulting equations do not have single, neat solutions. Illiquidity is inherently unpredictable—no mathematical model can tell you when buyers will decide that a financial instrument isn’t worth the risk at any price. To account for this behavior, models have to accommodate a range of possible solutions, and deciding between them may be problematic. “The models I’m working on are potentially useful for estimating illiquidity risk, but they’re far from perfect,” Jarrow says.

Unfortunately, missing illiquidity risk wasn’t the only major problem. Financial risk models have been designed to focus on the risk faced by an individual institution. That always seemed to make sense because institutions are concerned only with their own risk, and regulators assumed that if the risk to each individual institution is low, then the system is safe. But the assumption turned out to be poor, says Rama Cont, director of Columbia University’s Center for Financial Engineering. In a system where many interdependent components each have a low risk of failure, he notes, systemic risk can still be excessive. Imagine 30 people walking side by side across a field with their arms around one another’s shoulders—any one person may be unlikely to stumble, but there’s a decent chance someone in the group will, and that one stumbler could bring down a chunk of the line. That’s the situation financial institutions are in, Cont says. “Up through 2008, regulators weren’t considering the connections between these banks in assessing risk,” he observes. “They should have at least noticed that were all highly invested in the subprime mortgage market.”

The Disaster Map
The electric power industry faces an analogous problem, Cont observes. The chances of an individual power plant failing is tiny, but one does occasionally fail somewhere, and it can overload other plants on the grid, threatening a large-scale blackout of the kind the U.S. saw in 1965, 1977 and 2003. To lower such systemic risk, power companies do N-1 testing—running scenarios in which a single plant goes down in an effort to predict what will happen to the grid. But Cont points out that the power industry has the advantage of knowing how all its plants are connected. The financial system, in contrast, is a black box. “Right now nobody knows what the financial system looks like,” he says. “We don’t know exactly who transacts what with whom and for how much. That means we can’t predict the consequences to other banks of the failure of a Lehman Brothers. In 2008 regulators had 48 hours to come up with a guess.”

The obvious solution is to map out those connections. Cont has been among those actively lobbying to force financial institutions to report all their transactions to a centralized data-gathering arm set up by the government—not just domestically but also internationally because money moves fluidly across borders now. Banks are loath to report those data, however. Telling the world about an ongoing large investment could trigger copycat buying and raise prices, whereas a big sell-off could signal financial problems and lead investors to yank their money out. Those concerns can be addressed by ensuring that all reports are confidential to the data-gathering agency, Cont says. “Governments have been sharing confidential data about nuclear capabilities with international agencies for years,” he explains. “Financial data aren’t more sensitive than that.” In fact, the Dodd-Frank Act, signed into law in the U.S. in 2010, provides for an “office of financial research” that could in principle serve as a data-collection agency for American institutions. Still, there is no evidence as of yet that any agency will be able to collect all the data needed to fashion a detailed, up-to-date map of the global financial system, which means that we may remain as oblivious to systemic risk as we were in 2007.

Even if regulators had enough data, models are not yet sophisticated enough to handle them. Existing models, argues Stanford University finance professor Darrell Duffie, are probabilistic—they make no assumptions about the future but rather kick out the chances of a default under any of the infinite number of conditions that might prevail in the future. Needless to say, doing so reliably demands not only vast rivers of data but also a superb understanding of all the various forces at play, complex math and enormous computing power. And that’s just for individual banks. The notion of extending those already daunting demands to the entire financial system is almost absurd, Duffie says.

Duffie proposes an alternative: scenario stress testing, or simply spelling out a number of clear-cut future scenarios that might pose unusual risks to a bank’s health. Identifying default risk under a constrained scenario is a simpler problem. For example, if you were trying to get a handle on the risk of your not being able to make your mortgage payments at some point, consider how much easier you would find it to guess how you would weather a 10 percent pay cut than to have to calculate how you would fare in the face of any or all possible future events. For banks, the selected scenarios might include a stock-market plunge, mortgage defaults, skyrocketing interest rates, and so forth. These scenarios would also include one or more financial institutions defaulting, to see how such defaults would affect the bank doing the testing. “The idea is to send huge, simulated shocks through a bank’s portfolio and see how the bank would perform going forward,” Duffie says. “It doesn’t matter what the probability is of that particular scenario occurring; it still tells you a lot about where you might have problems.”

Duffie recommends asking banks to respond to perhaps 10 or so different judiciously chosen scenarios, each involving the possible default of any of 10 different banks. Make 10 banks do that, Duffie says, and you have a 10-by-10-by-10 matrix that should give regulators a good sense of where the systemic risks lie. If key banks had been specifically asked in, say, 2006 to assess the impact on their portfolios of exploding mortgage defaults and the collapse of two giant financial institutions, regulators might well have had all they needed to know to take action in prodding the financial system to smoothly unwind its precarious position. The downside of this approach, he concedes, is that stress testing can realistically cover only an infinitesimal fraction of the scenarios that might be encountered—a bank couldn’t be asked to churn out risk estimates for thousands of different scenarios involving defaults of hundreds of different banks. That means that even after scenario testing has shown the system to be relatively stable against the specified shocks, the system could still be taken down by one of the countless scenarios that weren’t part of the testing.

Another problem with making complex models is that at some point their very complexity gets in the way. Paul Wilmott, an applied mathematician and former hedge fund manager, says modelers often end up bogging their creations down with dozens of terms loaded with different variables and parameters—each one adding more potential error, so that the net effect is grossly inaccurate. Wilmott advocates for finding what he calls the “math sweet spot,” where a model has enough terms to provide a reasonable approximation of reality but is still simple enough for its functioning and limitations to be fully understood. Few modelers succeed in finding that balance, he adds.

It’s a safe bet that financial risk models will remain unreliable for years to come. So what should we do about it? The only real option is not to trust the models, no matter how good the equations seem to be in theory. Such thinking, though, conflicts with the core ethos of Wall Street. “There has never been any incentive to distrust the models because the people in control keep making lots of money using them,” Jarrow says. “Everyone thought the models were working right up until the crisis. Now they’re trusting them again.” The models and data are likely to improve, he asserts, but not enough to justify much faith in the results.

If regulators heeded these cautions, they would force banks to keep more cash on hand and make safer investments. The price of this reasonable caution, Avellaneda notes, will be a system that doesn’t operate as efficiently—in other words, investors will get less rich off it, on average. Banks will have lower profits and less money to loan. We all will find it a little tougher to get ahead, but we will be less likely to fly headlong and clueless into a crash. That’s the trade-off.