Messy Markets: Financial models get put to the test in the New York Stock Exchange on Wall Street and other markets.
Image: ToonariPost
More In This Article
-
Overview
A Formula For Economic Calamity
-
The Best Science Writing Online 2012
Showcasing more than fifty of the most provocative, original, and significant online essays from 2011, The Best Science Writing Online 2012 will change the way...
Read More »
When it comes to assigning blame for the current economic doldrums, the quants who build the complicated mathematic financial risk models, and the traders who rely on them, deserve their share of the blame. [See “A Formula For Economic Calamity” in the November 2011 issue]. But what if there were a way to come up with simpler models that perfectly reflected reality? And what if we had perfect financial data to plug into them?
Incredibly, even under those utterly unrealizable conditions, we'd still get bad predictions from models.
The reason is that current methods used to “calibrate” models often render them inaccurate.
That's what Jonathan Carter stumbled on in his study of geophysical models. Carter wanted to observe what happens to models when they're slightly flawed--that is, when they don't get the physics just right. But doing so required having a perfect model to establish a baseline. So Carter set up a model that described the conditions of a hypothetical oil field, and simply declared the model to perfectly represent what would happen in that field--since the field was hypothetical, he could take the physics to be whatever the model said it was. Then he had his perfect model generate three years of data of what would happen. This data then represented perfect data. So far so good.
The next step was "calibrating" the model. Almost all models have parameters that have to be adjusted to make a model applicable to the specific conditions to which it's being applied--the spring constant in Hooke's law, for example, or the resistance in an electrical circuit. Calibrating a complex model for which parameters can't be directly measured usually involves taking historical data, and, enlisting various computational techniques, adjusting the parameters so that the model would have "predicted" that historical data. At that point the model is considered calibrated, and should predict in theory what will happen going forward.
Carter had initially used arbitrary parameters in his perfect model to generate perfect data, but now, in order to assess his model in a realistic way, he threw those parameters out and used standard calibration techniques to match his perfect model to his perfect data. It was supposed to be a formality--he assumed, reasonably, that the process would simply produce the same parameters that had been used to produce the data in the first place. But it didn't. It turned out that there were many different sets of parameters that seemed to fit the historical data. And that made sense, he realized--given a mathematical expression with many terms and parameters in it, and thus many different ways to add up to the same single result, you'd expect there to be different ways to tweak the parameters so that they can produce similar sets of data over some limited time period.
The problem, of course, is that while these different versions of the model might all match the historical data, they would in general generate different predictions going forward--and sure enough, his calibrated model produced terrible predictions compared to the "reality" originally generated by the perfect model. Calibration--a standard procedure used by all modelers in all fields, including finance--had rendered a perfect model seriously flawed. Though taken aback, he continued his study, and found that having even tiny flaws in the model or the historical data made the situation far worse. "As far as I can tell, you'd have exactly the same situation with any model that has to be calibrated," says Carter.
That financial models are plagued by calibration problems is no surprise to Wilmott--he notes that it has become routine for modelers in finance to simply keep recalibrating their models over and over again as the models continue to turn out bad predictions. "When you have to keep recalibrating a model, something is wrong with it," he says. "If you had to readjust the constant in Newton's law of gravity every time you got out of bed in the morning in order for it to agree with your scale, it wouldn't be much of a law But in finance they just keep on recalibrating and pretending that the models work."
See what we're tweeting about
98 Comments
Add CommentComing at this issue from a background in theoretical math and computer science, I am surprised this has not been generally known for a long time. It is generally known that any finite set of data points can be perfectly modeled by an infinite number of different equations, each of which will yield a different "prediction" of the next element in the set. By creating models that require calibration, the modelers are engaging in equation fitting, not knowing which of the infinite number of equations that they have chosen or understanding its true relevence to the fitted data .
Reply | Report Abuse | Link to thisHmmm no wonder all those hurricanes they predicted just
Reply | Report Abuse | Link to thisdidn't happen. I think I'll just stick with the farmers
almanac...hahahha
Agreed!!!! The only thing that you know for a fact when you build a run a computer model is that the results are likely only one flawed version of reality.
Reply | Report Abuse | Link to thisMy first question when I see the results of a model (whether it is one of my own or someone else's) is: what is the reconstructive error and what data did you use to calibrate the model? The error at least indicates that the result is mathematically stable (albeit not proof of a 'correct' result) and the calibration data, or ground truth, tells me if the model is providing a result that is valid.
No matter what the application (economics, geology/geophysics, climate modelling, etc) a simple axiom applies to computer models: garbage in, garbage out.
No kidding!! I was thinking the exact same thing when I read this article. The real secret is that the model referenced in this article is EXACTLY the same type of n-dimensional model that is used for climate modelling (albeit with different physics employed).
Reply | Report Abuse | Link to thisHistory matching is the key to validating the base model and continued matching going forward is the only way to validate the results. Therefore, the realistic assessment of risk in the forward model MUST be determined as accurately as possible and considered in any actions that are taken based upon the forward projection.
To validate a model one should perform a sensitivity analysis by changing variables. A model that is sensitive to key variables is not stable and should be rejected.
Reply | Report Abuse | Link to thisWhere can we see the original study?
Reply | Report Abuse | Link to this@klkeegan: In modeling you specify the functional form of the model before you calibrate. You only have trouble then if you have too many free parameters for the data you collected.
@MadScientist72: I'd think the pool of possible parameters is too large or even infinite. Also, model specifications tend to change over time so it probably wouldn't be feasible.
The problem of economic forecasting is more basic than the article's mathematical modeling issues.
Reply | Report Abuse | Link to thisWho predicted the iPhone and when it would arrive?
Where did you see what sort of n-dimensional model they used?
Reply | Report Abuse | Link to thisI wrote this 20yrs ago, seems it still applies now..
Reply | Report Abuse | Link to thisThe variables are so multifarious, we will never fully
understand everything about anything....
They are also the reason why long range weather
forecasting and computer modelling/simulation
are mostly so inaccurate.
i would also like to see the original study... at any rate, i would like to know more about how the author calculated errors for his parameters taken from calibration. also, when you say that the model failed to make predictions accurately, how far from the prediction is reality? if the errors on the parameters were done correctly, one would expect to make accurate predictions for the future (albeit imprecise ones)
Reply | Report Abuse | Link to this@RDH that question is easy. steve jobs predicted the iPhone and announced when it would arrive to market and that it would be wildly successful... and he was right! WOW!
yes, you're right. i'd really like to know if the study author did this correctly... or if economists know to do this, for that matter
Reply | Report Abuse | Link to this"...I am surprised this has not been generally known for a long time"
Reply | Report Abuse | Link to thisIt has of course been known for a very long time. Half the practice of econometrics is about how models are specified and calibrated, and the problems of underspecification and number of variables are among the fundamental questions in the discipline.
Economists (despite appearances) aren't idiots; at least the good ones aren't. They are dealing with some tough questions but those with the appropriate humility, whose focus is on solving problems rather than writing op-eds, know how tough the questions are and put caveats on their answers.
I actually just know the models are n-dimension which means they have a number of interacting parameters that govern them in four dimensional space. The n-dimension come into each of the grid cells that are used to represent reality with a discrete element. In the reservoir model (as discussed here) it is underground formations in the climate models it is volumes of atmosphere.
Reply | Report Abuse | Link to thisAdditionally, there is the issue with upscaling (so that a computer can run the simluation in some sort of reasonable time) which further diminishes the accuracy of the results.
All models have interacting parameters that govern the behavior of the response variable. That's what makes them models.
Reply | Report Abuse | Link to thisSome of the other comments here are on the right track. How well a model predicts depends on how well it conforms to reality. For example, if you have a function which is by nature a Gaussian distribution, you can model it with a summation of cosines. However, a change in the system will not be correctly predicted by the sum of cosines function, but should be if the correct Gaussian is used.
Reply | Report Abuse | Link to thisSo if a model is not working, then it likely does not conform well with the underling reality. Science is admitting when you have got it wrong and looking for a better solution. If you are not doing that, you are not a scientist.
You've all heard of smoke & mirrors? Well, this is the smoke.
Reply | Report Abuse | Link to thisIrrespective of your chosen set of parameters, a model will only tell you what you may expect given your inputs - which are generally chosen within approximately two standard deviations of historical figures. What a model doesn't tell you is how that information is going to be used.
The economic modelling data was entirely ignored when it showed that devastation fast approaching.
The economic collapse which we suffered in 2008, and many of us continue to suffer, was caused primarily by rating agencies Standard & Poors, Moody's, etc. This would be together with the banks that packaged the toxic $hit, which they then - with the blessings of a triple A credit rating - foisted on the unsuspecting. Then S&P has the balls to drop the U.S. rating?
Now they've gotten you to discuss mathematical models as though they were somehow responsible for our economic collapse!
Try not to be so narrowly focused.
Out of curiosity, could you run multiple models and average the results for a more accurate prediction?
Reply | Report Abuse | Link to thisToday's economic collapse was predicted thousands of years ago.
Reply | Report Abuse | Link to thisEconomic forecasts are even less reliable than physical ones: people are aware of the model's predictions and attempt to benefit from them. Even if the model was 100% accurate when the forecast was made, the most influential players in the economy are quite aware of these predictions. That knowledge will change their behavior as they try to maximize their income beyond what the model predicts.
Reply | Report Abuse | Link to thisPhysical models of global climate change can also be affected by human actions -- the massive injection of carbon dioxide into the atmosphere, for example. But given basic physics and the huge amounts of CO2 that we're emitting, the prediction that things will get a lot hotter is not very difficult to make. Exactly how that temperature increase will affect weather is harder to nail down, but the basic physics is pretty straightforward.
Don't economic models have an additional fundamental flaw - to the degree they are successful, the people that use the models then adjust their own behavior to take advantage of of the knowledge provided by the model. If this occurs at a large enough scale, the model will induce enough behavior change to render the model invalid (since the model does not model the changed behavior).
Reply | Report Abuse | Link to this@ richieo "The only true wisdom is in knowing you know nothing." - Socrates
Reply | Report Abuse | Link to thisDid anyone mention fractal? Since the rhythm of our heartbeats can be expressed in fractal, and our blood vessels are also following a fractal equation similar to most biological beings... can anyone test whether our financial decision making and investment/social behaviours are also following a fractal?
Reply | Report Abuse | Link to thisAs an industrial researcher, it is obvious to me what went wrong here. There was no theoretical basis for his model - he just empirically fit a bajillion different formulae to the data.
Reply | Report Abuse | Link to thisIn real life, what we do is start with an understanding of the process we are modeling, based on engineering or first principles. This greatly reduces the number of models that could be selected down to a manageable few, of which you can empirically determine the best fit.
In the many cases where we don't have such knowledge, we approximate the model as follows for three factors, where lower case is a constant and upper case is the variable:
Output = aA + bB + cC + dAB + eAC + fBC + gABC + error
This strongly restricts the model, obviously, and then we can empirically (statistically) determine which of those constants are discernible from zero. It is not expected to be an exact model, but it allows us to determine which factor(s)(A, B, or C) control the output, and thus are important to control in the process.
This is an example of how an approximate model can outperform one found by selecting "ideal" ones from an effectively infinite set of models. That model above could be "wrong," but still useful (to paraphrase George Box).
Oh - addendum...
Reply | Report Abuse | Link to thisAnd this is why climate models are pretty good predictors of reality (and global warming). They are not just randomly fit numerical models, they are based on physical principles, thus usefully constraining the formula-space to ones that adequately model reality.
There was a lot of wrong doing by people who should have known better. The US government put its trust in institutions who betrayed this trust. It was not the institutions, it was the corrupt people who ran them.
Reply | Report Abuse | Link to thisTake AAA credit ratings for valueless bonds, this was worse than a ponzi scheme. No one has been punished. The ratings agencies were given immunity from prosecution. The only thing that will turn things around is prosecution of those who did the evil, not doing so encourages others to continue.
Like in Libya, a dead Gadafi though it was an extra-judicial killing, satisfies the masses, and brings stability. The US needs to hold people to account before the people take matters into their own hands.
Justice must be seen to have been done.
all we need to make this work is more handwaveium in the system. "Ignore that man behind the curtain, for I am the Great And Powerful Oz."
Reply | Report Abuse | Link to thisThe difference between what practitioners know, and what the public "Knows", is the curtain.
People with just plain common sense have always known that you can't predict with 100% accuracy what will happen, in any area. But there always seems to be some people who never seem to get that, and they will try anyway. The trouble comes when they get enough people to believe that their predictions have some potential validity that they bet the farm on them. And they not only wind up losing the farm, but crashing the whole economy with it.
Reply | Report Abuse | Link to thisThen the survivors of that catastrophe enact laws to prevent it from recurring. Those laws stay on the books until that generation passes away, and another foolish generation comes along which repeals those laws because they didn't suffer the consequences of the previous catastrophe and therefore they didn't understand why those laws were enacted, thus enabling the catastrophe to occur again for that generation.
Since humans obstinately refuse to learn from history, what can you do?
It is disingenuous to say that economic models are "always wrong". In fact certain subgroups in the field of economics have historically shown to be far more reliable than others in predicting major economic events.
Reply | Report Abuse | Link to thisIn particular, historically and over the last 80+ years, Monetarists have had a far better record of publicly predicting economic events than have Keynesians.
And Austrians are on record as having an even better record of prediction than Monetarists.
If you consider yourself to be a believer in the Scientific Method, and therefore that the entire worth of a theory is its ability to predict, then you should join with other rational people and expose Keynesian economics for the utter failure that it has almost invariably been, for more than 80 years.
Before you deride me for saying "Keynesian economics" for longer than it has been a formal theory, the fact is that many mainstream economists followed what are essentially Keynesian principles before he formalized them. Among them was Irving Fisher, famous for having publicly proclaimed just how wonderfully the economy was doing, the day before the crash of '29. When in contrast Von Hayek had predicted clear back in February of that year that government monetary policy was going to cause a crash.
That is only one example. They have been both frequent and consistent these 80+ years.
It is disingenuous to say that economic models are "always wrong". In fact certain subgroups in the field of economics have historically shown to be far more reliable than others in predicting major economic events. "Always wrong" may be true in the same sense as in how physics has been "always wrong", but some theories have been far closer to the truth than others.
Reply | Report Abuse | Link to thisIn particular, historically and over the last 80+ years, Monetarists have had a far better record of publicly predicting economic events than have Keynesians.
And Austrians are on record as having an even better record of prediction than Monetarists.
If you consider yourself to be a believer in the Scientific Method, and therefore that the entire worth of a theory is its ability to predict, then you should join with other rational people and expose Keynesian economics for the utter failure that it has almost invariably been, for more than 80 years.
Before you deride me for saying "Keynesian economics" for longer than it has been a formal theory, the fact is that many mainstream economists followed what are essentially Keynesian principles before he formalized them. Among them was Irving Fisher, famous for having publicly proclaimed just how wonderfully the economy was doing, the day before the crash of '29. When in contrast Von Hayek had predicted clear back in February of that year that government monetary policy was going to cause a crash.
That is only one example. They have been both frequent and consistent these 80+ years.
Pardon the duplicate. It was not intentional.
Reply | Report Abuse | Link to thisWhat about those who are selling them on the "sure thing"? Especially when -- like the mortgage and finance companies -- they were supposed to be the "experts" and were saying "Sure! You can afford this!"
Reply | Report Abuse | Link to thisIf the people were relying on the experts, and the experts were (and oh, so they were) lying to them, then who is to blame?
@MadScientist72
Reply | Report Abuse | Link to this"It seems to me that there's a simple solution to this - run all the models that match the historical data then, as you go forward, eliminate those whose predictions don't match the new observations. Eventually, you'll be left with the right model."
Would this competition include models that contain so many special parameters that they are effectively just a long list? "On day 1, the stock price will be £18.32. On day 2, it will be £18.88..." and so on for hundreds of special values. The model that wins will perfectly match all behaviour so far but will still fail tomorrow. However, it's okay because knowing tomorrow's stock price will allow us to create a better model that is still "winning"...
In my undergraduate physics class the teacher showed us this wonderful software that would draw a curve through our data points, and print the formula for it like magic. "Now I'm going to show you why we don't use that feature". He generated a thousand random numbers and got the computer to draw a curve through them. It did it! The formula for the curve was a thing to behold. It essentially contained a copy of all the random numbers encoded in a particular way - the information was preserved perfectly, merely disguised. How good would it be at predicting the next random number? Useless, because we didn't give it that piece of information.
Meanwhile, there's a more fundamental reason why no scientifically determined economic model can ever be usefully predictive to *any* degree in the long run. If it can be scientifically determined, multiple people will discover it and publish it in short order. Then everyone will know it, and will start using it to predict things, and so will change their behaviour accordingly. The model cannot have accounted for its own influence on people's behaviour, and so is instantly wrong as soon as it starts to be used. (If we try to imagine a model of mass human behaviour that takes into account the effect of the humans knowing about the model and using its predictions to guide their behaviour, we are straying far into Godel's territory).
You can build models for the macro behaviour of systems of microscopic particles. But this is because microscopic particles don't read newspapers.
In the recent PNAS paper http://bit.ly/uI1nxG [bit.ly] one can read that prevailing economic models of credit risk assume that price fluctuations form a bell-shaped curve, with very large fluctuations essentially never occurring. But during financial crises, wild fluctuations occur more frequently than these models predict. Authors developed a method to incorporate these fluctuations in their analysis of financial data from 488 publicly traded manufacturing firms for each quarter from 2000–2009. The researchers used multiple types of known calculations to analyze financial data such as the ratio of working capital to total assets, and sales divided by total assets. These data were plugged into multiple ratio calculations to estimate credit risk for the companies. Particular attention was paid to the years 2007–2009, a time of overall financial crisis. According to the authors, the results suggest that even during stock market crashes, the basic dynamics that underlie less volatile periods still govern credit risk. The study revealed that credit risk follows slowly decaying functional form, implying that dangerous credit positions are more likely than is commonly believed. According to the authors, the credit rating approach may help improve the estimation of credit risk, particularly in the event that financial services companies respond slowly to changes in corporate credit quality.
Reply | Report Abuse | Link to thisActually, it is a widely understood problem in geophysics, hydrology and elsewhere. In hydrology and environmental sciences, we tend to use the buzzword "equifinality" for this (plenty of literature on this). In practice, using prior information for calibration (i.e., imposing extra constraints other than historical data) seems to be the most practical way of overcoming this problem. For example, you may restrict your parameters to lie within some bounds (from other experiments people have carried out), or you may choose solution that have a certain characteristic (the smoothest variation in temporal signal, for example). Or there may be parameter combinations that need to be avoided for the model to be stable.
Reply | Report Abuse | Link to thisThe biggest issue with these things is how to arrive at this prior information, how to quantify it in a sensible manner, and how to combine it sensibly with the observations.
Even in the 'perfect' Newtonian world, once you have 3 bodies you can't perfectly predict the future. Doesn't mean you can't predict fairly far with the right calculation method. Models are only useful for their predictive power. Usually all you can get is a range of possible outcomes as the future is inherently unpredictable beyond very simple systems. Given that I still predict several hurricanes and many tornadoes will hit the U.S. next year. Their actual path? I have no idea except in the same general area as before.
Reply | Report Abuse | Link to thisOne should be careful with the conclusions presented in this article regarding forecasting models.
Reply | Report Abuse | Link to thisIn my experience, there are two basic facts regarding statistical modeling: first, there is not a single model that can explain perfectly a set of observations and, at the same time, predict with precision future observations, and second, you can produce good predictions with simple models.
The problem of explaining a given set of observations with a mathematical model is in the core of most science and engineering endeavors today. The model is commonly defined in terms of parameters (numbers) that are chosen using a matching criterion to produce the best explanation among all possibilities available to the modeler.
To build such a model one needs to answer two questions:
1)How am I going to use the observations to compute those parameters?
2)Which fitness criterion am I going to select to compare different models?
The first question is extremely relevant. If I use all my data to compute the model (this stage of the process is called “training” or “statistical learning”), it is possible that I can find a model that produces values as close to the original observations as I need. However, would have not even one observation that I could use to test the predicting capabilities of the model (this second stage of the process is called “validation”), so I cannot conclude that my model produces in fact the best explanation of the data.
Instead, on should divide the data in two subsets: a training set for model computation, and a validation set, for generalization study, and compute the fitness of the model to both sets. Only then, I have all the ingredients needed to select a model that explains both the training data and generalizes well.
The second question relates with the criterion used to compare models. The most widely used is called the Minimum Mean Squared Error rule. This criterion chooses those parameters that produce the smallest average squared error between the data produced by a given model and the real data, among all possible parameter choices. It produces a model with errors distributing uniformly among the data.
However, it is not the only choice that we should use. Maybe we would like to build a model that is simple enough and runs fasts. Or maybe we would like to add some additional constraints such as a maximum risk level or a minimum expected loss. In any case, there are many more choices available today than can be used in an increasing variety of fields that can produce great results in practice.
How was this not 'discovered' ages ago when people tired to verify there calibration methods? Out Of Sample data anyone?
Reply | Report Abuse | Link to thisI work with Newtonian physics simulation every day and my models do match very well and predict things going forward but it goes without saying that things drift apart over time, this is inevitable even between pure software models.
This does not make models wrong or useless (climate models anyone...) it just means you have to know what you are looking at and interpret accordingly.
The economy, like the weather, is almost surely a chaotic system. Has everyone forgotten about chaos theory already?
Reply | Report Abuse | Link to thisIn computer science this is commonly known as model bias, you will get poor results if you just assume your regression is valid without any validation. Modern financial models include probabilistic measures that can overcome model bias.
Reply | Report Abuse | Link to thisScientific American should establish a prize for lossless compression of econometric data, similar to the Hutter Prize for Lossless Compression of Human Knowledge:
Reply | Report Abuse | Link to thishttp://prize.hutter1.net
That is the only rational way to deal with the over-fitting problem at this point.
Mathematics can be used for rigorous prediction only if the formula can be shown to result from a deeper law. For example, Kepler's Laws can be derived from Newton's formulas for gravity and motion. Buckling of a beam can be predicted from an elaboration of Hooke's Law. With econometrics, we have mostly empirical formulas or theoretical models that only loosely fit reality. This is why catastrophe theory has been mostly forgotten - none of the models it was applied to had any deep theoretical basis. Fractals are useful for modeling textures and describing distributions but also don't have any deep theoretical roots.
Reply | Report Abuse | Link to thisEconomic models are helpful when applied in context. It would seem to me, we are in dire need of a model that doesn't discount pervasive wrongdoings. While you're not going to predict the future, given the appropriate amount of sane data; you can discover weak areas. That is, areas of depletion, which like a tree, can be trimmed or treated.
Reply | Report Abuse | Link to thisI'd be willing to bet that our economy is in a weakening state, because, the roots of our economy are also a major source of depletion. It isn't simply a branch of our economy that is reducing the whole--it is the root system overall. Economic progress is impeded by an abundance of ambiguous laws, regulations, and enforcement. Likewise, it is also impeded by a lack of definitive laws, regulations, and enforcement.
If people were open about their deeds, we could have a model that somewhat accurately reflected the state of an economy. Maybe, although still unlikely, we could better predict the future based on the model's inputs and outputs. Until then, we'll be all hands on deck trying to identify and undo the spaghetti-bowl of depleting factors in play today. That is, before they manifest into unmitigated disasters, such as the financial crises, of the past and present.
Exactly the same critique should be applied to climate models - GCM: general circulation models
Reply | Report Abuse | Link to thiswhich are most curve fitting parametrizations with very large uncertainties in the measured input values.
Yet GCMs are given a free pass.
Economic models will always be wrong because humans use economic models to predict the future.
Reply | Report Abuse | Link to thisThis creates inherently unstable systems which cannot be predicted.
Climatology and economic modelling are actually very analogous disciplines.
Reply | Report Abuse | Link to thisEven though we think of climatology as a hard science and economics as a social science, they both engage in copious amounts of data collection, modelling, and prediction.
No. GCMs are mostly curve fitting parametrizations with inputs that have very large statistical and systematic uncertainties.
Reply | Report Abuse | Link to thisHere is an excellent video and paper from MIT that follows the general theme and provides additional specific examples.
Reply | Report Abuse | Link to thisVIDEO: "Warning: Physics Envy May be Hazardous to Your Wealth"
mitworld.mit.edu/video/794
ACADEMIC PAPER: "Warning: Physics Envy May be Hazardous to Your Wealth!" (See "one-click download" at top)
papers.ssrn.com/sol3/papers.cfm?abstract_id=1563882
Excellent examples throughout. My personal favorite is the coin-tossing machine by Diaconis, Holmes, and Montgomery (2007). Coin tossing is random, right? WRONG! Their carefully adjusted machine tosses a coin that lands heads up, 100% of the time. It's an important lesson on checking your assumptions at the door.
ARTICLE: "Overheard at MIT: Why Economics Isn't Like Physics"
sloanreview.mit.edu/the-magazine/2010-fall/52113/why-economics-isnt-like-physics/
I'm surprised with how pop-culture-ish this article is. A geophysics master's student has discovered that his model calibration doesn't work. The argument is that this fact is supposed to uncover a deep dark truth about modeling economic and other types phenomena.
Reply | Report Abuse | Link to thisSomeone needs a modeling class, including the writer and editor of this article. Maybe start with overfitting...
Here is an excellent video and paper from MIT that follows the general theme and provides additional specific examples.
Reply | Report Abuse | Link to thisVIDEO: "Warning: Physics Envy May be Hazardous to Your Wealth"
mitworld.mit.edu/video/794
ACADEMIC PAPER: "Warning: Physics Envy May be Hazardous to Your Wealth!" (See "one-click download" at top)
papers.ssrn.com/sol3/papers.cfm?abstract_id=1563882
Excellent examples throughout. My personal favorite is the coin-tossing machine by Diaconis, Holmes, and Montgomery (2007). Coin tossing is random, right? WRONG! Their carefully adjusted machine tosses a coin that lands heads up, 100% of the time. It's an important lesson on checking your assumptions at the door.
ARTICLE: "Overheard at MIT: Why Economics Isn't Like Physics"
sloanreview.mit.edu/the-magazine/2010-fall/52113/why-economics-isnt-like-physics/
There is a basic problem with Economic Modeling.
Reply | Report Abuse | Link to thisIt isn't that people might use the results, and that can't be modeled. This kind of thing is done in engineering all the time. It is called feedback. It is relatively easy to model. Nearly all industrial control systems use it. It is feedback that makes control of the process possible.
No, with Economics, the basic problem is that there is no real firm foundation.
It is always easy to find an Economics professor to agree with any proposed political action, and to find one to explain any past economic situation in a way that will support any of a broad range of political or societal actions. This is because economics is a 'soft' science. Opinions matter more than experimentally verified facts.
In the Physical sciences, by contrast, it is very hard to find a Chemistry Professor, say, who will give any hope for not having combustion of carbon and oxygen yielding water as a byproduct, no matter how important politically that result is. It is similarly hard to find a Physics professor who will support repealing the law of Gravity, or supporting any political demand that like charges attract.
There is also a problem with using models that are not really well understood. Any computer model relies on a set of assumptions. These assumptions are either valid or invalid. If valid, they are valid only over a restricted range. What that range is must be established before the model can be considered as valid.
continued...
Reply | Report Abuse | Link to thisThis means that ALL computer models have some range of conditions where they produce invalid (wrong) results.
One good example of this is the famous 'hocky stick' graph that was the basis for much of the last ten years of 'Global Warming' excitement. The first time I looked at the graph I noted "Oh, a divide by zero error". The sharp rise with no flattening gives it away. I made a similar error in my undergraduate engineering education. It can be very hard to find in a complex program, but, when one variable goes close to zero in the denominator, then the total value being modeled rises very high, very fast.
Computers are a very useful tool, but we should remember that they are just a tool. Computers don't replace experience. I have seen in many engineering situations where use of a computer without thorough understanding of the process gave nonsense answers.
Just be glad you aren't driving over a bridge that was designed using a computer with no input from an experienced Structural Engineer.
There have also been several blackouts that were caused by computers controlling electrical power plants that went beyond their validity ranges.
It isn't just Economics, it is also Climate Research, Engineering, Cosmology and even occasionally industrial production. There have been more than one pipeline rupture because of improper control algorithms.
1995 Economics Nobel Prize winner Robert Lucas wrote an influential paper in 1976 about the inability to calibrate economic predictive models. The concept, known as the "Lucas Critique," even has a wikipedia page. http://en.wikipedia.org/wiki/Lucas_critique
Reply | Report Abuse | Link to thisThe basic problem is that over the historical calibration period, an economic policy is in effect, and so the predictive results are biased by that policy. Any change to existing policy going forward is not calibrated to the historical model.
Economists correct for this by using DSGE models, dynamic stochastic general equilibrium models. These models are much more accurate because they attempt to capture expectations, future behavior, of economic agents. Thomas Sargent won the recent Economics Nobel prize for work in this field, "Rational Expectations."
Rational Expectations implies that individuals will maximize their own welfare and undo the adverse impacts of the negative effects of government policies. So, as the Fed has substantially increased the money supply, which is potentially inflationary, to boost the economy, the consumer, during this recession, has decreased the velocity of money, the money multiplier, to limit inflationary effects of the Fed's policies. Obama and Congress have attempted to stimulate the economy using government debt, so the consumer has been aggressively deleveraging, paying down debt, using credit cards much less, etc., to lower the total debt of individuals and government.
Rational Expectations implies that external, unanticipated events, will affect economic results. Economic models do not give one predictive number, as one would think from the media. The models give a range with probabilities. Too often, we only hear about the most likely number without what its probability or confidence band is, or what the range of possible values is. Even then, the world is uncertain, unexpected and strange things happen and not everything that can affect economic levels is foreseeable.
Actually Typhoon, you are quite wrong. Not my area of expertise, understand, but if you are unaware of things such as insolation rates, atmospheric insulation (even at the level of black body radiation if nothing more complicated than that) fluid dynamics, La Nina, El Nino, blah blah blah, you are not informed enough to comment knowledgeably on the subject. All of these physical constraints are accounted for in modern climate models. Their predictive efficiency is quite high at the climate scale.
Reply | Report Abuse | Link to thisEven if they weren't, there are very numerous other direct and indirect observations that all accord with the theory, so you can't just dismiss AGW by dismissing one of the many supporting threads.
Sounds like you are describing climate "science" and all their perfect models sans predictions.
Reply | Report Abuse | Link to thisIt is well known in trading that one can describe any period of stock prices perfectly by fitting the data and that model will have zero predictive ability (or the same as any random result). It looks great though!
Reply | Report Abuse | Link to thisThis is very well known.
Reply | Report Abuse | Link to thisThere have been a million papers in economics and finance that discuss the related problems of identification and issues related to consistent estimation. A million papers...
Identification and estimation are not problems in all models but they are problems in some.
This is a silly article pointing out a problem that has been understood for a century or more.
I'll bet that Micheal Mann could get a hockey stick from the financial models, too.
Reply | Report Abuse | Link to this.
It seems strange that the article does not mention the climate models that have so been the focus of attention in recent years.
Reply | Report Abuse | Link to thisThey are likely to be as spurious as the economic models discussed.
Important to bring together several threads.
Reply | Report Abuse | Link to thisFirst, any financial model is merely a subset of the market, just as a map is a less detailed version of reality. Useful for broad navigation but not a proper substitute for the real thing. Especially as regards policymaking.
For example, the most primitive model says “One in twelve loans will default, so commercial banks must hold 8% capital…” Up a step, the next model says “For these rating classes of loans, commercial banks will hold the following percentages of principal as capital…” (Basel 2). And today the models say “Do the following scenario tests and hold the following…” (Basel 3).
Second, no financial model can anticipate, much less deter, human manipulation. “Mortgages need 8% capital? But I can repackage the same as a security and only hold 4%...!?”
Third, every model faces over-determined equations. There is no analytic solution to market clearing prices; it’s worse than Navier-Stokes. Consider dollar-yen exchange rates. There “must” be a unique and mandatory one-year forward rate, based solely on the difference between US and Japanese one-year interest rates. That is, if the US interest rate is 3% higher than the Japanese rate, then the forward exchange rate “must” be 3% lower than the current exchange rate. Except not. I have a thirty year history of dollar-yen, and the implied forward almost NEVER hits the realized future exchange rate; it’s perfectly random. Why? Because of exogenous shocks, balance of trade, political shifts…
Thus the parlous state of modeling and the concomitant need for modesty among those addicted to central planning.
The market is its own, best one-to-one simulation, and the unbeatably best allocator of information.
Great article. The “modeling” as far as Global Warming is concerned has never had a sound mathematical basis, nor does it stand up to rudimentary applications of common sense. Take the idea that manmade CO2 will drastically change the atmosphere and cause the entire planet to heat up and become virtually unlivable. CO2 makes up about 400 parts per million of the gases in our atmosphere. Man accounts for about 20% of that 400, about 80 parts per million. 80 parts per million equates to 8 parts per 100,000.
Reply | Report Abuse | Link to thisIf the gases in our atmosphere were a $100,000 investment portfolio, and manmade CO2 were a stock in that portfolio, no investment expert would say, “You’d better watch that one share of $8 stock like a hawk because your entire retirement hinges on the value of that one share.” But when it comes to Global Warming that is essentially what the so-called experts are saying; basically, that the one $8 share of stock is all that matters. The rest of your $100,000 portfolio (all other gases in the atmosphere and everything else that interacts or impacts it) is irrelevant to your retirement nest egg. Does that come close to passing any type of proportionality test or even the “Sniff Test”?
To put it another way: if you were starting out on a 10,000 mile road trip—from New York City to Los Angeles to Miami to Minneapolis to Dallas and back to New York City—no one would think they had gone anywhere after the first .8 miles—the distance from your house to the main road. Yet, that same ratio .8 parts per 10,000 is what manmade CO2 contributes to the gases in the atmosphere.
The fact that the “Theory of Global Warming” is based heavily on modeling and that this article shows how terribly unreliable models can be, combined with the basic facts not coming anywhere near to passing any type of reasonability test should cause any serious person to reject any “conclusions” the Global Warming “experts” derive.
Truly excellent article. The only way it could be improved would be to change the title to " Why Economic Models (and Climate Change Models) Are Always Wrong".
Reply | Report Abuse | Link to thisNo one should look at the predictions of climate change models as anything more than speculation, hypotheses on how the climate works, not predictions. Spending money on counteracting the a model's predicted effects of climate change is as mad as investing money based on an economic model of the economy. Both systems are poorly understood, and dynamic.
Some models work extremely well, with their validated bounds. Complex models should not be relied on for all the reasons above. The particular attack on global warming seems odd though.
Reply | Report Abuse | Link to thisYes, the global warming debate needs to hold to the predictions of global climate models lightly, for all the reasons stated above. However, the global warming models were prompted by considerations involving relatively simple physics and chemistry. As of yet, I do not know of anyone who has come up with a mechanism to undermine the idea that infra-red radiation from earth is effectively trapped by CO2 and CH4, and this should (all other things being equal) warm up the troposphere. Combined with disappearing islands and definite rises in global temperatures, this should at least produce a little pause for thought. Simple physics, simple early warnings. The complex models are not to be trusted for details, but they generally converge on the bulk predictions. To write them off as totally irrelevant, and dismiss all the other data seems cavalier. Perhaps those people would not investigate the intruder noises downstairs at night, because the burglar alarms are all so unreliable.
Hi,
Reply | Report Abuse | Link to thisIt is even more than given a finite number of data points one can find an infinite number of equations to fit them: economic behavior like weather is non-linear so that no one can find one equation to model the system. Do you remember your non-linear differential equations, fractal geometry and chaos theory? The corollary: a non-linear system being just a little off on an input can produce a massive output change like the butterfly flapping its wings off the coast of Australia and causing a hurricane off the coast of Florida.
I believe that the original paper can be found at: http://www3.imperial.ac.uk/pls/portallive/docs/1/39282.PDF
Reply | Report Abuse | Link to this... note that it is a PDF file.
Many comments here have missed the point of the original paper.
The paper sought to test the predictive value of computer models used to predict the future production of an oil field based on models of the using physical measurements of three factors in oil field production They arbitrarily defined one model as representing the "real" production of a field and used it to generate the data for subsequent models. They then created multiple data sets based on the margin of error within the measurements themselves. They used those slightly varying data sets to generate other models of the "real" oil field. They then matched the models against the "historical" production of the "real" field and its "future" production.
What they found was that they had many models that matched the "historical" production but which then deviated significantly from the "future" production. They concluded that just because a computer model fits known data that does not mean it has predictive value otherwise. That is because there are multiple models which will reproduce the historical data but which don't match future reality.
The application to economic or climate modeling is obvious since both use a models degree of match with historical data to argue for the models future predictive power.
While the model might say something about the economists to whom in-the-box observers give their attention, what does it say about economists who could not garner attention but could and did predict the peak and downturn? The UK's Fred Harrison, the US's Fred Foldvary, and Australia's Phil Anderson all used Homer Hoyt's work on the land price cycle to accurately, in print, in advance predict what came to pass and explained why it had to happen (and will repeat until rents are recovered and shared); Anderson in 2006 even advised his subscribers to short bank stocks. A more useful model is one showing the business cycle tagging after the land-price cycle. More at geonomics.
Reply | Report Abuse | Link to thisIt is well known. Look up any course notes on model selection. VC dimension, AIC, BIC, etc. are meant to gauge how well a model will perform outside the training data.
Reply | Report Abuse | Link to thisHow did Paul Wilmott peep in un-introduced at the end of the article?
Reply | Report Abuse | Link to thisLooks like a 2005 paper was used by David Freedman, to make a point about financial models. The paper looks like it is:
Reply | Report Abuse | Link to thisOur Calibrated Model has No Predictive Value: An Example from the Petroleum Industry, J.N. Carter, P.J. Ballester, Z. Tavassoli and P.R. King
http://library.lanl.gov/cgi-bin/getdoc?event=SAMO2004&document=samo04-45.pdf
A line of regression represents the best fit of a data. We can use higher degree equations for better predictions.We should consider the free will of each individual in highly empowered societies like west. That is why east is more predictable than west were people are more docile.
Reply | Report Abuse | Link to thisThe type of model used in the fitting is very important. If you are just fitting measured data to a high-order polynomial, regardless of the underlying physics will almost always guarantee failure. It sounds like he used the same equation that "matched the physics" and attempted to re-derive the coeff's. Since he used a randomly selected equation, was the inverse well-posed? If not the estimation process will amplify any errors int the estimate. How many parameters was he fitting and how many data points was he using? Did he insure each data point was independent of the other's? If he was sampling in time, did he analyze the system "bandwidth" and sample no-faster than Nyquist? There are many ways a model-fitting parameter estimation problem can go bad, that is not fundamental, but procedural.
Reply | Report Abuse | Link to thisI agree with klkeegan, this has been known for a while.
Reply | Report Abuse | Link to thisBut I think there is a further problem, which others might have mentioned (I haven't read all comments), namely the problem of modelling non-ergodic systems. Carter's "perfect" model seems to be ergodic, i.e. all the parameters are the same through time. Economists model human systems, which are non-ergodic i.e. the nature of their behaviour is continuously changing. Human systems are a type of complex system because there are multiple, heterogenous agents interacting with and adapting to each other. They are non-ergodic. Related to this, uncertainty prevails - the future is inherently unknowable.
Paul Davidson's essay on INET's website is excellent on this subject in the context of economics - http://ineteconomics.org/blog/inet/paul-davidson-response-john-kay.
Obviously my point obviously does not counter Carter's work, it is complementary to it, and demonstrates that orthodox economists are making mistakes in multiple dimensions. In terms of trying to anticipate future asset values, it also means that we cannot identify some knowable, future distribution of asset prices.
Greg Fisher, Managing Director, Synthesis IPS (a think tank devoted to taking Complexity theory in to policy).
This is probably Jonathan Carter et. al. paper
Reply | Report Abuse | Link to thishttp://www3.imperial.ac.uk/pls/portallive/docs/1/39282.PDF
Remember the Moon landing? There were 200,000,000 Americans that year. In 1995 there were 200,000,000 cars in the United States. Can our economists tell us how much Americans have lost on the depreciation of automobiles since 1969 or 1995? No, they do not compute or track that information even though John Kenneth Galbraith wrote about the planned obsolescence of automobiles 10 years before the Moon landing.
Reply | Report Abuse | Link to thisThe whole system is FUBAR! Are we supposed to believe scientists have no idea economists are ignoring Demand Side Depreciation, world wide? Can scientists think their way out of a paper bag? Is everyone locked into their own tunnel vision of overspecialization?
How about mandatory accounting in the schools? Double-entry accounting is 700 years old people. We have cheap computers everywhere. How hard can the accounting be?
The Accounting Game : Basic Accounting Fresh from the Lemonade Stand
http://www.exceltip.com/book-1570713960.html
If it had been mandatory since 1969 we might not have this mess.
I think that many responses are contributing from several complimentary angles. With a physical model, in well known scenarios, there should be a good understanding of shape of the equations, reasoning from the fundamental laws and know simplifications of such given the boundary conditions. It should be then possible to test for -
Reply | Report Abuse | Link to thisNon-linear effects
Linear sensitivities to variable change
Sensitivity to variables that have been ignored as have negligible effect through perturbation or otherwise
Sample rate effects, parameter dimension effects, and all the other good statistical considerations and analysis many learned contributors have mentioned.
This presumes that the equations can and are well defined, understood and known.
This seems to mean that future predictive power is reasonable, over relatively short projections. The futher one projects, the greater the potential deviation.
In the case of economics it would appear that the changeablitity of human behaviour puts all of the equations used in doubt. Different equations seem to hold for different markets and historical systems. In addition, no physical law will observe the scientists summation of it's behaviour and then change in response. Market players always do this.
If one adds to this the possibility that not all the parameters are considered in the first place, and there are serious doubts as to whether all the paramenters could ever be found, and that some behaviour maybe outside of the scope of analysis mathematically, the case seems hopeless.
Given all of the above, how do any predictive econimic models ever work? Even in the 40's and 50's there were successful analogue computation models of the UK economy, used to good effect, namely the MONIAC, built by Bill PHillips and adopted by the LSE and other UK universities. http://en.wikipedia.org/wiki/MONIAC_Computer
Climate models would seem to fall inbetween, theoretically completely knowable due to their physical, inanimate nature, but practically intractable, due to their high level of complexity and non-linearities.
Any suggestions?
The late John N. Warfield developed a useful 'theory of modeling', which ensures that the models we make would be rather more reliable than those indicated in the article. See - "What is modeling?", which I shall post in parts here, after I send this in (as my message would exceeds the limits of what SciAm allows if I include it here).
Reply | Report Abuse | Link to thisA good bit of 'scientific modeling theory' may need to be rethought/revised in the light of Warfield's seminal contributions to systems science. More information about Warfield's work is available at http://www.jnwarfield.com and from the "John N. Warfield Collection" held at the library of George Mason University, where Warfield was Emeritus Professor - check out: http://ead.lib.virginia.edu/vivaead/published/gmu/vifgm00008.tp
GSC
What is modeling?
Reply | Report Abuse | Link to thisThe 'Structural Modeling' Approach – and how it is significantly different from any conventional approach
First, a quote, slightly paraphrased, from John N. Warfield:
Modeling is a process that begins with human perception. A sequence of the following nature describes the activity of modeling:
1) Perception
2) Storage in the brain
3) Identifying a context within which to place the perceptions, and within which they can potentially be integrated
4) Generating factors associated with that context and with the perceptions that are the focus of attention at the time
5) Identifying types of relations that appear to be associated with these factors in the chosen context
6) Structuring the factors to show how they are interrelated through specific relationships that are representative of the selected types
7) Interpreting the structures produced
8) Associating the factors with numerical algorithms that permit the relationships discovered to be quantified (if they are possible to quantify)
9) Assigning or computing numerical values to/for the factors
10) Interpreting the model-related information for purposes of design or decision-making
(Above paraphrased from “Structural Thinking”, J.N. Warfield: 1995-96 Essays on Complexity)
The above sequence describes Structural Modeling, the process underlying Interactive Management (and the One Page Management System - OPMS). Built into the above-outlined Structural Modeling process, when IM or OPMS is used, is an ongoing comparison of model-related information at each stage with the reality on the ground. These comparisons become sharper and more focussed as the models evolve and develop over time.
The conventional way (which the IM or OPMS process would not allow at all) is to start at Step 8 or at Step 9 of the above-outlined modeling sequence.
(Continued, next)...
In fact, most discussions between people not using IM/OPMS start out at Step 8 or Step 9, usually leaving out Steps 1 to 7, which are pre-requisite for clear understanding all round! (It is true that there are, on occasion, some context-clarifying remarks made, but these generally lack adequate focus to ensure truly clear understanding all round). Thus, many discussions between people are, in the conventional way, based on sets of ‘mental models’ that are significantly different from each other because of differing backgrounds of the people holding them. These mental models on which different people are basing their discussions are left entirely unclarified. Because of the differences in context, the very same words spoken by different people could often mean significantly different things. In any case, the context is entirely unclear. This leads to non-understanding, misunderstanding, confusion, and, finally, ineffective or incompetent action.
Reply | Report Abuse | Link to thisWe are interested in ensuring effective action at every level in the organisation – starting with the individual. Because discussions in the Structural Modeling process are always based on a significant clarification of the context of each idea and thought contributed to the discussion by each person, subsequent action is much more likely to be effective. (Step 3 of the sequence of Structural Modeling outlined above).
It should be observed that ‘Structural Modeling’ INCLUDES the ‘conventional modeling process’. The conventional ‘numerate models’ (showing numbers, e.g. how much money, how many copies will be sold, and so on - on which most people rely to the near-total exclusion of any structuring activity) will develop, in a natural way, as the structure of the interrelationships of various issues becomes clear. The difference is that the numbers developing through the Structural Modeling approach are based on a detailed consideration of all structural aspects of the issue, and will therefore have far higher reliability than the numbers developed through the conventional approach.
NOTE: Messages 75, 76 and this one, 77, relate to a view of modeling (or 'theory of modeling') than a great many of the models we get to see and use.
Error in "NOTE" of Message No. 77. It should read:
Reply | Report Abuse | Link to thisNOTE: Messages 75, 76 and this one, 77, relate to a view of modeling (or 'theory of modeling') that can give us significantly better results in all complex systems than a great many of the models we get to see and use.
Sorry about the goof-up.
GSC
As far as I can tell -
Reply | Report Abuse | Link to thisAssuming the model of modelling used above, in the Western Sciences, steps 1-2 are innate/cultural, steps 3, 4 and 5 (and some of 6, 7 and 8) would be associated mainly with Greek through to Islamic Science. Generally, they could not make good pattern matches to the natural world on the philosophical base provided. It took medieval theology and philosophy in Europe to get to Bacon et al and then Galileo and Newton via the University of Padua (if I remember correctly). Through this historical process the scientific method was developed, which might be represented by steps 4-10. Most of this context is not considered by Applied Mathematicians or Scientists when using this modelling process, it assumed and absorbed with each sip of post-graduate coffee, in most institutions.
Two questions then arise in my opinion-
How can an effective processes for economics etc. be formulated which mimics the effectiveness of the scientific model away from the hard sciences where they developed?
Has anyone used Warfield's Structural Modelling model to analyse Warfield's Structural Modelling model? After all, it's a model of modellin. Should it simply be assumed? :-)
Many comments seem to confuse curve fitting with dynamic modeling. It is true that curve fitting can suffer from overfitting in the hands of the ignorant or the crooked and that dynamic modeling can suffer from not understanding how to model some physics processes such as condensation (cloud formation). No one complains or criticizes scientists when they model water flow through aquifers or blast and shock effects from untested nuclear weapons; it's the same physics used in climate predictions. Many of the criticisms leveled at climate modeling seem to assume that curve fitting is used. Since it isn't, those criticisms are an apples and oranges issue.
Reply | Report Abuse | Link to thisTo assume that scientists are unaware of the effects of clouds on climate prediction (or weather prediction, which also seem to be confused by many comments) only shows the ignorance of the writer. It is well known that clouds are poorly understood and work is in progress to better understand their formation and impact. To say that "clouds control 60 percent of the weather" seems strange. How would you know that if you couldn't model it. It would also seem to make the 7 day forecasts totally bogus and undependable.
Other comments seem to indicate that there exists only one model and that all the scientists are such boneheads that they only look at their own models and are too stupid to consider multiple lines of research. Validation is a fairly "straight forward" exercise: take starting values (boundary values) for some past time and try to predict what actually happened for succeeding years. Only the layperson is boneheaded or stupid or just ignorant when it is assumed that scientists are not doing that. They also check the effects of parameter measurement error effects by varying those parameters and rerunning the models.
Others point to the Lorenz effects (using dynamic modeling) and say that a butterfly flapping its wings will drastically (chaotically) effect future predictions. What they forget is that Lorenz also found that, with regard to his work with predicting pendulum positions, he couldn't predict exactly where the pendulum would be, but he could predict the patterns it would follow. That is the difference between predicting weather and climate.
That settles it! I have lost all confidence in Scientific American rigor and objectivity after reading David H. Freedman's load of crap. Jeesh, maybe he could actually try to explain econometrics or its assumptions. Better yet, SciAm should just subject this sensational fluff to review by a couple of profs at any middling university. News Flash from obscure journalist: The 10,000 Ph.D. in the Econ Field are always wrong because they never consider the possibility of multiple local optima in an objective function! Oh, wait, forget it. We always consider that.
Reply | Report Abuse | Link to thisCarter and Freedman have hit on exactly why the global warming models created by the Duckly Luckys and Goosie Lucies of the world are wrong. Bad science, bad data and wrong assumptions always lead to the assumption that the end of the world is near. You can have perfect math and incomplete data. You can match the past data perfectly and fail to predect the future data.
Reply | Report Abuse | Link to thisA few misunderstandings and imprecisions there, if I may:
Reply | Report Abuse | Link to this- first, in the financial crisis, the so called "subprime crisis", the problem was not with the models, but in the lack of oversight. The executives that ruled the companies involved were absolutly aware of what was happening. They knew that the assets they were solding was overvalued. They sold it even knowing that a disaster would ensue. The real risk arose with practises and rewards that come along in the banking industry. The storm came by the fact that defaults (in housing loans) bust in, at the same time, in many different states over the US (see Freefall J. Stiglitz). Not because of wrong models but because of the men that decided... endgame.
- second, mathematical models are valid providing you don't expect too much. Statistical arbitrage worked pretty well, for example. They made a tremendous amount of money with "pairs trading" in the mid 80s at Morgan Stanley, with Tartaglia and his research group. It works pretty well yet, though at a much wider range, in what is known today as algorithmic trading. Hedge funds (and others) are hiring a bunch of mathematicians actually, and money is not a joke for those guys, right? No kidding!
- third, models work, I mean statistical models. Meteorology: massive computers give you a good idea of what will happen within five days (roughly speaking, it depends on the computing power). Insurance: they thrive on the reasonable assumption that not everybody will die at the same time. And so forth.
- fourth, calibration is only a part of the modelling process. One might say that the drift that can result from a little change in calibration can not be worse that the huge discrepancies that occurs in nature with many systems, discrepancies caused by a little disturbance in the past ("sensibilité aux conditions initiales"). One can't talk of flawed models only on that ground, because it has only to do with measure. A good method for measuring is necessary to think about a model, right. But you can't simply reject modelling in general, on that basis: there is a feedback between the calibration and the model, and this is within the modelling process itself. Change your model and your calibration.
- Last but not least, there is no essential difference between making a model about random phenomenons, and trying to explain others that, at only in a certain extent, can be viewed as deterministic (think about quantum mechanics).
More than unsuccessful models, there are unsuccessful mathematicians.
Take care.
AB
The more precise or "real" the model fits the data set the more wildly it will deviate once predictions beyond the data are attempted ( if I remember my theory correctly .-). Anyway - the problem with economics is that there is an incentive ( the profit motive ) to subvert the model, or data set, or any rules/regulations/laws that are promulgated. The whole economic and financial game consists of beating the rules, finding loopholes and/or subtle violations, not following them with any sincerity except for public consumption purposes. The game is NOT to follow the model, but to second it to the profit motive. Never mind that all economic theories are contaminated by politics, obfuscating both model and rules in order to garner votes from an unsophisticated public ! Thus is democracy !
Reply | Report Abuse | Link to thisI think that there is a far simpler explanation to why economies fail. No engineer would ever design a bridge, dam, aircraft or product whose life span is based upon compound interest - the exponential curve, making long-term durability impossible. Spin an exponential curve 360 degrees and you see that it forms a funnel. In economics, the base of the funnel (large diameter) represents 90% of the world’s population and the top (small diameter) 10% who control world finance. Until humans find a way to develop, more stable economies not based upon compound interest, that naturally funnels funds from the poor to the small rich minority, economies will always be top heavy and unstable as it allows 10% of the population control most of the world’s wealth at the expense of the rest..
Reply | Report Abuse | Link to thisI think that there is a far simpler explanation to why economies fail. No engineer would ever design a bridge, dam, aircraft or product whose life span is based upon compound interest - the exponential curve, making long-term durability impossible. Spin an exponential curve 360 degrees and you see that it forms a funnel. In economics, the base of the funnel (large diameter) represents 90% of the world’s population and the top (small diameter) 10% who control world finance. Until humans find a way to develop, more stable economies not based upon compound interest, that naturally funnels funds from the poor to the small rich minority, economies will always be top heavy and unstable as it allows 10% of the population control most of the world’s wealth at the expense of the rest..
Reply | Report Abuse | Link to thisFascinating read. It also points to the unreliability of the climate and global warming models and consequent theories used to justify the AGW claims.
Reply | Report Abuse | Link to thisIt is well known in Engineering that even the best model can predict dependable results only for small changes in the parameters from the data used to validate the model. Economic models can not be as rigorous as those in physics or Engineering. If some one is surprised at the failure of the models he must be either a theoretician or an ignorant.
Reply | Report Abuse | Link to thisEconomy is not science nor art, for years has result into a disaster; developed countries suddenly lost the control, reputed institutions became a mess, the world is at the edge of the Apocalipsis. Nobody is doing something even artificial to keep a healthy world.
Reply | Report Abuse | Link to thisMr. Greenspan has been blamed to do so, however economy was sound and growing. Are we going to pay the waste of resources made in the past? Is mother earth reverting all damages infringed? Although late we are on time to
direct the rational use of natural resources still available.
I'm glad looking people attaching economics to physical laws because I think they are right. Newton in his Mathematical Principles of Natural Philosophy recommended to derive all the phenomena of Nature by the same kind of resoning from mechanical principles and economics should not be different. There is a study related to synergy in which an equation is developed to handle any number of variables since it is a summatory; such equation has applications to wearing, merging, conditioning, projects, finance and sports and is proof using the equation for motion of projectiles developed by Newton. It is very likely that such equation would be able to produce economic models using proper variables properly calibrated. Should you be interested in the study? Try "Synergy and Management" in the web. Author Alejandro Sibaja Aceves.
Reply | Report Abuse | Link to thisYou make assumptions that are basically wrong and then you use wrong assumptions to proove that climate models are wrong.
Reply | Report Abuse | Link to thisCO2 is currently and roughly 400 parts per million of the atmospheric gas. You fail from the start because the greenhouse effect of the 999,600 parts per million is for all intends and purposes zero. In other words nitrogen and oxygen do not cause a greenhouse effect as far as we know.
I do not understand why this simple fallacy keeps cropping up and surviving.
All other deductions and conclusions based on this fallacy are irrelevant and incorrect.
Without CO2 your 100,000 dollar investment would not produce any warming at all. In fact we would be freezing to death because the average temperature would be some 33 degrees Celsius lower than what it is now.
The investment is in the greenhouse gas that we need to survive. It is not in nitrogen and oxygen as far as greenhouse warming is concerned.
Then if we increase the investment in CO2 by adding more to the atmosphere we will run into an imbalance and create potentially serious problems. That may be compared to economics modelling in a very vague way. Specifically and simply if you over-invest in something you stand to loose money. Case closed.
Unfortunately for Scientific American, this problem exists for all its solutions to economic problems, including those relating to Global Climate Change. Scientists may be able to generate models of how the climate will change, but the solutions to the problem require changes to human behaviour. Interventions by government and subsidisation of certain technologies, are economic solutions.
Reply | Report Abuse | Link to thisFinancial models may require calibration, but the model must also have a sound theoretical basis, which is not the case. When scientists advocate certain actions, they fail to realise that all actions are based on individual value judgments and that changes in the financial structure of society will result in different value judgments. The failure to incorporate "value" into models renders the models totally inaccurate.
The consequence of government subsidation of certain schemes results in a waste of taxpayer resources and a destruction of Capital. It is this Capital, which we need to address Climate Change and it is Capital that is being destroyed by scientists and governments colluding to ensure its destruction, by subsidising wasteful economic schemes.
While I agree that on the limited information provided, the "study" discussed in the article appears naive and unconnected to prior work, the comments touch on both real and imagined issues of vital importance today, whether in economics at a time of great apparent instability or in other contentious policy areas (such as climate) or less contentious but still involving major policy decisions (such as what physics experiments are worth heavy federal funding or how to balance energy development with its consequences).
Reply | Report Abuse | Link to thisThe time is ripe for a fat issue of SciAm summarizing ... and predicting ! ... developments in "Living with Prediction and Modeling; Making Sense of the Past and Present and Trying to Anticipate the Future." There is much sincere angst these days about economic crises, how we got here, and whether we can avoid the next one; there is wild debating about climate with IMHO particularly damaging misuse and distortion of the limits of prediction by the "deniers"; there is a bizarre and dangerous political on-going drama about intelligence prediction based on massive data screening (did the US anticipate X, how will it prevent Y in the future) with implications for military and security budgets, constitutional rights, and public trust.
So why not get some sober and thoughtful experts to pull together some of the things alluded to in the comments in this thread: wisdom is not certainty (the history of scientific method and modeling in "hard," "soft," and "squishy" areas of study and analysis; the history of reconsidering and addressing the validity of formulas, methodology, data collection, iterative or looping processes, chaos, scale effects, etc.); knowledge and the use of analyses of knowledge evolves (not a simple straight line, nor just a series of dramatic ahas, nor just chaotic or random); being human means living in an infinite number of feedback loops (how communication and thought, whether by leaders, consumers, patients, doctors, physicists, businesses, demonstrators, armies, or madmen, affects what they will do, efforts to understand what they have already done, and predictions based on what we believe they have done); and the paradox of coexisting certainty and uncertainty is a gift and a curse.
SciAm should include exciting examples from a wide variety of fields. For example, in addition to obvious formula-driven climate and economic issues: do changing histories for human origins alter predictions of violence and the spread of disease; is all politics really local; publicity effects in science.
the economy is a reflection of the human mind, most of us are plugged into this. Non-equlibrium theory tells us that there is a negative and positive feedback between us and the environment. So the economy is like an evolutionary system, with all its inundations and emergences. Maybe Prigogine and Janstsch still have a lot to teach us. using physics as the base line thinking is not going to help us here. We cant tell if the system is going to grow or shrink.
Reply | Report Abuse | Link to thisThere is so many things wrong with this "experiment" and implied conclusions that it's difficult to know where to begin. First, even a novice model creator knows that one should not "fit" data perfectly since such models are notoriously poor predictors. Secondly, the implication that models are not useful because they are "always wrong", mistates the purpose and use of models. Models should be used to help humans make decisions; not make decisions for them. Indeed, those in the financial industry that misused models by allowing them to make decision may very well have contributed to the financial crisis but that says more about the users than the efficacy of models themselves.
Reply | Report Abuse | Link to thisThe economy, like the weather, is almost surely a chaotic system. Has everyone forgotten about chaos theory already? Every single thing affects our economy, even the smallest business that surrounds us can be affected when financial trouble takes place; I think one of the best ways to avoid financial downfall is to get a <a href=http://www.corporation.com/businessConsult.html>business consultation</a>.
Reply | Report Abuse | Link to thisMacroeconomics is actually an artifical means for running our social system, devised by man and organized by him. It is therefore just as much an engineering system as the means for controlling the flight of a space-ship. Thus macroeconomics must by its nature be an eaxct science and it only due to those who otherwise wish to confuse it, that it is not regarded as being so.
Reply | Report Abuse | Link to thisThe model that I wish to illustrate is in Google Images as: DiagFuncMacroSyst.pdf which shows how our social system is actually organized. In order to accept it and use it requires 2 assumptions which many "experts" are unwilling to make. These are the engineering ones about aggregate quantities representing individual ones (as in the gas laws) and about idealization of the functions within the organization, thereby enabling a relatively small number of them to fully represent how the system is connected and opperates.
With these pre-requisits it should be clear that the subject can be treated as an exact science. It is about time that as falling within this category, the Scientific American (amongst others) should so regarded it and gave us some proper information about the discoveries in it, which are made after simulating its behaviour when making these assumptions.
The assumption that this problem applies to all kinds of models is not valid. The particular model used for macroeconomics can fall into a limited number of categories some of whaic can be chosen or designed so as to be sufficiently uncomplicated snd the business of calibration tyhen becomes a matter of taking a set of data for a particular year and fitting it into the model. With correct data and suitable model there is only one way that this can go. The model is then allowed to continue its progress through time with suitable introductions of the exogenous factors as they need to apply. The model I use is capable of this and consequently it is reasonable to expect it to be more scientific than what is reported here and to work properly.
Reply | Report Abuse | Link to thisUnfortunately the kinds of models used by most of the finance departments of various governments use a statistical methord for forcasting the future behavour, called econometrics. This methodology was originally developed for a small number of variables and then applied to the many variables in the social system motley. So the problem with it should be no surprize.