It is commonly assumed that the social sciences such as finance or economics, in contrast to the ‘hard sciences’, are unpredictable, or that the social sciences are ‘chaotic’ and hence not amenable to the sort of rigid, top-down thinking and model building of the hard sciences. Physics has models of high predictive value, but behavioral economics, the stock market, human nature is hopelessly confusing and cannot be explicated as a set of rules, or so goes the narrative.
I believe this is mostly wrong.
First, the math underlying physics is considerably more complicated than macro-econ-math or finance math (even though this math can be very hard too). Google what a ‘spherical harmonic’ is if you don’t believe me. This is is still way less complicated than general relativity or string theory, yet this 19th century physics concept, conceptually, still greatly exceeds that complexity of probably most econ papers.
Anyone who can understand even 19th century physics should have no trouble understanding macro econ or finance. I believe physicists and mathematicians can easily run laps around macro economists, sociologists, and behavior economists in terms of predicting things (contrary to conventional wisdom that economics is hard to predict) and making money (the success of Renaissance Technologies, which is staffed almost entirely by mathematicians, is evidence of this).
They (these PhDs, physicists, etc) can look at the data and with little effort elucidate the underlying rule or pattern without tons of trial and erorr or effort. The ‘madness of the crowds’ becomes very predictable once the underlying rules or systems are known. Newton when he famously remarked “I can predict the movement of heavenly bodies, but not the madness of crowds” was wrong (or at least he couldn’t, but this doesn’t mean others cannot). To say it cannot be done is an intellectual cop-out and not supported by the evidence of individuals and funds that can. Although in an HBD-sense I may be a determinist, I think human ingenuity in general should never be underestimated.
Crowds may be mad but they still follow predictable rules. Imagine a stadium with tens of thousands of people and a dozen doors. I don’t need to know which door each fan will take, only that when the game is over everyone will have vacated the stadium (conversely, the stadium being empty before the game begins), analogous to the endpoints of the system.
What about the failure of Long-Term Capital Management L.P. (LTCM)? The founders were smart by any objective sense, but they assumed considerable tail-risk that probably would have been obvious to anyone with even rudimentary math and reasoning skills. LTCM was purely a quant arbitrage fund that sought to profit from tiny differentials magnified with great leverage. That worked fine for 3 or so years, until the leverage backfired and the fund was rendered insolvent soon after.
LTCM failed because it was path dependent, so it required foresight of all possible pitfalls that lay in wait between the endpoints, not just knowledge of the endpoints. Regarding the stadium example, path dependency is like a condition in which the fund becomes insolvent if an excess number of fans exit/enter through a specific door.
Statistically, these pitfalls are absorbing barriers that when touched result in catastrophic, irreparable loss of capital and the random walk process terminating. Predicting path dependency is vastly harder to do, if not impossible, than just knowing the endpoints. The systems-based approach has minimal path dependency.
The idea behind a systems-based approach is that by knowing the final evolution or state of the system, you can predict the outcome without having to worry about the stuff in-between. As shown below, there are infinite possible paths, but only two fixed endpoints:
The system-based approach treats these paths as irrelevant.
That is not to say Quantitative finance does not have its uses: it is useful for optimizing models and risk management, such as the use of put options and modeling how such options behave under various market conditions. Quant strategies tend to have more consistent returns with smaller draw-downs, but the downsides are path dependency and various hidden risks. System-based approaches have greater absolute returns, little to no path dependency, but more volatility and less consistent gains.