The stock market measures how things change from what we predicted. If only things we predicted happened, the prices of stocks wouldn't really change.

You might think, "But Apple went from a tiny company to a giant company."

Sure, but if that was predictable, we all would have bought Apple stock back when it was a tiny company (since we knew it would become a giant company) and the price back then would have been much higher. The price of Apple stock was low in the past because we didn't know how big Apple would become. As that information became apparent, the price went up.

Simply put, anything people expect to happen will already be built into the price. The changes we see come from things unexpected.

A sudden trade war with China. Trump gets elected. A hurricane hits North Carolina. A particular drug is discovered to cure a disease, or maybe one is taken off the market because it causes birth defects.

So if we have a good model about how. People behave on a day to day basis, why can't this be transferred to stocks?

It is. The price of a stock today does include everything people expect to happen. The possibility that Apple would become what it is today has been factored into Apple's price all along. But that was mixed in with other then possible outcomes that didn't come to pass.

If you think about, most of the things we do in a day are pretty predictable.

Really?

Waking up,

You never oversleep.

deciding what sites to browse,

Not sure how many people plan that in the morning. I do not.

So if we have a good model about how. People behave on a day to day basis, why can't
his be transferred to stocks?

Because:

• You do not even come close to have a good model. You assume because you have or can name a couple of data points that is a good model. It is not even close to be an idea for a possibly model.


• Because the stock market is not only people and it is a LOT of actors all actions quite possibly quite rationally but together forming a complex chaotic system.

You can not go easily from "I know how Joe would react" to "I have a model that can model millions of different actors reliably".

Short answer: arbitrage.

The way I see it, you're not really betting on anything physical, nor even on how well companies perform; you're betting on what all the other investors/speculators think about all that.  And what everyone else thinks they think.  And so on, and so on…  It's a massive game of second-guessing, third-guessing, and umpteenth-guesing.

As Ben Kingley's character said in Sneakers, it's all about the information.  The markets are designed to use information as efficiently as possible.  (That's why insider trading is one of the most heinous financial crimes; it prevents the markets working efficiently.)

The result is that anything predictable cancels out.  As soon as something is predictable, traders react by buying or selling, which moves the price to counteract it.  And the more efficient the markets become, the faster and more accurately this happens — to the point where the only price movements you see are completely unpredictable.

As soon as there's a difference between what the price is, and what people think it should be or is going to be, they will take advantage of that difference (arbitrage), and the difference will disappear.  In effect, it's a very tight negative-feedback loop, filtering out everything but noise.

Few things in life are based around such a rapid and vast feedback loop, which is why it seems counterintuitive.

Whatever is predictable about a financial product is already factored into its price. For instance, if the company is predicted to do well, its stock becomes overvalued, and the opposite happens to companies those future looks grim.

The remaining uncertainty in a financial product after such corrections is truly random from the point of view of anyone who hasn't got a better prediction model. And if someone does, they will have the incentive to trade in a way that again factors that better knowledge into the price: they will buy products that they expect to perform well (thus increasing their price) and sell the products that they expect to perform poorly.

I'm not sure why you think it is unpredictable. Good companies do well bad companies do poorly. Virtually all companies will fall when the market is overpriced and wall Street finds a reason to offload. Virtually all companies with solid earnings will rise when the market is undervalued. You see the market is very predictable when it reaches obvious valuations, both over and under. The reason you say it is unpredictable is because you are entering and exiting too frequently. If all you did was jump in at the bottom of a crash and jump out when it reached lofty valuations you would say it was very predictable

The elephant in the room missing in the other answers is the irrationality inherent in stock (and other) markets.

Sure enough: There are many measurable, rational, "real" things influencing stock value. The monetary value of a company, how we expect the market for their products to develop, how expensive the stock is compared to the company's earnings, capital cost etc.

But one of the most influential reasons to buy or sell stock — driving its value up and down — is simply that we expect that stock to rise or fall in value. Speculation is as old as the markets.

The feedback loop which is thus established is pretty obvious: Part of the demand for stock has no discernible root in physical or financial reality; instead, it is driven by what other people think, which in turn is strongly influenced by that very stock value development.

This is how bubbles expand and burst, with oscillations that are orders of magnitude larger than the effects of the actual tangible operations of a company. Such feedback loops are non-linear and correspondingly hard to predict, and predicting them involves psychology more than economics. Some people do that better than others; you may be interested in Michael Lewis' The Big Short, specifically the story of Michael Burry², or examine the point in time when Warren Buffet started buying again in 2008 while everybody else thought the world was going to end.¹

My impression is that it is better for one's nerves to buy an index paper and stick with it ;-).

_{² Although truth be told, Michael Burry's shorting of CDOs was based on hard economics — he was about the only one reading the legally required fine print — and not psychology. He defied speculation arguing that eventually the underlying economic reality would put an end to it. He was right, in the end.}

Three reasons:

(a) thin margins

Companies sell a product for price P, and buy the stuff/services needed to build the product for price 0.9P. The profit is 0.1P, rather thin. If for some reason customers' willingness to pay decreases by 10%, the whole profit is lost. Thus, in times when economic conditions are not optimal, the company cannot pay dividend. The only reason people invest in stocks is in fact it, namely the dividend. (Some stocks don't pay dividend but their value is based on the expectation that someday they could start paying dividend. Note that dividends are equivalent to share buybacks, so if a company is buying back its shares, it's effectively paying dividend.)

(b) supply and demand

Prices do vary based on supply and demand of savings instruments. At times when saving money into stocks is trendy, stocks are expensive. At times when spending all of your money is trendy, stocks are cheap. For example, during a recession a huge number of people want to access their savings, driving down prices. If you are countercyclical, you will find that during a recession, you have a huge amount of good investment opportunities at ridiculously low prices.

(c) liquidity

Stock market is actually a safer investment than e.g. the housing market. See for example this: http://digitalarchive.maastrichtuniversity.nl/fedora/get/guid:cda347a3-7fdf-426e-99a4-1a2c30717764/ASSET1 (links seems to be rather slow as I'm writing this)

In this file, it is explained how house prices have gone both up and down. The maximum decrease was 80%. It's about the same than stocks during a very large depression. However, with stocks it's easy to diversify whereas most people have the money to purchase only 1 or perhaps 2 houses. So, due to lack of diversification it generally doesn't make sense to invest money into housing market but rather invest it into stock market.

Why people do think stocks are unpredictable and houses are predictable, then? Because stock market is liquid. The 80% crash in real house prices happened over a period of over 35 years between 1778-1779 and 1814-1815. Stocks could crash in few days or less.

So if we have a good model about how. People behave on a day to day basis, why can't this be transferred to stocks?

Stocks are, in essence, just a commodity to be bought and sold. And that means you have a price.

A price is simply a value we assign something (and we tend to go based on someone being willing to pay said price, not the pure opinion of the previous owner). Prices can be volatile. Milk might cost $3 per gallon, but what if some disease wipes out half the cows that make the milk? The price of milk goes up. Is there any way to predict that? And that's assuming we only have one factor driving the price to change.

Prices contain too much information to process meaningfully

Why prices are a mystery isn’t a mystery. The why of prices — or at least all the whys of prices — are simply unknowable. Sure, you might be able to know why person X wants $28.99 for his vintage Batman talking alarm clock while another person wants $80.99 for the same masterpiece, but you can’t know all the reasons why it originally sold for $9.99. Once you make your peace with that fact, the mystery ceases being mysterious.

What I think is fascinating is that prices aren’t unique. We think they are a category unto themselves. A price isn’t like anything else, right? Well, wrong — maybe. If prices are simply the agglomeration of disparate and complex information boiled down to a number, can’t other things be agglomerations of disparate and complex information boiled down to something else?

Anyway, the point I’m getting at is that there’s nothing — nothing — that we say, do, own, make, or believe that isn’t brimming with all of the sorts of information that go into a price.

The stock market is driven by a myriad of forces and interests. I have money in there for my retirement. Warren Buffet has money in there make money for other people. John Doe has money in there because he likes owning a certain brand of stock. Different goals. Maybe Warren Buffet dumps his stock in Company X, which drives the price down, while John hangs on to his no matter the price. Or maybe a bug in stock trading software causes havoc

When the market opened at 9:30 AM people quickly knew something was wrong. By 9:31 AM it was evident to many people on Wall Street that something serious was happening. The market was being flooded with orders out of the ordinary for regular trading volumes on certain stocks. By 9:32 AM many people on Wall Street were wondering why it hadn’t stopped. This was an eternity in high-speed trading terms. Why hadn’t someone hit the kill-switch on whatever system was doing this? As it turns out there was no kill switch. During the first 45-minutes of trading Knight’s executions constituted more than 50% of the trading volume, driving certain stocks up over 10% of their value. As a result other stocks decreased in value in response to the erroneous trades.

TL;DR

Stocks are based on prices (what people are willing to pay for them). Prices are based on too much information to know why they are priced the way they are at any given moment. Stocks are based on even more information than most commodities.

There are two types of prediction problems typically.

The level 1 prediction models are for things like weather - one could predict something, wait and observe, and then correct the error for next prediction. The key aspect is the fact whether it would or would not rain on a certain day is not dependent, i.e., it does not change, due to the prediction (that it is going to rain, or it is not going to rain).

However level 2 predictions are harder - which is precisely the stock market. Even if one develops an excellent model that predicts how the stock market is going to behave, the moment somebody actually starts using the model to buy and sell stocks - the market "sees" the increased buying/selling of a certain stock, and corrects itself appropriately. Soon this excellent model becomes obsolete.

And the duration for which this excellent model predicted the stock behavior correctly could be really small, that its short term success is pretty much indistinguishable from noise.

A term that has been used to describe the stock market is anti-inductive. See https://www.lesswrong.com/posts/h24JGbmweNpWZfBkM/markets-are-anti-inductive and
https://slatestarcodex.com/2015/01/11/the-phatic-and-the-anti-inductive/ . Our knowledge of a planet's orbit doesn't affect it. So we can learn more about an orbit, and then the orbit will be easier to predict. But with the market, learning about it changes its behavior. Whenever we learn more about the market, ;people act on that information, changing the market. So the more people work on trying to understand the market, the more people are affecting the market, and the harder it is to understand the market. Unlike with predicting planet's orbits, when you predict the market, you have to include the behavior of people, and people will be reacting to your analysis of them market. So your predictions have to predict the actions of people reacting to analysis that predict the behavior of people reacting to analysis that predicts ... and so on.If one person happens to be better than everyone else at analyzing the market, then they might be successful in predicting it. But people in general can't predict the market, because the average person, by definition, isn't any better than any other average person, so if one average person can figure out that stock will go up, then every other average person can figure that out too and bid the price up. Stock market fluctuations represent the unpredicted parts of the economy. If they were predicted, then they would have already been included in the price. We can't predict market fluctuations because the fluctuations are by definition what isn't predicted.

Why is the stock market so unpredictable? ...most of the things we do in a day are pretty predictable.

If you mean overall and you mean in the long run then the US stock market is predictable.

It is like how people are predictable in large numbers - what a random person will do is not predictable.

At a US theme park (like Disney) people, when given a choice, will tend to turn right instead of left. That can be established as fact, because people study this stuff and that is what they've found.

That does not mean that you can predict whether a specific person will choose to go to the left or right at a specific place.

The stock market is the same:

A certified financial planner will likely tell you that, "If you will need the money in the next five years, you probably should not put it in the stock market."

Why would they say that?

Because if you look at mutual funds with 10+ year track records, you'll find that they have a positive return in almost all periods of five consecutive years of their history.

(four point something times out of five if I remember correctly)

TL;DR;

The stock market is very likely to go up if you are looking at period of five or more consecutive years.

What a particular stock will do is anybody's guess...

and they're all guessing differently...

and people change their guess from one day to the next...

and that is what sets the price...

which why a particular stock moves so erratically...

But... over time (measured in years)... the market goes up :-)

There are two related reasons. The first is that if today's price is equal to yesterday's prices times a reward plus an appraisal error then the resulting difference equation can be proven to be intrinsically unstable, p(t+1)=Rx(t)+e(t+1). As long as the appraisal error has finite variance centered on zero, then any Frequentist statistical estimator will have no predictive power at all.

The second reason is that if an equilibrium price exists at each moment in time and the equilibrium reward plus a shock is the true reward then the distribution of returns around the equilibrium return will be the truncated Cauchy distribution if you have factored out liquidity risks, dividend risks, merger risks, and bankruptcy risks.

The Cauchy distribution is unusual in that it cannot have an average. You can always calculate the sample average but it won't ever settle down and converge to the true center of the distribution, which for stocks is the mode.

John Cook provides a graphical example of the difference of the behavior of the sample mean of a Cauchy distribution and the mean of the normal distribution. You can find it at Cauchy versus Normal

The mathematician Benoit Mandelbrot classified the behavior of random variables as having seven classifications. The type you generally encounter in day-to-day life is proper mild randomness. Equity security returns are the seventh class of randomness, which is extreme randomness.

In fact, you can predict things in equity markets. You can only use Bayesian methods. For a variety of reasons, it can be shown that it would be profoundly unwise to use a Frequentist methodology, but most do because it is what they know.

The difficulty is that most people try to predict things using Frequentist methods of proper mild randomness rather than Bayesian methods for extreme randomness. That includes professionals.

As there is no way to use MathML in this forum but I am about to produce a resource you could use though it is intended for those with doctorates in statistics, mathematics, finance, and economics. I will try and remember to post a link to it when it is in its final form. Elements of it could be used by anyone with a basic understanding of probability and basic calculus methods.

The physicist and statistical polemicist E.T. Jaynes wrote that had the first problems in statistics involved the Cauchy distribution the trajectory of the field of statistics would have been radically different.