S&P 500 project

The Densities of Returns of the 100 stocks of the S&P100.

Dear viewer,

This web was designed to be viewed on our campus in that the size of the videos that you can see are all greater than 40 megabytes each and can't be very reasonably downloaded elsewheres... I do apologise, but I hope that you may yet enjoy the still pictures.

The S&P100 is an index made up of the stock prices of 100 large U.S. companies. Using the daily returns of these 100 companies I have looked at the distributions of the so-called underlying process that 'generates' stock price movements. Is the underlying process describable? Is it completely random or is there some element of predictability? Enquiring minds want to know!

Here is a view of the time period from a popular stock trading software, Telechart 2000: The green bits show the highs and lows for each day, the purple line is a moving average. You can click on the image to enlarge it.

    The daily returns are the amount of money one would gain or lose if you bought $100 of a stock one day and then sold it the next; if the price has gone up then of course you have made money, if it goes down you lose. These numbers are from the CRSP database.
   Of course stocks also pay dividends, and if you owned a stock at dividend paying time the company would also send you a check for whatever amount they choose. So this data also includes these dividend payments. And of course we don't measure returns in terms of dollars and cents, but in terms of a percentage increase or decrease. Over a year's time, stocks have given investors about 5or 6% return to their investment; on a daily basis this means about 0.01 - 0.02%.
   The time period I am looking at is from 1986 to 1989, and includes the stockmarket crash of October 19, 1987, Black Monday. You can see this crash throughout the following images.

Non-parametric techniques are used throughout to estimate the densities. Everyone is familiar with histograms; a histogram shows you the probability of getting each value of some random variable. Histograms are in fact a naive form of non-parametric density estimation. The more elaborate methods used here use a Normal kernel, and allow for confidence intervals to be created, and for the true density of the data to be seen more clearly.
You can click here to see a demonstration of the shortcomings of the näive histogram.

What I have done is to take the daily returns data for these 100 stocks that comprise the index and to look at their distributions. Each day has a different distribution for the 100 stocks. You can see these distributions through time in this video:
Econometricians, economists, and stock-market people like to make models that attempt to predict future values of the stock market so that they can convince other people to give them money; one common assumption of these models had been that returns are distributed Normally, and, more recently, log-Normally.

click on picture to see video!

Next, I consider this hypothesis by showing the N-P distributions with their '95% confidence intervals," and comparing them with Normal distributions of the same mean and variance.
One can see from this video that the confidence intervals (in purple) do not usually contain much or most of the Normal density that was once commonly used (in green). I don't show log-Normal distributions here, but the result is much the same. People use other distributions these days too, the t-distribution and what-have-you; still the results are the same.

click on picture to see video!

Having looked at the distributions of the daily returns one might be somewhat depressed, perhaps. What with things jumping around so much, how can we say much of anything about the 'true process' that is underlying this observed phenomena. In a sense this is begging the question - what exactly is the 'true process?' Isn't it a bit ridiculous for us to imagine there to be any single process, much less any process that we can regularly describe with a distribution of some particular functional form? If one were to actually go out and try and describe the stages of the process of the formation of a price for a stock they would inevitably throw up their hands trying to model each step in some mathematical fashion. And yet there do still seem to be some statements that we can generalize about the data. Notice that in the above graph how the estimated distribution is crosses the Normal distribution about one-third and two-thirds of the way. If we look at a larger sample of data we can see this artifact more consistently.

This video shows the returns of five consecutive days grouped together so that instead of 100 observations we can see the density of 500 observations. Then it moves forward one day and takes a look at the next 5 days of observations. In other words each density has four days in common with the next. In a sense this is cheating, it will suggest that there is some kind of regularity to the data over time when in fact it is really all the same data - at least for dates within five days of each other. On the other hand we'll get to see a more steady estimate without the jumpiness of the one-day estimates.

distributions of a moving grouping of 5 days of daily returns

click on picture to see video!

If you have enough time for another download, you can watch each of these densities lain out one after the next like was done above for the single-day densities. It also has some nice music, and is relaxing to watch as well.

Finally, how about looking at the distributions of the returns of the various stocks separately? I mean, we have been grouping these 100 differents stocks together and looking at the distributions of their returns, but one might imagine that since these are all different companies in different industries, each with its own economics, each with its own news to react to etc etc, that in fact it would make more sense to consider just one of the companies' stocks at a time. Of course we couldn't estimate a distribution from just one day's observation of price and dividend - one needs a few hundred observations to get a reliable estimation - so I have taken 150 days of observations of returns for each company and drawn a distribution. Then, I have moved forward in time by, say, 7 days, and taken another 150 days of observations and done an estimation. Again, as in the five-day estimation above, there is a degree of overlap between distributions, of 143 days, so there is some structure that is only there because of this construction. Nevertheless, any two distributions separated by enough distance will be unrelated, and we could compare the two and make some conclusions about the consistency of the underlying 'true process.'	click on picture to see video!
The most interesting thing about this video is that it is quite apparant how closely connected the different stocks are. Even in the single frame above, we can see how they all have a hump in the densities at the same time. Of course that hump is the crash, when returns were much more spread out and more widely distributed. Usually, we can see, that returns concentrate themselves around some central mean and stay there with a pretty high likelihood. Also of note is the degree to which the shapes move in tandem in general. On the one hand this might be expected as all of these companies are members of the same index, and many people invest in index funds which spread their money evenly across all of the members of the index. Thus one decision to take money out of this type of fund or to put more money in would affect all of the companies identically. And yet one is often told that people invest in companies for more educated reasons, considering the future of the industry, the outlook of the company, the particulars of the situation. Hence one might just as well expect more variety.
Finally, here is a static view of the same time period from two different angles.