"Surely you're being a bit too broad in your comments here."
Nah. I think it's just you who's getting bogged down in the minutiae. :-)
But good comments, though ...
My thoughts :
1) No one thinks speculation is a good idea. People have always been against it. The problem with speculation is that there's *no* principled way to distinguish it from use of the market in the "right" way.
In both cases people buy now, hoping to sell later for a profit. In both cases people look to the behavior of those around them to learn about the world. (The whole point of a market is that aggregates information publicly, so you can't distinguish "speculators" from "honest investors" by whether they base their judgment on prices.)
"Speculation" can be suspected when prices are increasing rapidly. But there can usually be a plausible story told about why prices *should* be rising rapidly this time. (Eg. the internet changes everything.)
Then speculation is only widely acknowledged *later* when there's a crash and people realize that the prices were "wrong". (By which time, by definition, you've admitted that it's possible for there to be difference between the market price and the "reality")
2) You could *assume* that the "speculators" were the high-frequency (day-traders) and try to filter out the high-frequency movements (ie. by forcing a minimal holding time; or by, say, imposing a tax on purchases which reduces the viability of frequent trades for small gains.)
What you're up against here is a) whether the high-frequency signal really *is* the speculation. And b) whether the market loses certain flexibilities by filtering out the higher frequencies.
As I understand it, the market seems to exhibit "drunkards walk" properties (ie. random at all frequencies). The only real "trend" is a steady, underlying increase in share prices. But I suspect that tells us very little about the world and mainly tracks the increase in the money supply, so it's just a kind of inflation. [Update : See update at end of for me pulling back from this claim.]
As an aside, remember that when people talk about the stock-market "outperforming" the rate of inflation, it's nonsense. Shares are a "good" just like anything else. If their price is increasing, that's inflation. All "outperforming" means is that they're increasing at a faster rate than salaries, goods, services and the rest of the economy. In other words, the government is increasing the money supply and the capitalist class is grabbing an increasingly large proportion of it.
the current financial crisis has partly arisen because not enough theoretical work has been done on the manner of these feedback loops and therefore the kind of system dampening that is required to prevent wildly destructive effects.
At this point I realise that I should read more before pondering more, but Phil, is this the kind of thing that you've been exploring with your OPTIMAES project?
During the 20th century economics has focussed on a particular kind of mathematical model : ie. analytic treatment of equilibria. Only recently has the mainstream starting moving away from this to think about other kinds of models (informational ones, agent based ones, cybernetic ones.)
I'm particularly interested in looking at markets from a cybernetics / dynamical systems perspective. There's a rather fascinating underground current of thinking in the 20th century which is agreeably "left wing" and holistic, one that seems to encompass cybernetics, a more liberal organization theory, some psychoanalysis, spirituality, alt.money etc. (Think Stafford Beer or Francisco Varela)
What distinguishes this tradition from mainstream economic theory - which is far more game theoretical, methodologically individualist and appealing to the right - is a belief in feedback, non-linearity and emergence (not just of order but also emergent catastrophic events). I think that economic thinking needs (and is going to get) its next breakthroughs in understanding from this cybernetic / dynamical systems tradition. And the method of discovery will be computer simulation. So, of course, OPTIMAES is all about this :-)
Update : there's an interesting diversion on the comments where I'm discussing the long term trend of share-prices rising. I hypothesized above that this may just be in line with the growth of the money supply. In fact, Darius later convinced me that shares can rise in line with growth of goods and services rather than purely inflation due to the money supply increasing. (I guess I was running with a sort of gold-bug prejudice that I picked up somewhere.) So, ok, I pull back from that earlier suggestion.