Big data for trading

[JacobJ]

How relevant is big data for trading securities? Is there a bias towards a particular class of asset? Is it more relevant for particular trading style like social trading, low latency trading ??

Below is the response from Prof.Andrew Sheppard to a conversation I had with him. Kindly do offer your comments on the topic

Andrew Sheppard said: ↑
Big Data is no substitute for good trading ideas! (Meaning spotting opportunities, executing well, and managing the risks.)

In fact, a more general statement would be that Big Data is no substitute for good thinking -- Big Ideas, if I can put it that way. But Big Data can augment good ideas (and I include in this good ideas that come from a machine, say, through Machine Learning).

That said, Big Data can be very useful in prospecting for profitable market opportunities. The same can be said for opportunities that are financial in nature but not market related.

Let me illustrate with a handful of examples:

1. High Frequency Trading (HFT). Trading volumes in certain asset classes is sometimes very high, and you can accumulate terabytes of data per day, which mounts to very quickly. HFT is really divisible into two distinct aspects: a) execution, and b) trading strategy development and back testing. "a" is all about latency and speed of execution. "b" is really about Big data; backtesting many trading strategies to see which are profitable and why.
2. Event Driven Investing. I have heard of some hedge funds that use social media (now definitely a source of streaming Big Data) to find trading strategies that are driven by events first detected via social media networks. And example would be a hedge fund that puts bets on company stocks in the event a product recalls. "Oct 17, 2013 - Toyota Motor Co. is recalling 885,000 vehicles worldwide over electrical problems that could prevent airbags from deploying in a crash." That's a story item I just pulled off Google. That's the sort of news that moves stock prices! It may perhaps drive down the company's stock price. So, suppose, for example, that that piece of news could be picked up from chatter on Twitter before it hit the mainstream newswire? A trade could be put on the sells short the company's stock in anticipation of a price decline. To the extent that social networks are "early warning systems", this could be made into profitable trading opportunities. But Twitter alone generates huge amounts of data every day. Taken together with other social media websites (Facebook, etc.) then that's Big Data. If you can store it and view the stream in real-time better than anyone else, I would imagine that it could be very profitable indeed.
3. Better control over risk. Oftentimes there is such an emphasis on profit that people neglect the flip side of the coin: loss. Sure it's important to make money, but it's also important to keep what you earn! The first rule of trading is preservation of capital; lose that and you are out of the game. Moreover, a dollar made through risk management is in some ways a better quality dollar -- less volatile -- than a dollar earned. Consider, for example, a credit card company that uses it's vast about of transactions data to reduce fraud. A few percentage points of improvement in fraud prevention can mean big savings. And all done without acquiring new customers or having to process more transactions (in fact, merely denying a small number of fraudulent transactions). That's the power of Big Data!

In terms of what areas in which Big Data can be a source of ideas and profitable opportunities, then that is obviously very dependent on availability of data (and in this case Big Data). There may well be areas where Big Data does not exist (or is too expensive, which amounts to the same thing I suppose). But data is growing in most areas and, subject to access being restricted by cost, it's likely that more and more asset groups and markets will have more useful Big Data over time.

Anyway, just my 2 cents worth!