What problems does machine learning solve in Quantitative Finance

[SUMIT SEN]

I am asking this question because I have done my Ph.D. in 1998 in Fuzzy set theory based Robust Clustering Algorithms for pattern recognition. The work focuses on some areas of Robust Statistics, Fuzzy logic, Neural Net and Genetic Algorithm and implementation in C and Java programs and solves problems in clustering, classification, supervised and unsupervised learning, and robust algorithms against noisy (corrupted) data. Fuzzy set theory based set of intelligent computing methodologies like neural network, genetic algorithms, probabilistic reasoning all of which are for the most part complementary to each other rather than competitive and belong Machine Learning. I have a few published papers on them.

But after graduation I didn't find any appropriate opportunity so continued in the conventional IT industry as software development, enterprise search etc.. Nowadays I hear a lot about Machine learning in quantitative finance. I still wonder what types of problems do they solve. Thanks.

 

[mhy]

I'm not sure if this is exactly 'quantitative finance', but one example is data mining done by some financial data firms / vendors. They use machine learning algorithms to parse Twitter feeds, Bloomberg feeds, etc faster than humans can read and understand the news, come up with trade recommendations and predictions, and sell that data to trading firms.

 

[pingu]

I am asking this question because I have done my Ph.D. in 1998 in Fuzzy set theory based Robust Clustering Algorithms for pattern recognition. The work focuses on some areas of Robust Statistics, Fuzzy logic, Neural Net and Genetic Algorithm and implementation in C and Java programs and solves problems in clustering, classification, supervised and unsupervised learning, and robust algorithms against noisy (corrupted) data. Fuzzy set theory based set of intelligent computing methodologies like neural network, genetic algorithms, probabilistic reasoning all of which are for the most part complementary to each other rather than competitive and belong Machine Learning. I have a few published papers on them.

But after graduation I didn't find any appropriate opportunity so continued in the conventional IT industry as software development, enterprise search etc.. Nowadays I hear a lot about Machine learning in quantitative finance. I still wonder what types of problems do they solve. Thanks.

I find a little disconcerting you went through a PhD program in Machine Learning and don't know how to apply that knowledge to solve problems in finance. I wouldn't say ML techniques will work for sure but at least you should have an idea of how to apply it. Don't you think?

 

[CasanovaJ]

I find a little disconcerting you went through a PhD program in Machine Learning and don't know how to apply that knowledge to solve problems in finance. I wouldn't say ML techniques will work for sure but at least you should have an idea of how to apply it. Don't you think?

Not really that surprising-- just because he knows a whole lot about machine learning doesn't mean he knows anything about finance.

 

[SUMIT SEN]

Not really that surprising-- just because he knows a whole lot about machine learning doesn't mean he knows anything about finance.

Exactly. My PhD. focus was to develop several pattern recognition algorithms to solve problems in Computer Vision, Robotics, Object recognition area. My algorithms can find multiple patterns simultaneously within the data in-spite data is corrupted with (lot of) noise or outliers. My algorithms can find all the groups or clusters even when the number groups or clusters in data are not known . Besides I can derive a multi-criteria optimized decision form qualitative and quantitative input conditions (that impact results). They are all published in journals

I want know if these techniques can solve any (what kind) problems in Finance area. Thanks.

 

[pingu]

Exactly. My PhD. focus was to develop several pattern recognition algorithms to solve problems in Computer Vision, Robotics, Object recognition area. My algorithms can find multiple patterns simultaneously within the data in-spite data is corrupted with (lot of) noise or outliers. My algorithms can find all the groups or clusters even when the number groups or clusters in data are not known . Besides I can derive a multi-criteria optimized decision form qualitative and quantitative input conditions (that impact results). They are all published in journals

I want know if these techniques can solve any (what kind) problems in Finance area. Thanks.

Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?

 

[SUMIT SEN]

Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?

The core topic: Take a picture (range image data) of a any physical object and reverse engineer to reconstruct its size, orientation (by finding the edges, planes through robust fuzzy clustering algorithms, fuzzy logic etc.) . My Ph.D. focus was in the algorithm development to find the clusters in the image data. Real life range image data (gathered through a laser emitter) always generate lot of corruptions, incompleteness in data etc. I did my Ph.D. in NJIT and this was a sponsored project from US Dept. of Energy.

Due to limited orientation of financial field I am inquiring what kind of realistic financial data that may have multiple hidden patterns (trends or clusters) embedded inside that. What kind of benefits can be derived by extracting the patterns. Where can I find some samples? Thanks.

 

[CasanovaJ]

Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?

How are you getting that he was just "solving a problem given to him, but unable to ask questions himself?"

If he has a problem (which he probably does), it's that he doesn't seem to know a damn thing about finance-- not that there was something wrong with his PhD research

 

[ShiSaw]

Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?

If he can define his problem in finance and he already has skills why will he work for you ? Won't he just pitch his ideas around and smart people will spot it and provide him start up funding.

 

[James Ma]

The core topic: Take a picture (range image data) of a any physical object and reverse engineer to reconstruct its size, orientation (by finding the edges, planes through robust fuzzy clustering algorithms, fuzzy logic etc.) . My Ph.D. focus was in the algorithm development to find the clusters in the image data. Real life range image data (gathered through a laser emitter) always generate lot of corruptions, incompleteness in data etc. I did my Ph.D. in NJIT and this was a sponsored project from US Dept. of Energy.

Due to limited orientation of financial field I am inquiring what kind of realistic financial data that may have multiple hidden patterns (trends or clusters) embedded inside that. What kind of benefits can be derived by extracting the patterns. Where can I find some samples? Thanks.

backtesting. you might have models that are profitable, some not so profitable, some suddenly become less profitable. ML would help me to find optimum model parameters in my line of work.

 

[pingu]

If he can define his problem in finance and he already has skills why will he work for you ? Won't he just pitch his ideas around and smart people will spot it and provide him start up funding.

you must be joking! I don't think you have worked in finance at all. Ideas are dime a dozen, execution is the key.

 

[Andy Nguyen]

We have a nice Big Data in Finance article written by Andrew Sheppard in our QuantNet Guide. Download it and take a look
https://www.quantnet.com/threads/ma...uants-mfe-financial-engineering-students.535/

My feeling is that as it's a relatively new, the applications and techniques are getting discovered everyday. It may change the world or it may be just another tool to help you do your job better.

 

[SUMIT SEN]

We have a nice Big Data in Finance article written by Andrew Sheppard in our QuantNet Guide. Download it and take a look
https://www.quantnet.com/threads/ma...uants-mfe-financial-engineering-students.535/

My feeling is that as it's a relatively new, the applications and techniques are getting discovered everyday. It may change the world or it may be just another tool to help you do your job better.

Thanks Andy!

 

[SUMIT SEN]

backtesting. you might have models that are profitable, some not so profitable, some suddenly become less profitable. ML would help me to find optimum model parameters in my line of work.

I just barely read about "Backtesting" and the issue you proposed of finding optimum model parameters combination that produce desired (profitable) results assuming the model parameters are changing over time. They all appear to be fitting with the solutions of my ML algorithms. Would appreciate if you could let me know how to find more details (possibly some sample data-set) about this. Thanks.

 

[ShiSaw]

you must be joking! I don't think you have worked in finance at all. Ideas are dime a dozen, execution is the key.

I don't know what the word idea means to you. Ideas are precious, skills are dime a dozen. What is skill anyway, mastery of what some other person wrote in a book.

All of China, India and third world are ready with their math and CS Ph.D. from good schools across the world. But they are all working for some trading shop not necessarily run by quants.

Have you heard of desk quant roles ? Ph.Ds are the technical staff supporting the traders code their ideas. I was at Societe Generale being one of these fellow. Job mandate is different, my job was to change the idea into math model and trader's job was to come up with ways of making money. As you sleep on how to improve your model he sleeps on how to manipulate money based on something he is observing.

Most quants and PhDs are just a technical coolie. "Most" is the word here.

 

[Anik Roy]

Here's a very simple example with free high frequency data: http://truefx.com/?page=downloads

Use machine learning to come up with your best estimate for the future volatility of the currency. Having a good estimate of future volatility (can be 1 min, 10 mins, or days from now) plays a key role in generating theoretical option prices.

 

[SUMIT SEN]

Here's a very simple example with free high frequency data: http://truefx.com/?page=downloads

Use machine learning to come up with your best estimate for the future volatility of the currency. Having a good estimate of future volatility (can be 1 min, 10 mins, or days from now) plays a key role in generating theoretical option prices.

Thanks Anik. This really helps.

 

[yetanotherquant]

One of the most famous paper, in which a (really simple) pattern recognition algorithm is applied, is this one:
http://www.cis.upenn.edu/~mkearns/teaching/cis700/lo.pdf (cited in more than 800 papers)

Can you do better?

 

[C S]

How are you getting that he was just "solving a problem given to him, but unable to ask questions himself?"

If he has a problem (which he probably does), it's that he doesn't seem to know a damn thing about finance-- not that there was something wrong with his PhD research

An important part of the PhD is learning to stand on your own two feet. That means, in particular, knowing how to find out what it is you don't know, finding pointers to the literature on your own. OP strangely, wasn't able to find out what basic problems in finance machine learning would be useful for (even those infamous interns that aren't able to Google their way out of a paper bag should be able to find some survey papers on this). Being able to find and download free financial data shouldn't be an issue either....

Anyway, I've seen a lot of different types that complete PhDs. Some people, as Pingu said, have a problem just handed to them. Others have to spend years to come up with a single good problem. And of course a lot of people are somewhere on that spectrum.

The kinds of questions OP is asking, and the way he's asking, maybe they sound like reasonable questions to you, and yeah they are... but it sounds a bit odd to me, and I wager, to anyone that's done a PhD or at least been around a lot of PhD types. (And now I'm probably nit-picking, but the fact that he implemented his algorithms in C and Java is probably the least interesting aspect of that line of work, yet he mentioned them so prominently. It makes me wonder which end of that spectrum we're hearing from.)

 

[SUMIT SEN]

One of the most famous paper, in which a (really simple) pattern recognition algorithm is applied, is this one:
http://www.cis.upenn.edu/~mkearns/teaching/cis700/lo.pdf (cited in more than 800 papers)

Can you do better?

Yes. Going through the paper, need sometime to absorb it. I will update after applying the algorithms to the same problem. Thanks.