• C++ Programming for Financial Engineering
    Highly recommended by thousands of MFE students. Covers essential C++ topics with applications to financial engineering. Learn more Join!
    Python for Finance with Intro to Data Science
    Gain practical understanding of Python to read, understand, and write professional Python code for your first day on the job. Learn more Join!
    An Intuition-Based Options Primer for FE
    Ideal for entry level positions interviews and graduate studies, specializing in options trading arbitrage and options valuation models. Learn more Join!

What problems does machine learning solve in Quantitative Finance

SUMIT SEN

Joined
4/13/15
Messages
7
Points
11
I am asking this question because I have done my Ph.D. in 1998 in Fuzzy set theory based Robust Clustering Algorithms for pattern recognition. The work focuses on some areas of Robust Statistics, Fuzzy logic, Neural Net and Genetic Algorithm and implementation in C and Java programs and solves problems in clustering, classification, supervised and unsupervised learning, and robust algorithms against noisy (corrupted) data. Fuzzy set theory based set of intelligent computing methodologies like neural network, genetic algorithms, probabilistic reasoning all of which are for the most part complementary to each other rather than competitive and belong Machine Learning. I have a few published papers on them.

But after graduation I didn't find any appropriate opportunity so continued in the conventional IT industry as software development, enterprise search etc.. Nowadays I hear a lot about Machine learning in quantitative finance. I still wonder what types of problems do they solve. Thanks.
 
I'm not sure if this is exactly 'quantitative finance', but one example is data mining done by some financial data firms / vendors. They use machine learning algorithms to parse Twitter feeds, Bloomberg feeds, etc faster than humans can read and understand the news, come up with trade recommendations and predictions, and sell that data to trading firms.
 
SUMIT SEN said:
I am asking this question because I have done my Ph.D. in 1998 in Fuzzy set theory based Robust Clustering Algorithms for pattern recognition. The work focuses on some areas of Robust Statistics, Fuzzy logic, Neural Net and Genetic Algorithm and implementation in C and Java programs and solves problems in clustering, classification, supervised and unsupervised learning, and robust algorithms against noisy (corrupted) data. Fuzzy set theory based set of intelligent computing methodologies like neural network, genetic algorithms, probabilistic reasoning all of which are for the most part complementary to each other rather than competitive and belong Machine Learning. I have a few published papers on them.

But after graduation I didn't find any appropriate opportunity so continued in the conventional IT industry as software development, enterprise search etc.. Nowadays I hear a lot about Machine learning in quantitative finance. I still wonder what types of problems do they solve. Thanks.
Click to expand...
I find a little disconcerting you went through a PhD program in Machine Learning and don't know how to apply that knowledge to solve problems in finance. I wouldn't say ML techniques will work for sure but at least you should have an idea of how to apply it. Don't you think?
 
pingu said:
I find a little disconcerting you went through a PhD program in Machine Learning and don't know how to apply that knowledge to solve problems in finance. I wouldn't say ML techniques will work for sure but at least you should have an idea of how to apply it. Don't you think?
Click to expand...
Not really that surprising-- just because he knows a whole lot about machine learning doesn't mean he knows anything about finance.
 
CasanovaJ said:
Not really that surprising-- just because he knows a whole lot about machine learning doesn't mean he knows anything about finance.
Click to expand...
Exactly. My PhD. focus was to develop several pattern recognition algorithms to solve problems in Computer Vision, Robotics, Object recognition area. My algorithms can find multiple patterns simultaneously within the data in-spite data is corrupted with (lot of) noise or outliers. My algorithms can find all the groups or clusters even when the number groups or clusters in data are not known . Besides I can derive a multi-criteria optimized decision form qualitative and quantitative input conditions (that impact results). They are all published in journals

I want know if these techniques can solve any (what kind) problems in Finance area. Thanks.
 
SUMIT SEN said:
Exactly. My PhD. focus was to develop several pattern recognition algorithms to solve problems in Computer Vision, Robotics, Object recognition area. My algorithms can find multiple patterns simultaneously within the data in-spite data is corrupted with (lot of) noise or outliers. My algorithms can find all the groups or clusters even when the number groups or clusters in data are not known . Besides I can derive a multi-criteria optimized decision form qualitative and quantitative input conditions (that impact results). They are all published in journals

I want know if these techniques can solve any (what kind) problems in Finance area. Thanks.
Click to expand...
Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?
 
pingu said:
Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?
Click to expand...

The core topic: Take a picture (range image data) of a any physical object and reverse engineer to reconstruct its size, orientation (by finding the edges, planes through robust fuzzy clustering algorithms, fuzzy logic etc.) . My Ph.D. focus was in the algorithm development to find the clusters in the image data. Real life range image data (gathered through a laser emitter) always generate lot of corruptions, incompleteness in data etc. I did my Ph.D. in NJIT and this was a sponsored project from US Dept. of Energy.

Due to limited orientation of financial field I am inquiring what kind of realistic financial data that may have multiple hidden patterns (trends or clusters) embedded inside that. What kind of benefits can be derived by extracting the patterns. Where can I find some samples? Thanks.
 
pingu said:
Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?
Click to expand...
How are you getting that he was just "solving a problem given to him, but unable to ask questions himself?"

If he has a problem (which he probably does), it's that he doesn't seem to know a damn thing about finance-- not that there was something wrong with his PhD research
 
pingu said:
Actually you are wrong. If I want to hire somebody in finance, I want somebody to have good ideas applied in the field. It seems that your PhD was a solution to a problem given to you but you were unable to ask the question itself. The latter is a great skill to have. Even if you are wrong at least is good to have ideas about the problems so you can formulate a solution. You need to be able to formulate the question. It seems you don't even have a question.

So, you found a solution using pattern recognition to a problem given to you in Computer vision data, some sort of clustering solution. Well, think about getting a bunch of financial data and you are tasked to find patterns or clusters of similar patterns which might not be obvious. How simple is that? Couldn't you have thought about it? Same ideas, different data sets.

I just wonder, who chose your PhD topic? did you go to school in US?
Click to expand...

If he can define his problem in finance and he already has skills why will he work for you ? Won't he just pitch his ideas around and smart people will spot it and provide him start up funding.
 
SUMIT SEN said:
The core topic: Take a picture (range image data) of a any physical object and reverse engineer to reconstruct its size, orientation (by finding the edges, planes through robust fuzzy clustering algorithms, fuzzy logic etc.) . My Ph.D. focus was in the algorithm development to find the clusters in the image data. Real life range image data (gathered through a laser emitter) always generate lot of corruptions, incompleteness in data etc. I did my Ph.D. in NJIT and this was a sponsored project from US Dept. of Energy.

Due to limited orientation of financial field I am inquiring what kind of realistic financial data that may have multiple hidden patterns (trends or clusters) embedded inside that. What kind of benefits can be derived by extracting the patterns. Where can I find some samples? Thanks.
Click to expand...

backtesting. you might have models that are profitable, some not so profitable, some suddenly become less profitable. ML would help me to find optimum model parameters in my line of work.
 
ShiSaw said:
If he can define his problem in finance and he already has skills why will he work for you ? Won't he just pitch his ideas around and smart people will spot it and provide him start up funding.
Click to expand...
you must be joking! I don't think you have worked in finance at all. Ideas are dime a dozen, execution is the key.
 
James Ma said:
backtesting. you might have models that are profitable, some not so profitable, some suddenly become less profitable. ML would help me to find optimum model parameters in my line of work.
Click to expand...

I just barely read about "Backtesting" and the issue you proposed of finding optimum model parameters combination that produce desired (profitable) results assuming the model parameters are changing over time. They all appear to be fitting with the solutions of my ML algorithms. Would appreciate if you could let me know how to find more details (possibly some sample data-set) about this. Thanks.
 
pingu said:
you must be joking! I don't think you have worked in finance at all. Ideas are dime a dozen, execution is the key.
Click to expand...
I don't know what the word idea means to you. Ideas are precious, skills are dime a dozen. What is skill anyway, mastery of what some other person wrote in a book.

All of China, India and third world are ready with their math and CS Ph.D. from good schools across the world. But they are all working for some trading shop not necessarily run by quants.

Have you heard of desk quant roles ? Ph.Ds are the technical staff supporting the traders code their ideas. I was at Societe Generale being one of these fellow. Job mandate is different, my job was to change the idea into math model and trader's job was to come up with ways of making money. As you sleep on how to improve your model he sleeps on how to manipulate money based on something he is observing.

Most quants and PhDs are just a technical coolie. "Most" is the word here.
 
Here's a very simple example with free high frequency data: http://truefx.com/?page=downloads

Use machine learning to come up with your best estimate for the future volatility of the currency. Having a good estimate of future volatility (can be 1 min, 10 mins, or days from now) plays a key role in generating theoretical option prices.
 
Anik Roy said:
Here's a very simple example with free high frequency data: http://truefx.com/?page=downloads

Use machine learning to come up with your best estimate for the future volatility of the currency. Having a good estimate of future volatility (can be 1 min, 10 mins, or days from now) plays a key role in generating theoretical option prices.
Click to expand...

Thanks Anik. This really helps.
 
CasanovaJ said:
How are you getting that he was just "solving a problem given to him, but unable to ask questions himself?"

If he has a problem (which he probably does), it's that he doesn't seem to know a damn thing about finance-- not that there was something wrong with his PhD research
Click to expand...

An important part of the PhD is learning to stand on your own two feet. That means, in particular, knowing how to find out what it is you don't know, finding pointers to the literature on your own. OP strangely, wasn't able to find out what basic problems in finance machine learning would be useful for (even those infamous interns that aren't able to Google their way out of a paper bag should be able to find some survey papers on this). Being able to find and download free financial data shouldn't be an issue either....

Anyway, I've seen a lot of different types that complete PhDs. Some people, as Pingu said, have a problem just handed to them. Others have to spend years to come up with a single good problem. And of course a lot of people are somewhere on that spectrum.

The kinds of questions OP is asking, and the way he's asking, maybe they sound like reasonable questions to you, and yeah they are... but it sounds a bit odd to me, and I wager, to anyone that's done a PhD or at least been around a lot of PhD types. (And now I'm probably nit-picking, but the fact that he implemented his algorithms in C and Java is probably the least interesting aspect of that line of work, yet he mentioned them so prominently. It makes me wonder which end of that spectrum we're hearing from.)
 
Back
Top