• C++ Programming for Financial Engineering
    Highly recommended by thousands of MFE students. Covers essential C++ topics with applications to financial engineering. Learn more Join!
    Python for Finance with Intro to Data Science
    Gain practical understanding of Python to read, understand, and write professional Python code for your first day on the job. Learn more Join!
    An Intuition-Based Options Primer for FE
    Ideal for entry level positions interviews and graduate studies, specializing in options trading arbitrage and options valuation models. Learn more Join!

Machine Learning, Deep Learning and AI: Perspectives

Daniel Duffy

C++ author, trainer
Joined
10/4/07
Messages
10,091
Points
648
These are hot topics and every now and again people ask if they help you get a job. My answer is: I don't know!

Here is a quote from someone seemingly working in the field. I am only the messenger, so stay calm ;)

It would be nice to get feedback from those working in the field.

Speaking as someone who has done machine learning and data science for businesses for almost 10 years now: no, absolutely not. In fact, you should forget that deep learning exists. Most of what you are hearing about deep learning is hype from the marketing departments of large companies rather than a realistic assessment of its importance. It is almost never used in production anywhere, and "knowing deep learning" will not get you a job, unless you've been grad students of people with names like LeCun, Bengio and Hinton. Deep learning is extremely computationally inefficient, and solving useful problems requires physical construction of special computers (video card toasters).

Learn linear algebra, classical statistics, gradient based optimization, filters (wavelets, Kalman filters), linear/logistic regression, decision trees, then ensemble techniques like Gradient Boosted decision trees and Random Forests. From there, you'll have a solid baseline to go into something more advanced, and you'll actually be useful to someone who might give you a job.

In the past, it takes 15 years for a technology to emerge from the laboratories to production.
 
Last edited:
I worked in the statistics academia (machine learning = statistics + a touch of computer science) before entering the world of finance. machine learning is useless in finance. forget about it. I've worked on buy side & sell side ---> machine learning is not big or widely used, when it is used it's simple stuff --> kaman filter, gradient descent, quasi MLE.

Statistics in academia is entirely different to statistics used in the industry. I remember one machine learning 'expert' scolding me for using a model, his advice was to use 10 models and to 'average' out the result using some criterion like BIC. I didn't even bother telling him how much money and resources is needed for 1 model, let alone 10. You'll notice that a lot of quants / quant traders are FAILED academics. Failed is not an insult there, rather a compliment. They saw the use of academia but also the problems associated with it. In the industry, theories must work in practice. That means implementing models, usually simple or quasi vanilla at best. Academia doesn't stop at simple models - it looks at complicated models.

Linear algebra, classical statistics, 'practical' probability theory (sde, filters, pde, markov chain), pricing models (BS, Bachelier, Heston, SABR, LMM basics), asset/market knowledge and coding in one of Python/C++/R/Java. That is all you need (probably more) for a quant position in a fund / IB.
 
I worked in the statistics academia (machine learning = statistics + a touch of computer science) before entering the world of finance. machine learning is useless in finance. forget about it. I've worked on buy side & sell side ---> machine learning is not big or widely used, when it is used it's simple stuff --> kaman filter, gradient descent, quasi MLE.

Statistics in academia is entirely different to statistics used in the industry. I remember one machine learning 'expert' scolding me for using a model, his advice was to use 10 models and to 'average' out the result using some criterion like BIC. I didn't even bother telling him how much money and resources is needed for 1 model, let alone 10. You'll notice that a lot of quants / quant traders are FAILED academics. Failed is not an insult there, rather a compliment. They saw the use of academia but also the problems associated with it. In the industry, theories must work in practice. That means implementing models, usually simple or quasi vanilla at best. Academia doesn't stop at simple models - it looks at complicated models.

Linear algebra, classical statistics, 'practical' probability theory (sde, filters, pde, markov chain), pricing models (BS, Bachelier, Heston, SABR, LMM basics), asset/market knowledge and coding in one of Python/C++/R/Java. That is all you need (probably more) for a quant position in a fund / IB.
THANKS !
 
One follow-on remark is that putting down ML/AI on your resume for a university placing in quant finance is not an obvious headstart?
 
I worked in the statistics academia (machine learning = statistics + a touch of computer science) before entering the world of finance. machine learning is useless in finance. forget about it. I've worked on buy side & sell side ---> machine learning is not big or widely used, when it is used it's simple stuff --> kaman filter, gradient descent, quasi MLE.

Statistics in academia is entirely different to statistics used in the industry. I remember one machine learning 'expert' scolding me for using a model, his advice was to use 10 models and to 'average' out the result using some criterion like BIC. I didn't even bother telling him how much money and resources is needed for 1 model, let alone 10. You'll notice that a lot of quants / quant traders are FAILED academics. Failed is not an insult there, rather a compliment. They saw the use of academia but also the problems associated with it. In the industry, theories must work in practice. That means implementing models, usually simple or quasi vanilla at best. Academia doesn't stop at simple models - it looks at complicated models.

Linear algebra, classical statistics, 'practical' probability theory (sde, filters, pde, markov chain), pricing models (BS, Bachelier, Heston, SABR, LMM basics), asset/market knowledge and coding in one of Python/C++/R/Java. That is all you need (probably more) for a quant position in a fund / IB.
I like honest posts such as these.
 
These are hot topics and every now and again people ask if they help you get a job. My answer is: I don't know!

Here is a quote from someone seemingly working in the field. I am only the messenger, so stay calm ;)

It would be nice to get feedback from those working in the field.

Speaking as someone who has done machine learning and data science for businesses for almost 10 years now: no, absolutely not. In fact, you should forget that deep learning exists. Most of what you are hearing about deep learning is hype from the marketing departments of large companies rather than a realistic assessment of its importance. It is almost never used in production anywhere, and "knowing deep learning" will not get you a job, unless you've been grad students of people with names like LeCun, Bengio and Hinton. Deep learning is extremely computationally inefficient, and solving useful problems requires physical construction of special computers (video card toasters).

Learn linear algebra, classical statistics, gradient based optimization, filters (wavelets, Kalman filters), linear/logistic regression, decision trees, then ensemble techniques like Gradient Boosted decision trees and Random Forests. From there, you'll have a solid baseline to go into something more advanced, and you'll actually be useful to someone who might give you a job.

In the past, it takes 15 years for a technology to emerge from the laboratories to production.
Pretty solid advice. Thank you. :)
 

I saw this when it came out. I'm not sure who you got the quote from but it really is in contradiction to what QuantStart says and also what a large number of recruiters have said to me in NYC/European cities.

First point - No one can predict the future. I think it is better to say - focus on the fundamentals, but there are other areas emerging which may be worth looking into.

Second point - Most of the people on this forum will be postgraduates/undergraduates looking to get into quantitative finance. Due to the sheer size (or lack of) of the quant industry and availability of entry-level quant jobs, most people entering MFE programs will not get a job in quantitative finance (taking into account lower tiered schools etc).

These same individuals could hedge their bets by learning practical techniques and tools from Data Science and Machine Learning. Perhaps Deep Learning is far out of reach due to the computing power it requires but it doesn't make sense why anyone would advocate to put blinders on and only focus on the past.
 
The gradient descent method (in all its incarnations) seems to universal in this domain. Speaking from a numerical analysis viewpoint, the method has several major issues in general that limit its applicability. From my limited reading till now, the literature mentions these problems but I am not sure how big these are.
In any case GDM does not add 'intelligence' as such to applications. It is a tool IMO.

What do experts in the field think?

Some issues offhand:

1. Convergence to local minima.
2. You need gradients (what happens if you don't have them or they don't exist?)
3. Initial guess must be 'good'.
4. What if you don't care about/don't want to compute gradients?
5. Is there a way around having to use threshold functions? IMO they seem like a mechanism to select fit/unfit members of a population..
..
 
Last edited:
NYT on AI

Opinion | Artificial Intelligence Is Stuck. Here’s How to Move It Forward.

Even the trendy technique of “deep learning,” which uses artificial neural networks to discern complex statistical correlations in huge amounts of data, often comes up short.
This is a very interesting and important (imo) point, that the bottom up techniques (deep learning etc.) cannot know, let alone understand, things that are not in the data (eg. black swans).

It happens I am searching for a MSc thesis topic idea related to finance with machine learning now, and this might be a subject with enough meat to chew on. I would love to have a chat with people also interested in the topic.
 
This is a very interesting and important (imo) point, that the bottom up techniques (deep learning etc.) cannot know, let alone understand, things that are not in the data (eg. black swans).

Nice remark.
 
JPMorgan seems to be betting on it:

JPMorgan will soon be using a first-of-its-kind robot to execute trades across its global equities algorithms business, after a European trial of the bank’s new artificial intelligence (AI) programme showed it was much more efficient than traditional methods of buying and selling.

Subscribe to read
 
I think everyone here has bashed machine-learning from a forecasting standpoint (and I agree completely), but what about as a source of data generation (for example natural language processing is a great example). Many buyside shops have adopted this mindset...
 
It’s not useless, and anyone who says that either isn’t working in the industry as a modern “quant” or not really aware of how to properly use ML.

ML does a good job with pattern recognition, it might find that a mean reverting relationship exists in your time series when the value goes beyond a certain threshold, for example.

Even if your models can’t forecast perfectly, if they’re right like 60% of the time and can help you lean in a certain direction on whatever asset you’re trading, it’s pretty useful. The main thing to remember is that you have to properly phrase the question, and figure out the right inputs
 
From 2014 to 2021, Kite was a startup using AI to help developers write code. We have stopped working on Kite, and are no longer supporting the Kite software.



The future is bright

We can’t wait for AI to revolutionize programming. Computers are so magical; it will be amazing to experience a step-function increase in what they can do for us.
 
Back
Top