Couldn't agree more. I have noted previously the OP sounds passionate about the job, so the salaries won't be a drawback, but what is a drawback is when you factor in the constant turbulence banking has been in for the last 20 years and, as I said before, NPV it out.
Marketing on this site suggests a world of opportunities for mathematicians (true-ish) and it's a good thing as sites like this helped me avoid dreary jobs such as actuary or accountancy. But underneath it is not a bed of roses. What the OP needs to look at is where is there is investment, not "loads of jobs" - in many sectors job vacancies are on the rise where firms prefer to have lots of short-term contractors (i.e. experienced personnel they don't have to train or supervise), creating an illusion of employment opportunity (e.g. IT in some sectors in the UK).
Trends in quant jobs are rarely understood by non-finance people and, as Ian rightly points out, there is a disparity at the moment between the so-called "bouyant" Wall Street market and reality of hell for quants. Financial firms aren't expanding or hiring inexperienced newbies, but squeezing value out of their existing portfolio to record better profits.
In contrast an example of actual investment is Big Data - not sure I entirely buy into the buzz, but most of the newly employed maths people I know are in this field or biostats.
