Unless you have tenure at a university, yes. Once you have tenure and are careful to keep your head below the parapet by not saying anything politically or culturally controversial, you have a sinecure for life. If you have a tenure-track position, you have some hope. But for the unwashed masses who constitute today's Ph.D. lumpenproletariat, life is no bed of roses. I think the number of physics Ph.D.s produced each year in the USA is enough to staff all US academic physics departments. In mathematics, in the early '90s, a non-tenure-track assistant professorship at the U of New Mexico (hardly a ranking school) attracted 700 applications. No matter how hard getting quant employment may be, it's a lark compared to get a real academic appointment. And the payscales are, as you've indicated, vastly different. A full professor can often be earning less than six figures in a research university. For those who join the semi-starving army of adjuncts or the army of insecure post-docs, life is much harder.
On top of that, no respect. In 19th century Berlin and Vienna, waiters would obseqiously address academics as "Herr Professor," and Ph.D.s as "Herr Doctor." Those days are long gone.
Too much bureaucracy. Too much in-fighting for scraps of money. The Damocles' sword of cutbacks and retrenchments. So for all these reasons it's no wonder that people like Wilmott, Joshi, and Derman drift into the orbit of quant finance.
By way of anecdote, the full-time faculty members of South Dakota's Mount Marty College were earning barely $20,000 annually in the late '90s (and it probably hasn't gone up much since then); when attending a national conference in Chicago in 1997, they saved money by staying at a homeless persons' shelter (source: the book Academic Keywords, by Nelson and Watt, published by Routledge, 1999)
