I come from a different field (Machine learning/AI/data science), but aim to ask a philosophical question with the utmost respect:
Why do quantitative financial analysts (analysts/traders/etc.) prefer (or at least seem) traditional statistical methods (traditional = frequentist/regression/normal correlation methods/ts analysis) over newer AI/machine learning methods? I've read a million models, but it seems biased?
Background: I recently joined a 1B AUM (I know it's not a ton) asset management firm. I was asked to build a new model for a sector rotation strategy (basically predicting which SP 500 sector would do the best over 6 months-- chose to use forward rolling 6 month returns) they employ and my first inclination was to combine ARIMA (traditional) with random forest (feature selection) and a categorical (based on normal distribution standard deviation) gradient boosted classifier for ETFs in each sector. Not to be rude, but I beat the ValuLine timeliness for each sector. I used the above mentioned returns as my indicator and pretty much threw everthing at the wall for predictors initially (basically just combing FRED), then used randomForest to select features. I ended up combining EMA and percent change to create a pretty solid model that, like I said, beat ValuLine. 

I've read a lot of literature, and I  haven't seen anyone do anything like this. Any help in terms of pointing me in the right direction for literature? Or any answers to the overarching idea of why isn't there more machine learning in equity markets (forgetting social/news analysis)?
EDIT: For clarification, I'm really interested in long-term predictions (I think Shiller was right) based on macro predictors.

Thanks 

PS- I've been lurking for a while. Thanks for all the awesome questions, answers, and discussions.

Because of:


The (extreme) dominance of noise over signal
The prevalence of non-repeating patterns (many of which we know are not going to repeat)
A pathetic sample size for cross-validation 
Regime changes due to exogenous events. These are typically in the cross-val window which makes it even worse. (GFC, financial integration, trade law changes, interest rate adjustments by central banks, some idiot in a bank was hiding trades and loses 5 billions dollars, etc).
It is well known that non-linear relationships are generally just artefacts of the in sample dataset


There is also the following:


Much price changes are driven by news such as a plane crashing or a merger announcement. Are you trying to forecast news (!?) by getting your model to learn non-linear relationships on price data? It should be clear that, if American Airlines price falls due to a terrorist hijacking, it is not going to be useful to have a random forest learn any patterns that result since it will not repeat. 


Because of these factors many (econometricians and practicioners) will try to use a priori knowledge to select features and impose constraints on the model in an attempt to improve generalization. This is perceived as necessary by econometricians since the data is too thin, noisy and nonstationary (i.e., the above reasons).

This is not to say that "machine learning" methods such as Lasso, NNG, Elastic Nets or Ridge can't be applied. They result in essentially linear models and you can impose whatever a priori constraints on it through the metaparameters in the loss function or by using a variant that preserves hierarchies when using 
indicator function interactions (Tibshirani 2013...). Edit: You will still need to select which features go into the algorithm (as a prior imposition) but you can use these to achieve slightly more sparsity than you would otherwise have and introduce some bias into your conditional expectation (or state probability if you're doing multinomial categorical GLM) for improvement in variance of sampling distribution.

I am however open to random forests with the right a priori constraints in place.

There are indeed hundreds of papers that use machine learning to forecast financial markets. Just google something silly like "fuzzy bayesian expert adaptive learners with PSO training S&P 500" and you will get a lesson in the file-drawer effect, publication bias and substandard research methodologies (e.g. selecting 3 of 50 algorithms and 2 of 50 indices  and hoping it convinces people).

However, the above is an optimist's view of the industry. From those I've spoken to at low frequency funds they are simply ignorant of machine learning and couldn't apply it because they lack the knowledge and skills. If they were actually interested in being true quants, who knows how much damage they could do with deep learning or something. 

If you want to do real machine learning in finance and actually do something that is meritocratic/skill/scientific instead of almost completely random and full of people who practice nonsense, go to a HFT firm (not that most people practice non-sense in low frequency funds, just that many do and this is something that is absolutely impossible to get away with in HFT). That said, I am continually and consistently underwhelmed when I hear of the research methods of low frequency quant funds.
