Q: Predicting continuous variables from text features I want to predict a continuous variable from text features. Lets say I have some student essays and I want to predict their quality, as measured by a human grader, using text features (mostly words they use). 
Linear regression is an obvious candidate, but if I have substantially more features than graded essays, this probably won't do well.
If I wanted to classify them into good/bad, I might try the Naive Bayes classifier. I don't, but maybe I can draw inspiration from that.
As I understand, Naive Bayes draws its power from assuming feature independence. Is there such a thing as Naive Multivariate Linear Regression, where you assume feature independence? 
I think this is the same as using Univariate Linear Regression for each regression coefficient. I would expect that to run into problems quickly, though. 
Is there something halfway between these two models? 
Putting a prior distribution on feature covariance that mostly expects conditional independence? Other models I should consider?
Bayesian models preferred.

A: A similar question has been asked on stackoverflow:


*

*https://stackoverflow.com/questions/15087322/how-to-predict-a-continuous-value-time-from-text-documents
One answer here was to use k-nearest-neighbor regression to predict a continuous value from text documents, see https://stackoverflow.com/a/15089788/179014.

A: I recommend Gradient Boosting with trees as described in chapter 10 "Boosting and Additive Trees" of The elements of statistical learning. These approach is suitable for bag-of-words-data, can catch interaction of word-features and can be used for both regression and classification. 

A: There is a type of Bayesian linear regression that handles the case of many features.  It is called latent factor regression and you can find a good description in the paper Bayesian Factor Regression Models
in the "Large p, Small n" Paradigm.  If the number of latent factors is large, then it is equivalent to linear regression.  Otherwise, it encourages the regression to follow the principal components of the features.
