I'm doing clinical research in medicine and have taken several statistics courses. I've never published a paper using linear/logistic regression and would like to do variable selection correctly. Interpretability is important, so no fancy machine learning techniques. I've summarized my understanding of variable selection - would someone mind shedding light on any misconceptions? I found two (1) similar (2) CV posts to this one, but they didn't quite fully answer my concerns. Any thoughts would be much appreciated! I have 3 primary questions at the end.
Problem and Discussion
My typical regression/classification problem has 200-300 observations, an adverse event rate of 15% (if classification), and info on 25 out of 40 variables that have been claimed to have a "statistically significant" effect in the literature or make plausible sense by domain knowledge.
I put "statistically significant" in quotes, because it seems like everyone and their mother uses stepwise regression, but Harrell (3) and Flom (4) don't appear to like it for a number of good reasons. This is further supported by a Gelman blog post discussion (5). It seems like the only real time that stepwise is acceptable is if this is truly exploratory analysis, or one is interested in prediction and has a cross-validation scheme involved. Especially since many medical comorbidities suffer from collinearity AND studies suffer from small sample size, my understanding is that there will be a lot of false positives in the literature; this also makes me less likely to trust the literature for potential variables to include.
Another popular approach is to use a series of univariate regressions/associations between predictors and independent variable as a starting point. below a particular threshold (say, p < 0.2). This seems incorrect or at least misleading for the reasons outlined in this StackExchange post (6).
Lastly, an automated approach that appears popular in machine learning is to use penalization like L1 (Lasso), L2 (Ridge), or L1+L2 combo (Elastic Net). My understanding is that these do not have the same easy interpretations as OLS or logistic regression.
Gelman + Hill propose the following:
In my Stats course, I also recall using F tests or Analysis of Deviance to compare full and nested models to do model/variable selection variable by variable. This seems reasonable, but fitting sequential nested models systematically to find variables that cause largest drop in deviance per df seems like it could be easily automated (so I'm a bit concerned) and also seems like it suffers from problems of the order in which you test variable inclusion. My understanding is that this should also be supplemented by investigating multicollinearity and residual plots (residual vs. predicted).
Is the Gelman summary the way to go? What would you add or change in his proposed strategy?
Aside from purely thinking about potential interactions and transformations (which seems very bias/error/omission prone), is there another way to discover potential ones? Multivariate adaptive regression spline (MARS) was recommended to me, but I was informed that the nonlinearities/transformations don't translate into the same variables in a standard regression model.
Suppose my goal is very simple: say, "I'd like to estimate the association of X1 on Y, only accounting for X2". Is it adequate to simply regress Y ~ X1 + X2, report the outcome, without reference to actual predictive ability (as might be measured by cross-validation RMSE or accuracy measures)? Does this change depending on event rate or sample size or if R^2 is super low (I'm aware that R^2 is not good because you can always increase it by overfitting)? I am generally more interested in inference/interpretability than optimizing predictive power.
- "Controlling for X2, X1 was not statistically significantly associated with Y relative to X1's reference level." (logistic regression coefficient)
- "X1 was not a statistically significant predictor of Y since in the model drop in deviance was not enough relative to the change in df." (Analysis of Deviance)
Is cross-validation always necessary? In which case, one might also want to do some balancing of classes via SMOTE, sampling, etc.