Q: Predictive power (or $R^2$) adjusted for certain variables I will frame this question for Ordinary Least Square (OLS) regression, but my question is for both OLS and Logistic.
Let's say we data over 10000 different individuals. For each person we have three variables - $x_1$, $x_2$, $x_3$ and a target variable $y$ which is how much they have spent on the product.
Let's say that the data spans over two products, $p=1$ and $2$.
Objective
Create a prediction model for $y$ (spend) using $x_1$, $x_2$, $x_3$, which should be product agnostic.
I will use the model only to determine top spenders. Hence I will be happy as long as the model rank orders (i.e. it gives higher score to person A than B if person A indeed is likely to spend more), and am not concerned about the point estimate's accuracy much.
Approach 1
Since there are two products and one product may have intrinsic higher spend (eg. offers more reward points), I will introduce a dummy variable, $d=1_{p=1}$.
I will build a model $y=a_0+a_1x_1+a_2x_2+a_3x_3+a_4d+\epsilon$
I will then ignore $a_4$, i.e. set it to zero while scoring since I want product agnostic rank ordering.
Approach 2
Don't use the product dummy.
I will build a model $y=a_0+a_1x_1+a_2x_2+a_3x_3+\epsilon$
Which approach is better? I feel Approach 1 should do a better job, since it adjusts for intrinsic difference in spend across products, which I want to discount in the analysis.
Metrics like $R^2$ will definitely be more in raw form for Approach 1, since there is an additional variable. However since I cannot use the dummy while scoring, I would like to discount its effect on measuring predictive power. How can we characterize the power of the model discounting the effect of dummy in Approach 1?
