  Identifying subgroups, which respond differently to a treatment, both in
terms of efficacy and safety, is an important part of drug development. A
well-known challenge in exploratory subgroup analyses is the small sample size
in the considered subgroups, which is usually too low to allow for definite
comparisons. In early phase trials this problem is further exaggerated, because
limited or no clinical prior information on the drug and plausible subgroups is
available. We evaluate novel strategies for treatment effect estimation in
these settings in a simulation study motivated by real clinical trial
situations. We compare several approaches to estimate treatment effects for
selected subgroups, employing model averaging, resampling and Lasso regression
methods. Two subgroup identification approaches are employed, one based on
categorization of covariates and the other based on splines. Our results show
that naive estimation of the treatment effect, which ignores that a selection
has taken place, leads to bias and overoptimistic conclusions. For the
considered simulation scenarios virtually all evaluated novel methods provide
more adequate estimates of the treatment effect for selected subgroups, in
terms of bias, MSE and confidence interval coverage.
