  In subgroup analysis, testing the existence of a subgroup with a differential
treatment effect serves as protection against spurious subgroup discovery.
Despite its importance, this hypothesis testing possesses a complicated nature:
parameter characterizing subgroup classification is not identified under the
null hypothesis of no subgroup. Due to this irregularity, the existing methods
have the following two limitations. First, the asymptotic null distribution of
test statistics often takes an intractable form, which necessitates
computationally demanding resampling methods to calculate the critical value.
Second, the dimension of personal attributes characterizing subgroup membership
is not allowed to be of high dimension. To solve these two problems
simultaneously, this study develops a novel shrinkage likelihood ratio test for
the existence of a subgroup using a logistic-normal mixture model. The proposed
test statistics are built on a modified likelihood function that shrinks
possibly high-dimensional unidentified parameters toward zero under the null
hypothesis while retaining power under the alternative. This shrinkage helps
handle the irregularity and restore the simple chi-square-type asymptotics even
under the high-dimensional regime.
