  We propose the $\mathbb{Z}_Q$ Berry phase as a topological invariant for
higher-order symmetry-protected topological (HOSPT) phases for two- and
three-dimensional systems. It is topologically stable for electron-electron
interactions assuming the gap remains open. As a concrete example, we show that
the Berry phase is quantized in $\mathbb{Z}_4$ and characterizes the HOSPT
phase of the extended Benalcazar-Bernevig-Hughes (BBH) model, which contains
the next-nearest neighbor hopping and the intersite Coulomb interactions. In
addition, we introduce the $\mathbb{Z}_4$ Berry phase for the spin-model-analog
of the BBH model, whose topological invariant has not been found so far.
Furthermore, we demonstrate the Berry phase is quantized in $\mathbb{Z}_4$ for
the three-dimensional version of the BBH model. We also confirm the bulk-corner
correspondence between the $\mathbb{Z}_4$ Berry phase and the corner states in
the HOSPT phases.
