  We propose an alternative approach to L\"uscher's formula for extracting
two-body scattering phase shifts from finite volume spectra with no reliance on
the partial wave expansion. We use an effective-field-theory-based Hamiltonian
method in the plane wave basis and decompose the corresponding matrix elements
of operators into irreducible representations of the relevant point groups. The
proposed approach allows one to benefit from the knowledge of the long-range
interaction and avoids complications from partial wave mixing in a finite
volume. We consider spin-singlet channels in the two-nucleon system and
pion-pion scattering in the $\rho$-meson channel in the rest and moving frames
to illustrate the method for non-relativistic and relativistic systems,
respectively. For the two-nucleon system, the long-range interaction due to the
one-pion exchange is found to make the single-channel L\"uscher formula
unreliable at the physical pion mass. For S-wave dominated states, the
single-channel L\"uscher method suffers from significant finite-volume
artifacts for a $L=3$~fm box, but it works well for boxes with $L>5$~fm.
However, for P-wave dominated states, significant partial wave mixing effects
prevent the application of the single-channel L\"uscher formula regardless of
the box size (except for the near-threshold region). Using a toy model to
generate synthetic data for finite-volume energies, we show that our
effective-field-theory-based approach in the plane wave basis is capable of a
reliable extraction of the phase shifts. For pion-pion scattering, we employ a
phenomenological model to fit lattice QCD results at the physical pion mass.
The extracted P-wave phase shifts are found to be in a good agreement with the
experimental results.
