  A natural medium for wave propagation comprises a coupled bounded
heterogeneous region and an unbounded homogeneous free-space. Frequency-domain
wave propagation models in the medium, such as the variable coefficient
Helmholtz equation, include a faraway decay radiation condition (RC). It is
desirable to develop algorithms that incorporate the full physics of the
heterogeneous and unbounded medium wave propagation model, and avoid an
approximation of the RC. In this work we first present and analyze an
overlapping decomposition framework that is equivalent to the full-space
heterogeneous-homogenous continuous model, governed by the Helmholtz equation
with a spatially dependent refractive index and the RC. Our novel overlapping
framework allows the user to choose two free boundaries, and gain the advantage
of applying established high-order finite and boundary element methods (FEM and
BEM) to simulate an equivalent coupled model.
  The coupled model comprises auxiliary interior bounded heterogeneous and
exterior unbounded homogeneous Helmholtz problems. A smooth boundary can be
chosen for simulating the exterior problem using a spectrally accurate BEM, and
a simple boundary can be used to setup a high-order FEM for the interior
problem. Thanks to the spectral accuracy of the exterior computational model,
the resulting coupled system in the overlapping region is relatively very
small. Using the decomposed equivalent framework, we develop a novel
overlapping FEM-BEM algorithm for simulating the acoustic or electromagnetic
wave propagation in two dimensions. Our FEM-BEM algorithm for the full-space
model incorporates the RC exactly. Numerical experiments demonstrate the
efficiency of the FEM-BEM approach for simulating smooth and non-smooth wave
fields, with the latter induced by a complex heterogeneous medium and a
discontinuous refractive index.
