  Recently, an open geometry Fourier modal method based on a new combination of
an open boundary condition and a non-uniform $k$-space discretization was
introduced for rotationally symmetric structures providing a more efficient
approach for modeling nanowires and micropillar cavities [J. Opt. Soc. Am. A
33, 1298 (2016)]. Here, we generalize the approach to three-dimensional (3D)
Cartesian coordinates allowing for the modeling of rectangular geometries in
open space. The open boundary condition is a consequence of having an infinite
computational domain described using basis functions that expand the whole
space. The strength of the method lies in discretizing the Fourier integrals
using a non-uniform circular "dartboard" sampling of the Fourier $k$ space. We
show that our sampling technique leads to a more accurate description of the
continuum of the radiation modes that leak out from the structure. We also
compare our approach to conventional discretization with direct and inverse
factorization rules commonly used in established Fourier modal methods. We
apply our method to a variety of optical waveguide structures and demonstrate
that the method leads to a significantly improved convergence enabling more
accurate and efficient modeling of open 3D nanophotonic structures.
