  Large discrete action spaces (LDAS) remain a central challenge in
reinforcement learning. Existing solution approaches can handle unstructured
LDAS with up to a few million actions. However, many real-world applications in
logistics, production, and transportation systems have combinatorial action
spaces, whose size grows well beyond millions of actions, even on small
instances. Fortunately, such action spaces exhibit structure, e.g., equally
spaced discrete resource units. With this work, we focus on handling structured
LDAS (SLDAS) with sizes that cannot be handled by current benchmarks: we
propose Dynamic Neighborhood Construction (DNC), a novel exploitation paradigm
for SLDAS. We present a scalable neighborhood exploration heuristic that
utilizes this paradigm and efficiently explores the discrete neighborhood
around the continuous proxy action in structured action spaces with up to
$10^{73}$ actions. We demonstrate the performance of our method by benchmarking
it against three state-of-the-art approaches designed for large discrete action
spaces across two distinct environments. Our results show that DNC matches or
outperforms state-of-the-art approaches while being computationally more
efficient. Furthermore, our method scales to action spaces that so far remained
computationally intractable for existing methodologies.
