  Real-time Digital Twinning of physical world scenes onto the Metaverse is
necessary for a myriad of applications such as augmented-reality (AR) assisted
driving. In AR assisted driving, physical environment scenes are first captured
by Internet of Vehicles (IoVs) and are uploaded to the Metaverse. A central
Metaverse Map Service Provider (MMSP) will aggregate information from all IoVs
to develop a central Metaverse Map. Information from the Metaverse Map can then
be downloaded into individual IoVs on demand and be delivered as AR scenes to
the driver. However, the growing interest in developing AR assisted driving
applications which relies on digital twinning invites adversaries. These
adversaries may place physical adversarial patches on physical world objects
such as cars, signboards, or on roads, seeking to contort the virtual world
digital twin. Hence, there is a need to detect these physical world adversarial
patches. Nevertheless, as real-time, accurate detection of adversarial patches
is compute-intensive, these physical world scenes have to be offloaded to the
Metaverse Map Base Stations (MMBS) for computation. Hence in our work, we
considered an environment with moving Internet of Vehicles (IoV), uploading
real-time physical world scenes to the MMBSs. We formulated a realistic joint
variable optimization problem where the MMSPs' objective is to maximize
adversarial patch detection mean average precision (mAP), while minimizing the
computed AR scene up-link transmission latency and IoVs' up-link transmission
idle count, through optimizing the IoV-MMBS allocation and IoV up-link scene
resolution selection. We proposed a Heterogeneous Action Proximal Policy
Optimization (HAPPO) (discrete-continuous) algorithm to tackle the proposed
problem. Extensive experiments shows HAPPO outperforms baseline models when
compared against key metrics.
