  In this paper, an IRS-aided integrated sensing and communications (ISAC)
system operating in the terahertz (THz) band is proposed to maximize the system
capacity. Transmit beamforming and phase-shift design are transformed into a
universal optimization problem with ergodic constraints. Then the joint
optimization of transmit beamforming and phase-shift design is achieved by
gradient-based, primal-dual proximal policy optimization (PPO) in the
multi-user multiple-input single-output (MISO) scenario. Specifically, the
actor part generates continuous transmit beamforming and the critic part takes
charge of discrete phase shift design. Based on the MISO scenario, we
investigate a distributed PPO (DPPO) framework with the concept of
multi-threading learning in the multi-user multiple-input multiple-output
(MIMO) scenario. Simulation results demonstrate the effectiveness of the
primal-dual PPO algorithm and its multi-threading version in terms of transmit
beamforming and phase-shift design.
