  Diffusion models are promising for joint trajectory prediction and
controllable generation in autonomous driving, but they face challenges of
inefficient inference steps and high computational demands. To tackle these
challenges, we introduce Optimal Gaussian Diffusion (OGD) and Estimated Clean
Manifold (ECM) Guidance. OGD optimizes the prior distribution for a small
diffusion time $T$ and starts the reverse diffusion process from it. ECM
directly injects guidance gradients to the estimated clean manifold,
eliminating extensive gradient backpropagation throughout the network. Our
methodology streamlines the generative process, enabling practical applications
with reduced computational overhead. Experimental validation on the large-scale
Argoverse 2 dataset demonstrates our approach's superior performance, offering
a viable solution for computationally efficient, high-quality joint trajectory
prediction and controllable generation for autonomous driving. Our project
webpage is at https://yixiaowang7.github.io/OptTrajDiff_Page/.
