  Recurrent Neural Network, Long Short-Term Memory, and Transformer have made
great progress in predicting the trajectories of moving objects. Although the
trajectory element with the surrounding scene features has been merged to
improve performance, there still exist some problems to be solved. One is that
the time series processing models will increase the inference time with the
increase of the number of prediction sequences. Another lies in which the
features can not be extracted from the scene's image and point cloud in some
situations. Therefore, this paper proposes an Obstacle-Transformer to predict
trajectory in a constant inference time. An ``obstacle'' is designed by the
surrounding trajectory rather than images or point clouds, making
Obstacle-Transformer more applicable in a wider range of scenarios. Experiments
are conducted on ETH and UCY data sets to verify the performance of our model.
