  In contrast with imaging using position-resolving cameras, single-pixel
imaging uses a bucket detector along with spatially structured illumination for
image recovery. This emerging imaging technique is a promising candidate for a
broad range of applications due to high signal-to-noise ratio (SNR) and
sensitivity, and applicability in a wide range of frequency bands. Here,
inspired by single-pixel imaging in the spatial domain, we demonstrate a
temporal single-pixel imaging (TSPI) system that covers frequency bands
including both terahertz (THz) and near-infrared (NIR) region. By implementing
a programmable temporal fan-out (TFO) gate based on a digital micromirror
device (DMD), we can deterministically prepare temporally structured pulses
with a temporal sampling size down to 16.00$\pm$0.01 fs. By inheriting the
advantages in detection efficiency and sensitivity from spatial single-pixel
imaging, TSPI enables the compressive recovery of a 5 fJ THz pulse and two NIR
pulses with over 97$\%$ fidelity. We demonstrate that the TSPI is robust
against temporal distortions in the probe pulse train as well. As a direct
application, we apply TSPI to machine-learning-aided THz spectroscopy and
demonstrate a high sample identification accuracy (97.5$\%$) even under low
SNRs (SNR $\sim$ 10).
