  End-to-end (E2E) speech-to-text translation (ST) often depends on pretraining
its encoder and/or decoder using source transcripts via speech recognition or
text translation tasks, without which translation performance drops
substantially. However, transcripts are not always available, and how
significant such pretraining is for E2E ST has rarely been studied in the
literature. In this paper, we revisit this question and explore the extent to
which the quality of E2E ST trained on speech-translation pairs alone can be
improved. We reexamine several techniques proven beneficial to ST previously,
and offer a set of best practices that biases a Transformer-based E2E ST system
toward training from scratch. Besides, we propose parameterized distance
penalty to facilitate the modeling of locality in the self-attention model for
speech. On four benchmarks covering 23 languages, our experiments show that,
without using any transcripts or pretraining, the proposed system reaches and
even outperforms previous studies adopting pretraining, although the gap
remains in (extremely) low-resource settings. Finally, we discuss neural
acoustic feature modeling, where a neural model is designed to extract acoustic
features from raw speech signals directly, with the goal to simplify inductive
biases and add freedom to the model in describing speech. For the first time,
we demonstrate its feasibility and show encouraging results on ST tasks.
