  In simultaneous speech translation, one can vary the size of the output
window, system latency and sometimes the allowed level of rewriting. The effect
of these properties on readability and comprehensibility has not been tested
with modern neural translation systems. In this work, we propose an evaluation
method and investigate the effects on comprehension and user preferences. It is
a pilot study with 14 users on 2 hours of German documentaries or speeches with
online translations into Czech. We collect continuous feedback and answers on
factual questions. Our results show that the subtitling layout or flicker have
a little effect on comprehension, in contrast to machine translation itself and
individual competence. Other results show that users with a limited knowledge
of the source language have different preferences to stability and latency than
the users with zero knowledge. The results are statistically insignificant,
however, we show that our method works and can be reproduced in larger volume.
