  Most existing stylistic text rewriting methods and evaluation metrics operate
on a sentence level, but ignoring the broader context of the text can lead to
preferring generic, ambiguous, and incoherent rewrites. In this paper, we
investigate integrating the preceding textual context into both the
$\textit{rewriting}$ and $\textit{evaluation}$ stages of stylistic text
rewriting, and introduce a new composite contextual evaluation metric
$\texttt{CtxSimFit}$ that combines similarity to the original sentence with
contextual cohesiveness. We comparatively evaluate non-contextual and
contextual rewrites in formality, toxicity, and sentiment transfer tasks. Our
experiments show that humans significantly prefer contextual rewrites as more
fitting and natural over non-contextual ones, yet existing sentence-level
automatic metrics (e.g., ROUGE, SBERT) correlate poorly with human preferences
($\rho$=0--0.3). In contrast, human preferences are much better reflected by
both our novel $\texttt{CtxSimFit}$ ($\rho$=0.7--0.9) as well as proposed
context-infused versions of common metrics ($\rho$=0.4--0.7). Overall, our
findings highlight the importance of integrating context into the generation
and especially the evaluation stages of stylistic text rewriting.
