  Sequence labeling is a core task in text understanding for IE/IR systems.
Text generation models have increasingly become the go-to solution for such
tasks (e.g., entity extraction and dialog slot filling). While most research
has focused on the labeling accuracy, a key aspect -- of vital practical
importance -- has slipped through the cracks: understanding model confidence.
More specifically, we lack a principled understanding of how to reliably gauge
the confidence of a model in its predictions for each labeled span. This paper
aims to provide some empirical insights on estimating model confidence for
generative sequence labeling. Most notably, we find that simply using the
decoder's output probabilities \textbf{is not} the best in realizing
well-calibrated confidence estimates. As verified over six public datasets of
different tasks, we show that our proposed approach -- which leverages
statistics from top-$k$ predictions by a beam search -- significantly reduces
calibration errors of the predictions of a generative sequence labeling model.
