  Sentiment analysis is an important tool for aggregating patient voices, in
order to provide targeted improvements in healthcare services. A prerequisite
for this is the availability of in-domain data annotated for sentiment. This
article documents an effort to add sentiment annotations to free-text comments
in patient surveys collected by the Norwegian Institute of Public Health
(NIPH). However, annotation can be a time-consuming and resource-intensive
process, particularly when it requires domain expertise. We therefore also
evaluate a possible alternative to human annotation, using large language
models (LLMs) as annotators. We perform an extensive evaluation of the approach
for two openly available pretrained LLMs for Norwegian, experimenting with
different configurations of prompts and in-context learning, comparing their
performance to human annotators. We find that even for zero-shot runs, models
perform well above the baseline for binary sentiment, but still cannot compete
with human annotators on the full dataset.
