  The article presents Harbsafe-162, a domain-specific data set for evaluating
distributional semantic models. It originates from a cooperation by Technische
Universit\"at Braunschweig and the German Commission for Electrical, Electronic
& Information Technologies of DIN and VDE, the Harbsafe project. One objective
of the project is to apply distributional semantic models to terminological
entries, that is, complex lexical data comprising of at least one or several
terms, term phrases and a definition. This application is needed to solve a
more complex problem: the harmonization of terminologies of standards and
standards bodies (i.e. resolution of doublettes and inconsistencies). Due to a
lack of evaluation data sets for terminological entries, the creation of
Harbsafe-162 was a necessary step towards harmonization assistance.
Harbsafe-162 covers data from nine electrotechnical standards in the domain of
functional safety, IT security, and dependability. An intrinsic evaluation
method in the form of a similarity rating task has been applied in which two
linguists and three domain experts from standardization participated. The data
set is used to evaluate a specific implementation of an established sentence
embedding model. This implementation proves to be satisfactory for the
domain-specific data so that further implementations for harmonization
assistance may be brought forward by the project. Considering recent criticism
on intrinsic evaluation methods, the article concludes with an evaluation of
Harbsafe-162 and joins a more general discussion about the nature of similarity
rating tasks. Harbsafe-162 has been made available for the community.
