Construction of a Japanese Word Similarity Dataset
lrec2018.tex
%
% File acl2017.tex
%
%% Based on the style files for ACL-2015, with some improvements
%%  taken from the NAACL-2016 style
%% Based on the style files for ACL-2014, which were, in turn,
%% based on ACL-2013, ACL-2012, ACL-2011, ACL-2010, ACL-IJCNLP-2009,
%% EACL-2009, IJCNLP-2008...
%% Based on the style files for EACL 2006 by 
%%e.agirre@ehu.es or Sergi.Balari@uab.es
%% and that of ACL 08 by Joakim Nivre and Noah Smith

\documentclass[10pt,a4paper]{article}
\usepackage{hyperref}
\usepackage{lrec}

\usepackage{times}
\usepackage{latexsym}
\usepackage{booktabs}
\usepackage{CJKutf8}
\usepackage[whole,autotilde,substmingoth]{bxcjkjatype}
%\usepackage[hyphens]{url}
%\usepackage[dvipdfmx]{graphicx}
\usepackage{multirow}
\usepackage{array}
\usepackage[pdftex]{graphicx}

% taken from xample.tex
\usepackage{multibib}
\newcites{languageresource}{Language Resources}
\usepackage{tabularx}
\usepackage{soul}
\usepackage{epstopdf}
%\usepackage[latin1]{inputenc}
\usepackage{xstring}

\newcommand{\secref}[1]{\StrSubstitute{\getrefnumber{#1}}{.}{ }}

%\aclfinalcopy % Uncomment this line for the final submission
%\def\aclpaperid{3019} %  Enter the acl Paper ID here

%\setlength\titlebox{5cm}
% You can expand the titlebox if you need extra space
% to show all the authors. Please do not make the titlebox
% smaller than 5cm (the original size); we will check this
% in the camera-ready version and ask you to change it back.

\newcommand\BibTeX{B{\sc ib}\TeX}

\title{Construction of a Japanese Word Similarity Dataset}

\name{Yuya Sakaizawa\thanks{Currently at JustSystems Corporation.} and Mamoru Komachi}

\address{Tokyo Metropolitan University\\
	    6-6 Asahigaoka\\
	    Hino city, Tokyo 191-0065, Japan\\
	    {\tt \{sakaizawa-yuya@ed.,komachi@\}tmu.ac.jp}}

\abstract{
 An evaluation of distributed word representation is generally conducted using a word similarity task and/or a word analogy task.
 There are many datasets readily available for these tasks in English.
 However, evaluating distributed representation in languages that do not have such resources (e.g., Japanese) is difficult.
 Therefore, as a first step toward evaluating distributed representations in Japanese, we constructed a Japanese word similarity dataset.
 To the best of our knowledge, our dataset is the first resource that can be used to evaluate distributed representations in Japanese.
 Moreover, our dataset contains various parts of speech and includes rare
 words in addition to common words. \\ \newline \Keywords{word embeddings,
 distributed representation, word similarity}
}

\begin{document}

\maketitleabstract

\begin{figure*}[htbp]
	\begin{center}
		\newcolumntype{Y}{>{\centering\arraybackslash}p{22mm}} 
			\begin{tabular}{|Y|Y|Y|Y|Y|Y|Y|Y|} \hline
			\multirow{2}{15mm}{Sentence} & \multicolumn{5}{c|}{I don't think it is likely to not include these people, or [exclude]} \\ 
				 & \multicolumn{5}{c|}{\begin{CJK}{UTF8}{min}まさかこういった方々を対象としない、[排除する]わけではないと思いますが\end{CJK}} \\  \hline 
			\multirow{2}{15mm}{Paraphrase} & ignore & ostracize & avoid & exclude & remove  \\
				 & \begin{CJK}{UTF8}{min}無視する\end{CJK} & \begin{CJK}{UTF8}{min}排斥する\end{CJK} & \begin{CJK}{UTF8}{min}敬遠する\end{CJK} & \begin{CJK}{UTF8}{min}排除する\end{CJK} & \begin{CJK}{UTF8}{min}除外する\end{CJK}  \\ \hline
			 \end{tabular}
		 \caption{An example of the dataset from a previous study
         \citelanguageresource{kodaira}.}
		\label{tab:example}
	\end{center}
\end{figure*}

\section{Introduction}

Traditionally, a word is represented as a sparse vector indicating the word itself (one-hot vector) or the context of the word (distributional vector).
However, both the one-hot notation and  distributional notation suffer from data sparseness since dimensions of the word vector do not interact with each other.
Distributed word representation addresses the data sparseness problem by constructing a dense vector of a fixed length, wherein contexts are shared (or distributed) across dimensions.
Distributed word representation is known to improve the performance of many NLP applications such as machine translation \cite{chen-guo:2015:ACL-IJCNLP} and sentiment analysis \cite{DBLP:journals/corr/TaiSM15} to name a few.
The task to learn a distributed representation is called representation learning.

However, evaluating the quality of learned distributed word representation itself is not straightforward.
In language modeling, perplexity or cross-entropy is widely accepted as a de facto standard for intrinsic evaluation.
In contrast, distributed word representations include the additive (or compositional) property of the vectors, which cannot be assessed by perplexity.
Moreover, perplexity makes little use of infrequent words; thus, it is not appropriate for evaluating distributed presentations  that try to represent them.

Therefore, a word similarity task and/or a word analogy task are generally used to evaluate distributed word representations in the NLP literature.
The former judges whether distributed word representations improve modeling contexts, and the latter estimates how well the learned representations achieve the additive property.
However, such resources other than for English (e.g., Japanese) seldom exist.
In addition,  most of these datasets comprise high-frequency nouns so that they tend not to include other parts of speech.
Hence, previous data fail to evaluate word representations of other parts of speech, including content words such as verbs and adjectives.

To address the problem of the lack of a dataset for evaluating Japanese
distributed word representations, we propose to build a Japanese dataset for
the word similarity task.

The main contributions of our work are as follows:

\begin{itemize}
	\item To the best of our knowledge, it is the first work that constructs a Japanese word similarity dataset.
	\item The dataset contains various parts of speech and includes rare words in addition to common words.
\end{itemize}

\section{Related Work}

In general, distributed word representations are evaluated using a word similarity task.
For instance, WordSim353 \citelanguageresource{2002:PSC:503104.503110}, MC
\cite{strongContextualHypothesis}, RG
\cite{Rubenstein:1965:CCS:365628.365657}, and SCWS
\citelanguageresource{Huang:2012:IWR:2390524.2390645} have been used to evaluate word similarities in English.
Moreover, \newcite{baker-reichart-korhonen:2014:EMNLP2014}
built a verb similarity dataset (VSD) based on WordSim353 because there was no
dataset of verbs in the word-similarity task.
Recently, SimVerb-3500 was introduced to evaluate human understanding of verb
meaning \citelanguageresource{Gerz:2016:EMNLP}.
It provides human ratings for the similarity of 3,500 verb pairs so that
it enables robust evaluation of distributed representation for verbs.
However, most of these datasets include English words only. 
There has been no Japanese dataset for the word-similarity task.

Apart from English, WordSim353 and SimLex-999
\citelanguageresource{Hill:2015:CL} have been translated and
rescored in other languages: German, Italian and Russian
\citelanguageresource{Leviant:2015:arXiv}.
SimLex-999 has also been translated and rescored in Hebrew and Croatian
\citelanguageresource{Mrksic:2017:TACL}.
SimLex-999 explicitly targets at similarity rather than relatedness and
includes adjective, noun and verb pairs. However, this dataset contains
only frequent words.

In addition, the distributed representation of words is generally learned using only word-level information.
Consequently, the distributed representation for low-frequency words and unknown words cannot be learned well with conventional models.
However, low-frequency words and unknown words are often comprise high-frequency morphemes (e.g., unkingly $\rightarrow$ un + king + ly).
Some previous studies take advantage of the morphological information to provide a suitable representation for low-frequency words and unknown words \cite{Luong-etal:conll13:morpho,soricut-och:2015:NAACL-HLT}.
Morphological information is particularly important for Japanese since Japanese
is an agglutinative language.

\begin{table}[t]
  \begin{center}
    \begin{tabular}{ccccc} 
    \toprule
     Frequency & 1- & 101- & 1001- & 10001- \\
    \midrule
      Verb & 239 & 539 & 710 & 598  \\
      Adjective & 183 & 322 & 523 & 350 \\
      Noun & 15 & 63 & 172 & 258 \\
      Adverb & 23 & 75 & 80 & 81 \\
     \bottomrule
    \end{tabular}
    \caption{The number of parts of speech classified into each frequency.}
    \label{tab:freq}
  \end{center}
\end{table}

\section{Construction of a Japanese Word Similarity Dataset}

What makes a pair of words similar?
Most of the previous datasets do not concretely define the similarity of word pairs.
The difference in the similarity of word pairs originates from each annotator's mind, resulting in different scales of a word. 
Thus, we propose to use an example-based approach (Table \ref{tab:lancers}) to control the variance of the similarity ratings. 
We remove the context of word when we extracted the word.
So, we consider that an ambiguous word has high variance of the similarity, but we can get low variance of the similarity when the word is monosemous.
  
For this study, we constructed a Japanese word similarity dataset\footnote{\url{https://github.com/tmu-nlp/JapaneseWordSimilarityDataset}}.
We followed the procedure used to construct the Stanford Rare Word Similarity
Dataset (RW) \citelanguageresource{Luong-etal:conll13:morpho}.

We extracted Japanese word pairs from the Evaluation Dataset of Japanese
Lexical Simplification \citelanguageresource{kodaira}.
It targeted content words (nouns, verbs, adjectives, adverbs).
It included 10 contexts about target words annotated with their lexical substitutions and rankings.
Figure \ref{tab:example} shows an example of the dataset.
A word in square brackets in the text is represented as a target word of simplification.
A target word is not only recorded in the lemma form but also in the conjugated form.
We built a Japanese similarity dataset  from this dataset using the following procedure.

\paragraph{Word selection: }

First, paraphrase candidates were extracted from this dataset.
Because the construction process of the simplification dataset was divided into a paraphrase acquisition phase and a simplification ranking phase, we simply discarded the simplification rankings from the dataset to obtain paraphrase candidates.
Table \ref{tab:freq} shows the frequency of extracted words in the Japanese Wikipedia as of May 2015.
As shown in the table, low-frequency words are included in the dataset.

\paragraph{Pair construction: }

Because extracted words are annotated with their paraphrase candidates, we picked up each pair from the candidate as a word pair.
Consequently, we acquired 5,051 verb pairs, 4,033 adjective pairs, 1,528 noun pairs and 902 adverb pairs.
To balance the numbers of verb and adjective pairs with other parts of speech, we extracted samples at random for verbs and adjectives.
Finally, we obtained 1,464 verb pairs and 960 adjective pairs.

We observed that the similarity of the pairs extracted from the dataset of Kodaira et al. (2016) was low  without providing contexts; thus, we did not augment the dataset by inserting pseudo-negative instances from WordNet's synsets, as was done in the RW corpus.
Another reason why we did not employ the synset from the Japanese WordNet \cite{ISAHARA08.609} was because its quality was not as good as the English WordNet except for concrete nouns\footnote{It might be because it was translated from the English WordNet.
This is why we decided not to translate the existing English word similarity dataset to create a Japanese version.}.

\begin{table}
  \begin{center}
  \scalebox{0.75}{
    \begin{tabular}{cc|cc|c} 
    \toprule
      \multicolumn{2}{c|}{word 1} & \multicolumn{2}{c|}{word 2} & \multirow{2}{5mm}{sim.}  \\
      EN & JA & EN & JA  \\
     \midrule
      close & \begin{CJK}{UTF8}{min}瞑る\end{CJK} & close & \begin{CJK}{UTF8}{min}つぶる\end{CJK} & 10 \\
      erase & \begin{CJK}{UTF8}{min}拭き取る\end{CJK} & wipe & \begin{CJK}{UTF8}{min}拭う\end{CJK} & 8 \\
      mopey & \begin{CJK}{UTF8}{min}塞ぎ込んだ\end{CJK} & sick & \begin{CJK}{UTF8}{min}病んだ\end{CJK} & 5 \\
      investigate & \begin{CJK}{UTF8}{min}手探る\end{CJK} & go & \begin{CJK}{UTF8}{min}行く\end{CJK} & 2 \\
      fly & \begin{CJK}{UTF8}{min}とばせる\end{CJK} & control & \begin{CJK}{UTF8}{min}制御できる\end{CJK} & 0 \\
     \bottomrule
    \end{tabular}
    }
    \caption{Example of the degree of similarity when we requested annotation at Lancers.}
    \label{tab:lancers}
  \end{center}
\end{table}

\paragraph{Human judgment: }

We opted to use the crowd-sourcing service (Lancers\footnote{\url{http://www.lancers.jp}}) to hire native Japanese speakers.
We asked annotators to assign the degree of similarity for each pair using the
same 10-point scale\footnote{In a crowdsourcing request, we indicated that a
similarity of pairs with different notations, such as
``write\begin{CJK}{UTF8}{min}（書いた）\end{CJK}'' and
``write\begin{CJK}{UTF8}{min}（かいた）\end{CJK}'' is 10.}.
We used only those annotators who were able to complete at least 95\% of their previous assignments correctly.
We collected similarity rating for each word pair from ten annotators and defined the average of their annotations as the similarity of the pairs.


Although \citelanguageresource{kodaira} gave the annotators the context during annotation, we removed the context and gave only pairs to annotators.
We did so because the previous datasets such as VSD and RW did not present any context during annotation\footnote{Another reason why we did not do so is because the SCWS has a very high variance even though it is annotated with contexts (Table 5).}.
To improve the quality of the annotation, we presented an example of the degree of similarity of the pairs during annotation (Table \ref{tab:lancers}).
Consequently, we collected 4,851 pairs overall.
Table \ref{tab:lanexample} shows an example of a pair from our dataset.
Inter-annotator agreements (IAA) of each POS are shown in Table \ref{tab:IAA}.   
The inter-annotator agreement is the average Spearman's $\rho$ between a single annotator and the average of all others.

\begin{table}
  \begin{center}
    \begin{tabular}{ccccc} 
    \toprule
        POS & verb & adj & adv & noun \\
     \midrule
	IAA & 0.69 & 0.67 & 0.61 & 0.56 \\
     \bottomrule
    \end{tabular}
    \caption{Inter-annotator agreements of each POS.}
    \label{tab:IAA}
  \end{center}
\end{table}

\begin{table*}[htbp]
  \begin{center}
    \begin{tabular}{p{8mm}p{3mm}cccccc} 
    \toprule
      \multirow{2}{13mm}{word 1} & EN & follow & exclude & challenge & storm & elucidate &  wander \\
                                           & JA  & \begin{CJK}{UTF8}{min}受け継ぐ\end{CJK} & \begin{CJK}{UTF8}{min}除外する\end{CJK} & \begin{CJK}{UTF8}{min}チャレンジする\end{CJK} & \begin{CJK}{UTF8}{min}しける\end{CJK} & \begin{CJK}{UTF8}{min}明白になる\end{CJK} & \begin{CJK}{UTF8}{min}迷う\end{CJK} \\
      \multirow{2}{13mm}{word 2} & EN & inherit & remove & wish & rough & reflect &  stop \\
                                           & JA  & \begin{CJK}{UTF8}{min}継承する\end{CJK} & \begin{CJK}{UTF8}{min}除去する\end{CJK} & \begin{CJK}{UTF8}{min}望む\end{CJK} & \begin{CJK}{UTF8}{min}あれる\end{CJK} & \begin{CJK}{UTF8}{min}反映される\end{CJK} & \begin{CJK}{UTF8}{min}止める\end{CJK} \\ 
      \multicolumn{2}{c}{similarity}  & 9.3 & 7.3 & 6.0 & 5.7 & 2.7 & 1.7 \\
     \bottomrule
    \end{tabular}
    \caption{Examples of verb pairs in our dataset.
     The similarity rating is the average of the ratings from ten annotators. }
    \label{tab:lanexample}
  \end{center}
\end{table*}

\section{Discussion}

\subsection{Comparison to Other Datasets}

Table \ref{tab:result} shows how several resources vary.
WordSim353 comprises high-frequency words and so the variance tends to be low.
In contrast, RW includes low-frequency words, unknown words, and complex words composed of several morphemes; thus, the variance is large.
VSD has many polysemous words, which increase the variance.
Despite the fact that our dataset, similar to the VSD and RW datasets, contains low-frequency and ambiguous words, its variance is 3.00.
The variance level is low compared with the other corpora.
We considered that the examples of the similarity in the task request reduced the variance level.

\begin{table}[t]
  \begin{center}
    \begin{tabular}{cc} 
    \toprule
      Dataset & Variance  \\
     \midrule
      WordSim353 & 3.16 \\
      VSD & 4.76 \\
      RW  & 5.70 \\
      SCWS  & 8.60 \\
      JWSD (our dataset) & 3.00 \\
     \bottomrule
    \end{tabular}
    \caption{Variance of each dataset.}
    \label{tab:result}
  \end{center}
\end{table}

We did not expect SCWS to have the largest variance in the datasets shown in Table \ref{tab:result} because it gave the context to annotators during annotation.
At the beginning, we thought the context would serve to remove the ambiguity and clarify the meaning of word; however after looking into the dataset, we determined that the construction procedure used several extraordinary annotators.
It is crucial to filter insincere annotators and provide straightforward instructions to improve the quality of the similarity annotation like we did.

To gain better similarity, each dataset should utilize the reliability score to exclude extraordinary annotators.
For example, for SCWS, an annotator rating the similarity of pair of  ``CD'' and ``aglow'' assigned a rating of 10.
We  assumed it was a typo or misunderstanding regarding the words.
To address this problem, such an annotation should be removed before calculating the true similarity.
All the datasets except for RW simply calculated the average of the similarity, but datasets created using crowdsourcing should consider the reliability of the annotator.

\subsection{Analysis}

We present examples of a pair with high variance of similarity as shown below:

\paragraph{Aspect of relatedness.} (e.g., a pairing of
``fast\begin{CJK}{UTF8}{min}（速い）\end{CJK}'' and
``early\begin{CJK}{UTF8}{min}（早い）\end{CJK}''.) 

Although they are similar in meaning with respect to the time, they have
nothing in common with respect to speed; Annotator A assigned a rating of 10,
but Annotator B assigned a rating of 1.

Another example, the pairing of ``be
eager\mbox{\begin{CJK}{UTF8}{min}（懇願する）\end{CJK}''} and ``request\begin{CJK}{UTF8}{min}（頼む）\end{CJK}''.
Even though the act indicated by the two verbs is the same, there are some cases where they express different degrees of feeling.
Compared with ``request'', ``eager'' indicates a stronger feeling.
There were two annotators who emphasized the similarity of the act itself
rather than the different degrees of feeling, and vice versa.
In this case, Annotator A assigned a rating of 9, but Annotator B assigned a rating of 2.

Although it was necessary to distinguish similarity and semantic relatedness
\cite{Mrksic:2016:NAACL} and we asked annotators to rate the pairs based on
semantic similarity, it was not straightforward to put paraphrase candidates
onto a single scale considering all the attributes of the words.
This limitation might be relaxed if we would ask annotators to refer to a
thesaurus or an ontology such as Japanese Lexicon
\citelanguageresource{GoiTaikei:1997}.

\paragraph{Comparing spell\footnote{We indicated these pair's similarity is 10. However, some annotators ignored this instruction. It would be necessary to clean the spellings of paraphrase candidates before requesting similarity annotation.}.} (e.g., a pairing of ``slogan\begin{CJK}{UTF8}{min}（スローガン）\end{CJK}'' and ``slogan\begin{CJK}{UTF8}{min}（標語）\end{CJK}''.) 

In Japanese, we can write a word using hiragana, katakana, or kanji characters; however because hiragana and katakana represent only the pronunciation of a word, annotators might think of different words. 
In this case, Annotator A assigned a rating of 8, but Annotator B assigned a rating of 0.
Similarly, we confirmed the same thing in other parts of speech.
Especially, nouns can have several word pairs with different spellings, which
results in their IAA became too low compared to other parts of speech.

\paragraph{Frequency or time expressions.} (e.g.,  a pairing of ``often\begin{CJK}{UTF8}{min}（しばしば）\end{CJK}'' and ``frequently\begin{CJK}{UTF8}{min}（しきりに）\end{CJK}''.)

We confirmed that the variance becomes larger among adverbs expressing frequency.
This is due to the difference in the frequency of words that annotators imagines.
In this case, Annotator A assigned a rating of 9, but Annotator B assigned a rating of 0.
Similarly, we confirmed the same thing among adverbs expressing time.

\section{Conclusion}

In this study, we constructed the first Japanese word similarity dataset.
It contains various parts of speech and includes rare words in addition to common words.
Crowdsourced annotators assigned similarity to word pairs during the word similarity task.
We gave examples of similarity in the task request sent to annotators, so that we reduced the variance of each word pair.
However, we did not restrict the attributes of words, such as the level of feeling, during annotation.
Error analysis revealed that the notion of similarity should be carefully defined when constructing a similarity dataset.

As a future work, we plan to construct a word analogy dataset in Japanese by translating an English dataset to Japanese.
We hope that a Japanese database will facilitate research in Japanese distributed representations.

\section{Bibliographical References}
\bibliographystyle{lrec}
\bibliography{lrec2018}
%\nocite{*}

\section{Language Resource References}
\bibliographystylelanguageresource{lrec}
\bibliographylanguageresource{lrec2018}

\end{document}

languageresource.bbl
\begin{thebibliography}{}

\bibitem[\protect\citename{Finkelstein \bgroup et al.\egroup
  }2002]{2002:PSC:503104.503110}
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman,
  G., and Ruppin., E.
\newblock (2002).
\newblock {Placing Search in Context: The Concept Revisited}.
\newblock {\em ACM Transactions on Information Systems (TOIS)}, 20(1):116--131.

\bibitem[\protect\citename{Gerz \bgroup et al.\egroup }2016]{Gerz:2016:EMNLP}
Gerz, D., Vulic, I., Hill, F., Reichart, R., and Korhonen, A.
\newblock (2016).
\newblock {SimVerb-3500: {A} Large-Scale Evaluation Set of Verb Similarity}.
\newblock In {\em Proceedings of the 2016 Conference on Empirical Methods in
  Natural Language Processing (EMNLP)}, pages 2173--2182.

\bibitem[\protect\citename{Hill \bgroup et al.\egroup }2015]{Hill:2015:CL}
Hill, F., Reichart, R., and Korhonen, A.
\newblock (2015).
\newblock {Simlex-999: Evaluating Semantic Models with (Genuine) Similarity
  Estimation}.
\newblock {\em Computational Linguistics}, 41(4):665--695.

\bibitem[\protect\citename{Huang \bgroup et al.\egroup
  }2012]{Huang:2012:IWR:2390524.2390645}
Huang, E.~H., Socher, R., Manning, C.~D., and Ng, A.~Y.
\newblock (2012).
\newblock {Improving Word Representations via Global Context and Multiple Word
  Prototypes}.
\newblock In {\em Proceedings of the 50th Annual Meeting of the Association for
  Computational Linguistics (ACL)}, pages 873--882.

\bibitem[\protect\citename{Ikehara \bgroup et al.\egroup }1997]{GoiTaikei:1997}
Ikehara, S., Miyazaki, M., Shirai, S., Yokoo, A., Nakaiwa, H., Ogura, K.,
  Ooyama, Y., and Hayashi, Y.
\newblock (1997).
\newblock {\em {A Japanese Lexicon}}.
\newblock Iwanami Shoten.

\bibitem[\protect\citename{Kodaira \bgroup et al.\egroup }2016]{kodaira}
Kodaira, T., Kajiwara, T., and Komachi, M.
\newblock (2016).
\newblock {Controlled and Balanced Dataset for Japanese Lexical
  Simplification}.
\newblock In {\em Proceedings of the ACL 2016 Student Research Workshop}, pages
  1--7.

\bibitem[\protect\citename{Leviant and Reichart}2015]{Leviant:2015:arXiv}
Leviant, I. and Reichart, R.
\newblock (2015).
\newblock {Judgment Language Matters: Multilingual Vector Space Models for
  Judgment Language Aware Lexical Semantics}.
\newblock {\em CoRR}, abs/1508.00106.

\bibitem[\protect\citename{Luong \bgroup et al.\egroup
  }2013]{Luong-etal:conll13:morpho}
Luong, M.-T., Socher, R., and Manning, C.~D.
\newblock (2013).
\newblock {Better Word Representations with Recursive Neural Networks for
  Morphology}.
\newblock In {\em Proceedings of the Seventeenth Conference on Computational
  Natural Language Learning (CoNLL)}, pages 104--113.

\bibitem[\protect\citename{Mrksic \bgroup et al.\egroup
  }2017]{Mrksic:2017:TACL}
Mrksic, N., Vulic, I., S{\'{e}}aghdha, D.~{\'{O}}., Leviant, I., Reichart, R.,
  Gasic, M., Korhonen, A., and Young, S.~J.
\newblock (2017).
\newblock {Semantic Specialisation of Distributional Word Vector Spaces using
  Monolingual and Cross-Lingual Constraints}.
\newblock {\em Transactions of the Association for Computational Linguistics},
  5:309--324.

\end{thebibliography}

lrec2018.bbl
\begin{thebibliography}{}

\bibitem[\protect\citename{Baker \bgroup et al.\egroup
  }2014]{baker-reichart-korhonen:2014:EMNLP2014}
Baker, S., Reichart, R., and Korhonen, A.
\newblock (2014).
\newblock {An Unsupervised Model for Instance Level Subcategorization
  Acquisition}.
\newblock In {\em Proceedings of the 2014 Conference on Empirical Methods in
  Natural Language Processing (EMNLP)}, pages 278--289.

\bibitem[\protect\citename{Chen and Guo}2015]{chen-guo:2015:ACL-IJCNLP}
Chen, B. and Guo, H.
\newblock (2015).
\newblock {Representation Based Translation Evaluation Metrics}.
\newblock In {\em Proceedings of the 53rd Annual Meeting of the Association for
  Computational Linguistics and the 7th International Joint Conference on
  Natural Language Processing (ACL-IJCNLP)}, pages 150--155.

\bibitem[\protect\citename{Isahara \bgroup et al.\egroup }2008]{ISAHARA08.609}
Isahara, H., Bond, F., Uchimoto, K., Utiyama, M., and Kanzaki, K.
\newblock (2008).
\newblock {Development of the Japanese WordNet}.
\newblock In {\em Proceedings of the Sixth International Conference on Language
  Resources and Evaluation (LREC)}, pages 2420--2423.

\bibitem[\protect\citename{Luong \bgroup et al.\egroup
  }2013]{Luong-etal:conll13:morpho}
Luong, M.-T., Socher, R., and Manning, C.~D.
\newblock (2013).
\newblock {Better Word Representations with Recursive Neural Networks for
  Morphology}.
\newblock In {\em Proceedings of the Seventeenth Conference on Computational
  Natural Language Learning (CoNLL)}, pages 104--113.

\bibitem[\protect\citename{Miller and Charles}1991]{strongContextualHypothesis}
Miller, G.~A. and Charles, W.~G.
\newblock (1991).
\newblock {Contextual Correlates of Semantic Similarity}.
\newblock {\em Language and Cognitive Processes}, 6(1):1--28.

\bibitem[\protect\citename{Mrksic \bgroup et al.\egroup
  }2016]{Mrksic:2016:NAACL}
Mrksic, N., S{\'{e}}aghdha, D.~{\'{O}}., Thomson, B., Gasic, M.,
  Rojas{-}Barahona, L.~M., Su, P., Vandyke, D., Wen, T., and Young, S.~J.
\newblock (2016).
\newblock {Counter-fitting Word Vectors to Linguistic Constraints}.
\newblock In {\em Proceedings of the 2016 Conference of the North American
  Chapter of the Association for Computatinal Linguistics: Human Language
  Technologies (NAACL-HLT)}, pages 142--148.

\bibitem[\protect\citename{Rubenstein and
  Goodenough}1965]{Rubenstein:1965:CCS:365628.365657}
Rubenstein, H. and Goodenough, J.~B.
\newblock (1965).
\newblock {Contextual Correlates of Synonymy}.
\newblock {\em Communications of the ACM}, 8(10):627--633.

\bibitem[\protect\citename{Soricut and Och}2015]{soricut-och:2015:NAACL-HLT}
Soricut, R. and Och, F.
\newblock (2015).
\newblock {Unsupervised Morphology Induction Using Word Embeddings}.
\newblock In {\em Proceedings of the 2015 Conference of the North American
  Chapter of the Association for Computational Linguistics: Human Language
  Technologies (NAACL-HLT)}, pages 1627--1637.

\bibitem[\protect\citename{Tai \bgroup et al.\egroup
  }2015]{DBLP:journals/corr/TaiSM15}
Tai, K.~S., Socher, R., and Manning, C.~D.
\newblock (2015).
\newblock {Improved Semantic Representations From Tree-Structured Long
  Short-Term Memory Networks}.
\newblock In {\em Proceedings of the 53rd Annual Meeting of the Association for
  Computational Linguistics and the 7th International Joint Conference on
  Natural Language Processing (ACL-IJCNLP)}, pages 1556--1566.

\end{thebibliography}
