  An interesting method of evaluating word representations is by how much they
reflect the semantic representations in the human brain. However, most, if not
all, previous works only focus on small datasets and a single modality. In this
paper, we present the first multi-modal framework for evaluating English word
representations based on cognitive lexical semantics. Six types of word
embeddings are evaluated by fitting them to 15 datasets of eye-tracking, EEG
and fMRI signals recorded during language processing. To achieve a global score
over all evaluation hypotheses, we apply statistical significance testing
accounting for the multiple comparisons problem. This framework is easily
extensible and available to include other intrinsic and extrinsic evaluation
methods. We find strong correlations in the results between cognitive datasets,
across recording modalities and to their performance on extrinsic NLP tasks.
