  There has been considerable interest in predicting human emotions and traits
using facial images and videos. Lately, such work has come under criticism for
poor labeling practices, inconclusive prediction results and fairness
considerations. We present a careful methodology to automatically derive social
skills of candidates based on their video response to interview questions. We,
for the first time, include video data from multiple countries encompassing
multiple ethnicities. Also, the videos were rated by individuals from multiple
racial backgrounds, following several best practices, to achieve a consensus
and unbiased measure of social skills. We develop two machine-learning models
to predict social skills. The first model employs expert-guidance to use
plausibly causal features. The second uses deep learning and depends solely on
the empirical correlations present in the data. We compare errors of both these
models, study the specificity of the models and make recommendations. We
further analyze fairness by studying the errors of models by race and gender.
We verify the usefulness of our models by determining how well they predict
interview outcomes for candidates. Overall, the study provides strong support
for using artificial intelligence for video interview scoring, while taking
care of fairness and ethical considerations.
