  Personality plays a pivotal role in our understanding of human actions and
behavior. Today, the applications of personality are widespread, built on the
solutions from psychology to infer personality. In software engineering, for
instance, one widely used solution to infer personality uses textual
communication data. As studies on personality in software engineering continue
to grow, it is imperative to understand the performance of these solutions.
This paper compares the inferential ability of three widely studied text-based
personality tests against each other and the ground truth on GitHub. We explore
the challenges and potential solutions to improve the inferential ability of
personality tests. Our study shows that solutions for inferring personality are
far from being perfect. Software engineering communications data can infer
individual developer personality with an average error rate of 41%. In the best
case, the error rate can be reduced up to 36% by following our recommendations.
