PR2: A Language Independent Unsupervised Tool for Personality Recognition from Text
Fabio Celli
CLIC-CIMeC
University of Trento
fabio.celli@unitn.it
We present PR2, a personality recognition system available online, that performs instance-based classification of Big5 personality types from unstructured text, using language-independent features. It has been tested on English and Italian, achieving performances up to f=.68.\\
14fc-pr2_arxiv.tex
\documentclass[11pt]{article}
\usepackage{acl-hlt2011}
\usepackage{times}
\usepackage{graphicx}  %%% for including graphics
\usepackage{url}       %%% for including URLs
\usepackage{latexsym}
\usepackage{amsmath}
\usepackage{multirow}
\DeclareMathOperator*{\argmax}{arg\,max}
\setlength\titlebox{6.5cm}  
\long\def\/*#1*/{}

\begin{document}

\title{PR2: A Language Independent Unsupervised Tool for Personality Recognition from Text}
\author{Fabio Celli \\ CLIC-CIMeC, University of Trento \\ \texttt{fabio.celli@unitn.it}\\
\And Massimo Poesio\\University of Essex\\ \texttt{poesio@essex.ac.uk}}
\maketitle
\begin{abstract}
We present PR2, a personality recognition system available online, that performs instance-based classification of Big5 personality types from unstructured text, using language-independent features. It has been tested on English and Italian, achieving performances up to f=.68.\\
\end{abstract}

\section{Introduction and Background}
Personality is an affect processing system that describes persistent human behavioural responses to broad classes of environmental stimuli (Adelstein et al. 2011)% \cite{adelstein.al11}
. It has been formalized in various ways, such as the Myers-Briggs type indicator (Briggs \& Myers 1980)%\cite{briggs.myers80}
, that defines four personality types, and the Big5 (Costa \& MacCrae 1992)%\cite{costa.maccrae85}
, that defines five bipolar traits, is widely used in the scientific community, and can be assessed by means of different questionnaires. 
The Big5 traits are: extroversion, emotional stability/neuroticism, agreeableness, conscientiousness and openness to experience. %Extraversion describes a person along the two opposite poles of sociability and shyness. Emotional stability, which is sometimes referred by its negative pole (neuroticism), describes the modality of impulse control along a scale that goes from control (a calm and stable person) to instability (an anxious and neurotic person). Agreeableness refers to the tendency to be sympathetic and cooperative towards others, rather than suspicious and antagonistic. Conscientiousness describes a person in terms of self-discipline versus disorganization. Openness to experience refers to the tendency to be creative and curious rather than unimaginative.\\ 
Written text, as well as speech, conveys a lot of information about author's personality (Mairesse et al. 2007)%\cite{mairesse.al07}
. Personality Recognition from Text is a NLP task, partially connected to stylometry (Luyckx \& Daelemans 2008)%\cite{luyckx.daelemans08}
, that consists in the automatic classification of authors' personality traits from linguistic features. Useful applications of this task range from deception detection (Fornaciari et al. 2013) %\cite{fornaciari.al13} 
to social network analysis (Celli \& Polonio 2013), and potentially many NLP tasks, such as opinion mining. In order to find new applications of personality in language, we present PR2, a personality recognition tool for NLP available online\footnote{http://clic.cimec.unitn.it/fabio/pr2demo.php}.\\ 
\indent The paper is structured as follows: in section \ref{prev} we introduce previous work and the problems we see in personality recognition from text. In section \ref{sys} and \ref{exp} we describe our tool for personality recognition and the experiments to test its performance. In section \ref{end} we draw some conclusions.

\section{Previous Work, Problems and Perspectives}\label{prev}
After the first pioneering works in personality recogniton in blogs and offline texts (Oberlander \& Nowson 2006;% \cite{oberlander.nowson06}; 
Mairesse et al 2007%\cite{mairesse.al07}
), recently there has been an increasing interest in the extraction of personality from social networks and in languages different from English (Kermanidis 2012; %\cite{kermanidis12}
Bai et al 2012; %\cite{bai.al12}; 
Quercia et al. 2011; %\cite{quercia.al11}; 
Golbeck et al 2011%\cite{golbeck.al11}
). \\%It is not easy to determine the state of the art in personality recognition, because almost all scholars used their own corpora and evaluation metrics.
%In a comparison of different approaches on the same datasets, Celli et al. 2013 %\cite{celli.al13} reported that almost all the systems achieved performances around f=.058 (averaged over the five traits), in line with the results obtained by Mairesse et al. 2007%\cite{mairesse.al07} . Ensemble methods (Verhoeven et al 2013) %\cite{verhoeven.al13} and feature selection over a very large feature space (Markovikj et al 2013) %\cite{markovikj.al13} seem to be the most effective techniques, boosting performances up to f=.72 and f=.902.\\
\indent Almost all the approaches to personality recognition from text are supervised. The main problems with this approach are the limitations in data annotation and language dependency. Data labeled with personality types are usually costly and time consuming to collect. In addition, when models are trained on a specific domain or language, are not very effective if used on different domains. The language problem also concerns the language dependency of the resources, such as LIWC (Tausczik \& Pennebaker 2010) %\cite{tausczik.pennebaker10} 
and MRC (Coltheart 1981). %\cite{coltheart81}
These are clear limitations in the exploitation of personality recognition from text in other NLP tasks.\\
\indent Many scholars working in the field of personality recognition from text reported correlations between linguistic cues and personality traits (Mairesse et al 2007; Iacobelli et al 2011; Quercia et al 2011; Golbeck et al. 2011) %\cite{mairesse.al07}, \cite{iacobelli.al11}, \cite{quercia.al11}, \cite{golbeck.al11} 
that can be exploited as models for unsupervised classification (Celli 2012). %\cite{celli12}.
Unsupervised personality recognition has some advantages: they require very few labeled data (mainly for validation) and are potentially language-independent. In the next section we describe our system for Unsupervised personality recognition from text.

\section{PR2: System description, Feature Set and Parameters}\label{sys}
PR2 is a personality recognition tool written in Perl and available online as a demo. It performs instance-based classification of Big5 personality types in an unsupervised way, using language-independent features (see table \ref{feat.c}). The system takes as input unlabeled text and  authors in a tab separated format, with authors in the first column and text in the second one. Examples of the input format are provided on the website. Big5 personality labels are formalized as 5-characters strings, each one representing one trait of the Big5. Each character in the string can take 3 possible values: positive (y), negative (n) and omitted/balanced (o). For example ``ynoon'' stands for an extrovert neurotic and not open mindend person. \\
\indent As initial feature set, the system exploits language-independent features extracted from LIWC %\cite{tausczik.pennebaker10} 
and MRC%\cite{coltheart81}
, whose correlations to personality are reported in Mairesse et al. 2007% \cite{mairesse.al07}
. 
\begin{table}[ht]
\centering \small
\begin{tabular}{lccccc}
\hline
feature & ext. & emo. & agr. & con. & ope.\\
\hline
ap & -.08** & -.04 & -.01 & -.04 & -10** \\
em &  -.00 & -.05* & .06** & .00 & -.03 \\
nb &  -.03 & .05* & -.03 & -.02 & -.06** \\
pa &  -.06** & .03 & -.04* & -.01 & .10** \\
qm &  -.06** & -.05* & -.04 & -.06** & .08** \\
qt &  -.05* & -.02 & -.01 & -.03 & .09** \\
tt & -.05** & .10** & -.04* & -.05* & .09**\\
wf & .05* & -.06** & .03* & .06** & .05**\\
\hline
\end{tabular}
\caption{\small{Feature/correlations set, adapted from Mairesse et Al. 2007.  * = \textit{p} smaller than .05 (weak correlation), ** = \textit{p} smaller than .01 (strong correlation).} \label{feat.c}}
\smallskip
\end{table} 
The features are: punctuation (ap); question marks (qm); quotes (qt); exclamation marks (em); numbers (nb); parentheses (pa); repetition ratio (tt), word frequency (wf).\\
\indent The pipeline of the personality recognition system has three phases. In the preprocessing phase the system filters repeated texts and sorts the authors that have more than one text. Then it samples 20\% of the input unlabeled data, assigns personality labels according to the correlations and computes the distribution of each feature of the correlation set and their firing rate. 
 %to perform bootstrapping. New instances that we exploited for domain adaptation, and also to the purpose to prevent overfitting.\\
\begin{figure}[h]
\begin{center}
\includegraphics[width=7.2cm]{phd-aprt-improve3.png}
\caption{\small{System pipeline.}} \label{spipe}
\end{center}
\end{figure}
In the processing phase, the system generates one personality hypothesis for each text in the dataset, mapping the features in the correlation set to specific personality trait poles, according to the correlations. Instances are compared to the average of the population sampled during the preprocessing phase and filtered accordingly: only feature values above the average are mapped to personality traits. For example a text containing more punctuation than average will fire negative correlations with extraversion and openness to experience (see table \ref{feat.c}). The system keeps track of the firing rate of each single feature/correlation and computes personality scores for each trait, mapping positive scores into ``y'', negative scores into ``n'' and zeros into ``o'' labels. In this phase the system computes also per-trait confidence, defined as the coverage of the majority label over all the labels generated for the author's texts. $ \textit{conf}=\frac{\textit{m}}{\textit{T}}$ where \textit{m} is the count of the majority label, and \textit{T} is the count of the author's texts. In the evaluation phase the system compares all the personality hypotheses generated for each single text of each author and retrieves one generalized hypothesis per author by computing the majority class for each trait. In the evaluation phase the system computes average confidence and variability. Average Confidence is derived from per-trait confidence scores and gives a measure of the robustness of the personality hypothesis. Variability (var) gives information about how much one author tends to write expressing the same personality traits in all the texts. It is defined as $ var=\frac{\textit{avg conf}}{\textit{T}}\label{v} $ where \textit{avg conf} is the confidence averaged over the five traits and \textit{T} is the count of all author's texts. The outputs of the system are: 1) list of authors with personality labels, confidence, number of texts and variability; 2) input text annotated with personality labels in a tab-separated format.\\
\indent The system provides the following optional parameters:
%\textbf{feature normalization (a)}. This option normalizes authors' personality scores by number of texts, allowing a better comparison of the authors. 
\textbf{weigthed correlations (w)}. When the parameter is activated, high feature firing rates, computed on the fly during the processing phase, decrease the personality score associated to that feature. This parameter boosts the infomation provided by low-frequency features.  
\textbf{variable hypothesis average (v)}. If this parameter is activated, the average distribution of each feature is recomputed on the fly during the processing phase. This allows to fit the data at hand but increases the error rate on the first instances processed. 
\textbf{hypotheses normalization (n)}. If this parameter is activated, the system normalizes hypotheses scores during the processing phase, allowing a better comparison of the authors. If paired with variable hypothesis average, these parameters force binary classification (no ``o'' labels), boosting recall. 
\textbf{weak traits correction (r)}. When the distribution of specific features is paricularly skewed, the system might generate labels just for one class. If this parameter is activated, the system detects skewed distributions for particular traits during the preprocessing phase and randomizes the personality score associated to those traits. This parameter can be useful to prevent errors with very small datasets. 
\textbf{pattern extraction (t)}. If this parameter is activated, the system automatically extracts new patterns from the data at hand, filtering out labels with low confidence, associates them to personality traits, and uses them as new correlations between patterns and personality traits. 

\section{Experiments: Testing the System}\label{exp}
We tested the system on two datasets different in size, domain and language: Essays (Mairesse et al 2007%\cite{mairesse.al07}
), a large collection of essays written in English, and PersFB (Celli \& Polonio 2013%\cite{celli.polonio13}
), a small dataset of Facebook posts in Italian. We split Essays by lines in order to have more or less the same length per each texts in the two datasets (19 words per text in Essays, and 12 words per text in PersFB). As test sets, we used a sample of 250 authors for Essays and about 25 authors for PersFB. Patterns (parameter t) were extracted from larger sets: 2000 authors for Essays, and about 1000 authors for PersFB.
\begin{table}[ht]
\centering \small
\begin{tabular}{lllll}
\hline
dataset & par & p & r & f\\
\hline
mbl-fb-it & - & .437 & 1 & .608 \\
fb-it & n  &  .477  & .855  & .612\\
fb-it & nw  &  .492  & .87  & .629\\ 
fb-it & nr  &  .478  & .86  & .614\\
fb-it & nv  &  .472  & 1  & .641\\
fb-it & nvr  &  .493  & 1  & .661\\
fb-it & tnvr & .534 & 1 & \textbf{.686}\\
%fb-it & nvrk=2  &  .483  & 1  & .651\\
%fb-it & nvrak=2 & .462  & 1 & .632 \\
\hline
mbl-es-en & - & .487 & 1 & .655 \\
es-en & n  &  .544  & .861  & .667\\
es-en & nw  &  .525  & .855  & .651\\
es-en & nr  &  .549  & .908  & .684\\
es-en & nv  &  .537  & 1  & \textbf{.699}\\
es-en & nvr  &  .536  & 1  & .698\\
es-en & tnvr & .523 & 1 & \textbf{.686}\\
%es-en & nvrk=2  &  .517  & 1  & .682\\
%es-en & nvrak=2 &  .553  & 1  & \textbf{.712}\\
\hline
\end{tabular}
\caption{\small{Average precision, recall and f-measure for different datasets in a 2-way classification task. 2-tailed test. Results are averaged over the five personality traits.}\label{parx1}}
\end{table}
 Since each personality trait is bipolar, we decided to run the tests, considering as true positives the correct predictions for both poles, as false positives the wrong predictions and as false negatives the missing predictions. The majority baseline is the mean of the predictions using all ``y'' labels and all ``n'' labels. Results, reported in table \ref{parx1}, show that similar performances can be obtained on very different data types. %To the best of our knowledge, it is the first system for personality recognition from text that has been tested on Italian. %For the experiments presented in the next section, we used feature score normalization and variable hypothesis average.


\section{Conclusions and Future Work}\label{end}
We presented PR2: a NLP tool, available online, for language-independent unsupervised personality recogniton from unlabeled text. Unlike supervised systems, that require large labeled datasets for training, this tool requires just a small labeled dataset for the validation of the annotation.\\
\indent In the future, we would like to improve the precision of the system and to provide a scorer for validation. %\cite{witten.frank05}.

%\begin{displaymath}
%P=\frac{\textit{tp}}{\textit{tp}+\textit{fp}} \> \> \>
%R=\frac{\textit{tp}}{\textit{tp}+\textit{fn}} \> \> \>
%F=\frac{2 \cdot (P \cdot R)}{(P + R)}
%\end{displaymath}

%\begin{itemize}
%\item item1
%\item item2
%...
%\end{itemize}

%\enumsentence{sentence \label{}}
%\ref{}
%\footnote[number]{text}
%\cite{}
%\paragraph{}
%\subsection{}
%\url{}

%\begin{table}[ht]
%\centering
%\begin{tabular}{|l|l|}
%\hline
%Stitle col1 & title col2\\
%\hline
%item col1 & item col2\\
%\hline
%\end{tabular}
%\caption{text here \label{}}
%\end{table}

%\begin{figure}[ht]
%\begin{center}
%\includegraphics[width=4cm]{name.png}
%\caption{Figure Title.} \label{fig:1}
%\end{center}
%\end{figure}

%\``
%\''

\begin{thebibliography}{}
\bibitem{adelstein.al11}Adelstein J.S., Shehzad Z., Mennes M., DeYoung C.G., Zuo X-N., Kelly C., Margulies D.S., Bloomfield A., Gray J.R., Castellanos X.F. and Milham M.P. Personality Is Reflected in the Brain's Intrinsic Functional Architecture. In \textit{PLoS ONE} 6:(11), 1--12. 2011.
%\bibitem{argamon.al05}Argamon, S., Dhawle S., Koppel, M., Pennebaker J. W. . Lexical Predictors of Personality Type. In \textit{Proceedings of Joint Annual Meeting of the Interface and the Classification Society of North America}. . 2005. %pp. 1--16
\bibitem{bai.al12}Bai, S., Zhu, T., Cheng, L.  Big-Five Personality Prediction Based on User Behaviors at Social Network Sites. In \textit{eprint arXiv:1204.4809}. 2012.
%\bibitem{arnstein.poesio08}Artstein, R. and Poesio, M. Inter-coder agreement for computational linguistics. \textit{Comput. Linguist.}, 34(4):555�-596. 2008.
%\bibitem{baayen.r}Baayen, R.H. Analyzing linguistic data: a practical introduction to statistics using R. \textit{Cambridge University Press}. 2008.
%\bibitem{bachenko.al08}Bachenko, J., Fitzpatrick, E., and Schonwetter, M. Verification and implementation of language-based deception indicators in civil and criminal narratives. In \textit{Proceedings of the 22nd International Conference on Computational Linguistics}. 1:41-�48. 2008. %Stroudsburg, PA, USA. Association for Computational Linguistics.
%\bibitem{bond.lee05}Bond, G. and Lee, A. Language of lies in prison: Linguistic classification of prisoners' truthful and deceptive natural language. \textit{Applied Cognitive Psychology} 19:313�-329. 2005.
\bibitem{briggs.myers80}Briggs, I. and Myers, P.B. \textit{Gifts differing: Understanding personality type}. Mountain View, CA: Davies-Black Publishing. 1980.
%\bibitem{carletta96}Carletta, J. Assessing agreement on classification tasks: the kappa statistic. \textit{Comput. Linguist.} 22(2):249�-254. 1996.
\bibitem{celli12}Celli, F. Unsupervised Personality Recognition for Social Network Sites. In \textit{Proc. of ICDS 2012 : The Sixth International Conference on Digital Society}, 59--62. 2012.
%\bibitem{celli.zaga}Celli, F. Zaga, C. Be Conscientious, Express your sentiment! In \textit{Workshop on Emotion and Sentiment in Social and Expressive Media: approaches and perspectives from AI}. 2013.
%\bibitem{celli.al13}Celli F., Pianesi F., Stillwell D., Kosinski M. Workshop on Computational Personality Recognition (Shared Task). In \textit{Proc. of WCPR13, in conjunction with ICWSM-13}. 2013.
\bibitem{celli.polonio13}Celli F., Polonio L. Relationships between Personality and Interactions in Facebook. In: Xin Ming Tu, Ann Marie White and Naiji Lu (Editors): Social Networking: Recent Trends, Emerging Issues and Future Outlook. Nova Science Publishers, Inc. 2013. %pp. 31-43.
%\bibitem{cohen95}Cohen, W. W. Fast effective rule induction. In \textit{Proc. of the 12th International Conference on Machine Learning}. 1995. %Lake Tahoe, CA.
%\bibitem{cortes.svm}Cortes C., Vapnik V. Support-vector networks.\textit{Mach Learn}. 20(3):273--297. 1995.
%\bibitem{costa.maccrae85}Costa P.T., Jr. and MacCrae, R.R. \textit{The NEO Personality Inventory manual}. Psychological Assessment Resources. 1985.
\bibitem{costa.maccrae92}Costa, P. T. and MacCrae, R. R. Normal personality assessment in clinical practice: The neo personality inventory. Psychological assessment, 4(1):5. 1992.
\bibitem{coltheart81}Coltheart, M. The MRC psycholinguistic database. In \textit{Quarterly Journal of Experimental Psychology}. 33(A):497-�505. 1981.
%\bibitem{deyoung10}DeYoung, C.G. Toward a Theory of the Big Five. In \textit{Psychological Inquiry}. 21. 2010. %pp. 26--33.
\bibitem{fornaciari.al13}Fornaciari T., Celli F., Poesio M. The Effect of Personality Type on Deceptive Communication Style. In \textit{Workshop on Forensic Text Analytics, in conjunction with the Intelligence and Security Informatics Conference (EISIC 2013)}. 2013.
\bibitem{golbeck.al11}Golbeck, J. and Robles, C., and Turner, K. Predicting Personality with Social Media. In \textit{Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems}. . 2011.%pp. 253--262
%\bibitem{hancock.al08}Hancock, J. T., Curry, L. E., Goorha, S., and Woodworth, M. On Lying and Being Lied To: A Linguistic Analysis of Deception in Computer-Mediated Communication. \textit{Discourse Processes}. 45(1):1�-23. 2008.
\bibitem{iacobelli.al11}Iacobelli, F., Gill, A.J., Nowson, S. and Oberlander, J. Large scale personality classification of bloggers. In \textit{Lecture Notes in Computer Science (6975)}. 568--577. 2011.%
%\bibitem{john.langley95}John, G. H. and Langley P. Estimating Continuous Distributions in Bayesian Classifiers. In: \textit{Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence}. 1995.
\bibitem{kermanidis12}Kermanidis, K.L. Mining Authors' Personality Traits from Modern Greek Spontaneous Text. In \textit{4th International Workshop on Corpora for Research on Emotion Sentiment \& Social Signals, in conjunction with LREC12}. 2012.
%\bibitem{koppel.al06}Koppel, M., Schler, J., Argamon, S., and Pennebaker, J. Effects of age and gender on blogging. In \textit{AAAI 2006 Spring Symposium on Computational Approaches to Analysing Weblogs}. 1--7. 2006.
%\bibitem{landis.koch77}Landis, R. J. and Koch, G.G. The measurement of observer agreement for categorical data. \textit{Biometrics}. 33(1):159�-174. 1977.
%\bibitem{locke00}Locke, K.D. Circumplex Scales of Interpersonal Values: Reliability, validity, and applicability to interpersonal problems and personality disorders. \textit{Journal of Personality Assessment}, 75: 249�-267. 2000.
\bibitem{luyckx.daelemans08}Luyckx, K. and Daelemans, W. Authorship attribution and verification with many authors and limited data. In \textit{Proc. of the 22nd International Conference on Computational Linguistics}. 1:513�-520, 2008a %Stroudsburg, PA, USA. Association for Computational Linguistics.
%\bibitem{luyckx.daelemans08b}Luyckx, K. and Daelemans, W. Personae: a corpus for author and personality prediction from text. In \textit{Proc. of LREC2008, the Sixth International Language Resources and Evaluation Conference}, 2981--2987. 2008b.
\bibitem{mairesse.al07}Mairesse, F., Walker, M.A., Mehl, M.R., and Moore, R.K. Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text. In \textit{Journal of Artificial intelligence Research}, 30: 457--500. 2007.
%\bibitem{markovikj.al13}Markovikj D, Gievska S., Kosinski M. and Stillwell D. Mining Facebook Data for Predictive Personality Modeling. In \textit{Proc. of WCPR13, in conjunction with ICWSM-13}. 2013.
%\bibitem{mischel.al07}Mischel, W., Shoda, Y., and Ayduk, O. \textit{Introduction to Personality: Toward an Integrative Science of the Person.} John Wiley \& Sons. Hoboken, NJ. 2007.
%\bibitem{mosteller.wallace64}Mosteller, F. and Wallace, D. \textit{Inference and Disputed Authorship: The Federalist}. Addison-Wesley, Boston, MA. 1964. 
%\bibitem{newman.al03}Newman, M. L., Pennebaker, J. W., Berry, D. S., and Richards, J. M. Lying Words: Predicting Deception From Linguistic Styles. \textit{Personality and Social Psychology Bulletin}. 29(5):665�-675. 2003.
%\bibitem{norman63}Norman, W., T. Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality rating. In \textit{Journal of Abnormal and Social Psychology}, 66. 574--583. 1963.%pp. 
%\bibitem{nguyen:twitter}Nguyen, D. and Gravel, R. and Trieschnigg, D. and Meder, T. How Old Do You Think I Am?"; A Study of Language and Age in Twitter. In \textit{Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media}, 2013 
\bibitem{oberlander.nowson06}Oberlander, J., and Nowson, S. Whose thumb is it anyway? classifying author personality from weblog text. In \textit{Proc. of the 44th Annual Meeting of the Association for Computational Linguistics ACL}. 627--634. 2006.
%\bibitem{peersman.al11}Peersman, C., Daelemans, W., and Van Vaerenbergh, L. Age and gender prediction on netlog data. \textit{Presented at the 21st Meeting of Computational Linguistics in the Netherlands (CLIN21)}. 2011. %Ghent, Belgium.
%\bibitem{pennebaker.al01}Pennebaker, J. W., Francis, M. E., Booth, R. J.  \textit{Inquiry and Word Count}. Lawrence Erlbaum, Mahwah, NJ. 2001.
%\bibitem{platt98}Platt, J. Machines using Sequential Minimal Optimization. In Schoelkopf, B., Burges, C., Smola, A. (ed), \textit{Advances in Kernel Methods, Support Vector Learning}. 1998.
%\bibitem{quinlan93}Quinlan, R.  \textit{C4.5: Programs for Machine Learning}. Morgan Kaufmann Publishers. 1993.
\bibitem{quercia.al11}Quercia, D. and Kosinski, M. and Stillwell, D., and Crowcroft, J. Our Twitter Profiles, Our Selves: Predicting Personality with Twitter. In \textit{Proceedings of SocialCom2011}.  2011. pp. 180--185.
%\bibitem{rammstedt.john07}Rammstedt, B., and John, O. P. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. \textit{Journal of Research in Personality}, 41(1), 203-212. (2007).
%\bibitem{riggio.friedman83}Riggio, R.E. and Friedman, H.S. Individual differences and Cues to Deception. \textit{Journal of Personality and Social Psychology}, 45(4). 899-915. 1983.
%\bibitem{solan.tiersma04}Solan, L. M. and Tiersma, P. M. Author identification in american courts. \textit{Applied Linguistics}. 25(4):448�-465. 2004.
%\bibitem{stein.al07}Stein, B., Koppel, M., and Stamatatos, E. Plagiarism analysis, authorship identification, and near-duplicate detection pan07. \textit{SIGIR Forum}. 41:68-�71. 2007.
%\bibitem{strapparava.mihalcea09}Strapparava, C. and Mihalcea, R. The Lie Detector: Explorations in the Automatic Recognition of Deceptive Language. In \textit{Proc. ACLShort09 - Proc. of the ACL-IJCNLP 2009 Conference.} 1--4. 2009. 
\bibitem{tausczik.pennebaker10}Tausczik, Y. R., and Pennebaker, J. W. . The psychological meaning of words: LIWC and computerized text analysis methods. \textit{Journal of Language and Social Psychology}, 29(1), 24-54. (2010).
%\bibitem{vaassen.daelemans11}Vaassen, F. and Daelemans, W. Automatic emotion classification for interpersonal communication. In \textit{2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis}. 1--7. 2011.
%\bibitem{vrij08}Vrij, A. Detecting Lies and Deceit: Pitfalls and Opportunities. In \textit{Wiley Series in Psychology of Crime, Policing and Law}. John Wiley \& Sons, 2nd edition. 2008.
%\bibitem{vrij05}Vrji, A. Criteria-based content analysis - A Qualitative Review of the First 37 Studies. \textit{Psychology, Public Policy, and Law}. 11(1):3�-41. 2005.
%\bibitem{verhoeven.al13}Verhoeven B., Daelemans W. and De Smedt T. Ensemble Methods for Personality Recognition. In \textit{Proc. of WCPR13, in conjunction with ICWSM-13}. 2013.
%\bibitem{witten.frank05}Witten, I.H., and Frank, E. \textit{Data Mining. Practical Machine Learning Tools and Techniques with Java implementations}. Morgan and Kaufman. 2005.
%\bibitem{zhou.al04}Zhou, L., Burgoon, J. K., Twitchell, D., Qin, T., and Nunamaker, J. F. A comparison of classification models for predicting deception in computer-mediated communication. \textit{Journal of Management Information Systems}. 20:139�-169. 2004.


\end{thebibliography}

%\bibliographystyle{alpha}
%\bibliography{bib}


%\appendix
%\chapter{Appendix}
%\section{Material and Key Answers}


\end{document}
1402.2796.extracted_bib
{adelstein.al11Adelstein J.S., Shehzad Z., Mennes M., DeYoung C.G., Zuo X-N., Kelly C., Margulies D.S., Bloomfield A., Gray J.R., Castellanos X.F. and Milham M.P. Personality Is Reflected in the Brain's Intrinsic Functional Architecture. In PLoS ONE 6:(11), 1--12. 2011. %
1402.2796.extracted_bib
{argamon.al05Argamon, S., Dhawle S., Koppel, M., Pennebaker J. W. . Lexical Predictors of Personality Type. In Proceedings of Joint Annual Meeting of the Interface and the Classification Society of North America. . 2005. %pp. 1--16
1402.2796.extracted_bib
{bai.al12Bai, S., Zhu, T., Cheng, L. Big-Five Personality Prediction Based on User Behaviors at Social Network Sites. In eprint arXiv:1204.4809. 2012. %
1402.2796.extracted_bib
{arnstein.poesio08Artstein, R. and Poesio, M. Inter-coder agreement for computational linguistics. Comput. Linguist., 34(4):555�-596. 2008. %
1402.2796.extracted_bib
{baayen.rBaayen, R.H. Analyzing linguistic data: a practical introduction to statistics using R. Cambridge University Press. 2008. %
1402.2796.extracted_bib
{bachenko.al08Bachenko, J., Fitzpatrick, E., and Schonwetter, M. Verification and implementation of language-based deception indicators in civil and criminal narratives. In Proceedings of the 22nd International Conference on Computational Linguistics. 1:41-�48. 2008. %Stroudsburg, PA, USA. Association for Computational Linguistics. %
1402.2796.extracted_bib
{bond.lee05Bond, G. and Lee, A. Language of lies in prison: Linguistic classification of prisoners' truthful and deceptive natural language. Applied Cognitive Psychology 19:313�-329. 2005.
1402.2796.extracted_bib
{briggs.myers80Briggs, I. and Myers, P.B. Gifts differing: Understanding personality type. Mountain View, CA: Davies-Black Publishing. 1980. %
1402.2796.extracted_bib
{carletta96Carletta, J. Assessing agreement on classification tasks: the kappa statistic. Comput. Linguist. 22(2):249�-254. 1996.
1402.2796.extracted_bib
{celli12Celli, F. Unsupervised Personality Recognition for Social Network Sites. In Proc. of ICDS 2012 : The Sixth International Conference on Digital Society, 59--62. 2012. %
1402.2796.extracted_bib
{celli.zagaCelli, F. Zaga, C. Be Conscientious, Express your sentiment! In Workshop on Emotion and Sentiment in Social and Expressive Media: approaches and perspectives from AI. 2013. %
1402.2796.extracted_bib
{celli.al13Celli F., Pianesi F., Stillwell D., Kosinski M. Workshop on Computational Personality Recognition (Shared Task). In Proc. of WCPR13, in conjunction with ICWSM-13. 2013.
1402.2796.extracted_bib
{celli.polonio13Celli F., Polonio L. Relationships between Personality and Interactions in Facebook. In: Xin Ming Tu, Ann Marie White and Naiji Lu (Editors): Social Networking: Recent Trends, Emerging Issues and Future Outlook. Nova Science Publishers, Inc. 2013. %pp. 31-43. %
1402.2796.extracted_bib
{cohen95Cohen, W. W. Fast effective rule induction. In Proc. of the 12th International Conference on Machine Learning. 1995. %Lake Tahoe, CA. %
1402.2796.extracted_bib
{cortes.svmCortes C., Vapnik V. Support-vector networks.Mach Learn. 20(3):273--297. 1995. %
1402.2796.extracted_bib
{costa.maccrae85Costa P.T., Jr. and MacCrae, R.R. The NEO Personality Inventory manual. Psychological Assessment Resources. 1985.
1402.2796.extracted_bib
{costa.maccrae92Costa, P. T. and MacCrae, R. R. Normal personality assessment in clinical practice: The neo personality inventory. Psychological assessment, 4(1):5. 1992.
1402.2796.extracted_bib
{coltheart81Coltheart, M. The MRC psycholinguistic database. In Quarterly Journal of Experimental Psychology. 33(A):497-�505. 1981. %
1402.2796.extracted_bib
{deyoung10DeYoung, C.G. Toward a Theory of the Big Five. In Psychological Inquiry. 21. 2010. %pp. 26--33.
1402.2796.extracted_bib
{fornaciari.al13Fornaciari T., Celli F., Poesio M. The Effect of Personality Type on Deceptive Communication Style. In Workshop on Forensic Text Analytics, in conjunction with the Intelligence and Security Informatics Conference (EISIC 2013). 2013.
1402.2796.extracted_bib
{golbeck.al11Golbeck, J. and Robles, C., and Turner, K. Predicting Personality with Social Media. In Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems. . 2011.%pp. 253--262 %
1402.2796.extracted_bib
{hancock.al08Hancock, J. T., Curry, L. E., Goorha, S., and Woodworth, M. On Lying and Being Lied To: A Linguistic Analysis of Deception in Computer-Mediated Communication. Discourse Processes. 45(1):1�-23. 2008.
1402.2796.extracted_bib
{iacobelli.al11Iacobelli, F., Gill, A.J., Nowson, S. and Oberlander, J. Large scale personality classification of bloggers. In Lecture Notes in Computer Science (6975). 568--577. 2011.% %
1402.2796.extracted_bib
{john.langley95John, G. H. and Langley P. Estimating Continuous Distributions in Bayesian Classifiers. In: Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence. 1995.
1402.2796.extracted_bib
{kermanidis12Kermanidis, K.L. Mining Authors' Personality Traits from Modern Greek Spontaneous Text. In 4th International Workshop on Corpora for Research on Emotion Sentiment \& Social Signals, in conjunction with LREC12. 2012. %
1402.2796.extracted_bib
{koppel.al06Koppel, M., Schler, J., Argamon, S., and Pennebaker, J. Effects of age and gender on blogging. In AAAI 2006 Spring Symposium on Computational Approaches to Analysing Weblogs. 1--7. 2006. %
1402.2796.extracted_bib
{landis.koch77Landis, R. J. and Koch, G.G. The measurement of observer agreement for categorical data. Biometrics. 33(1):159�-174. 1977. %
1402.2796.extracted_bib
{locke00Locke, K.D. Circumplex Scales of Interpersonal Values: Reliability, validity, and applicability to interpersonal problems and personality disorders. Journal of Personality Assessment, 75: 249�-267. 2000.
1402.2796.extracted_bib
{luyckx.daelemans08Luyckx, K. and Daelemans, W. Authorship attribution and verification with many authors and limited data. In Proc. of the 22nd International Conference on Computational Linguistics. 1:513�-520, 2008a %Stroudsburg, PA, USA. Association for Computational Linguistics. %
1402.2796.extracted_bib
{luyckx.daelemans08bLuyckx, K. and Daelemans, W. Personae: a corpus for author and personality prediction from text. In Proc. of LREC2008, the Sixth International Language Resources and Evaluation Conference, 2981--2987. 2008b.
1402.2796.extracted_bib
{mairesse.al07Mairesse, F., Walker, M.A., Mehl, M.R., and Moore, R.K. Using Linguistic Cues for the Automatic Recognition of Personality in Conversation and Text. In Journal of Artificial intelligence Research, 30: 457--500. 2007. %
1402.2796.extracted_bib
{markovikj.al13Markovikj D, Gievska S., Kosinski M. and Stillwell D. Mining Facebook Data for Predictive Personality Modeling. In Proc. of WCPR13, in conjunction with ICWSM-13. 2013. %
1402.2796.extracted_bib
{mischel.al07Mischel, W., Shoda, Y., and Ayduk, O. Introduction to Personality: Toward an Integrative Science of the Person. John Wiley \& Sons. Hoboken, NJ. 2007. %
1402.2796.extracted_bib
{mosteller.wallace64Mosteller, F. and Wallace, D. Inference and Disputed Authorship: The Federalist. Addison-Wesley, Boston, MA. 1964. %
1402.2796.extracted_bib
{newman.al03Newman, M. L., Pennebaker, J. W., Berry, D. S., and Richards, J. M. Lying Words: Predicting Deception From Linguistic Styles. Personality and Social Psychology Bulletin. 29(5):665�-675. 2003. %
1402.2796.extracted_bib
{norman63Norman, W., T. Toward an adequate taxonomy of personality attributes: Replicated factor structure in peer nomination personality rating. In Journal of Abnormal and Social Psychology, 66. 574--583. 1963.%pp. %
1402.2796.extracted_bib
{nguyen:twitterNguyen, D. and Gravel, R. and Trieschnigg, D. and Meder, T. How Old Do You Think I Am?"; A Study of Language and Age in Twitter. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media, 2013
1402.2796.extracted_bib
{oberlander.nowson06Oberlander, J., and Nowson, S. Whose thumb is it anyway? classifying author personality from weblog text. In Proc. of the 44th Annual Meeting of the Association for Computational Linguistics ACL. 627--634. 2006. %
1402.2796.extracted_bib
{peersman.al11Peersman, C., Daelemans, W., and Van Vaerenbergh, L. Age and gender prediction on netlog data. Presented at the 21st Meeting of Computational Linguistics in the Netherlands (CLIN21). 2011. %Ghent, Belgium. %
1402.2796.extracted_bib
{pennebaker.al01Pennebaker, J. W., Francis, M. E., Booth, R. J. Inquiry and Word Count. Lawrence Erlbaum, Mahwah, NJ. 2001. %
1402.2796.extracted_bib
{platt98Platt, J. Machines using Sequential Minimal Optimization. In Schoelkopf, B., Burges, C., Smola, A. (ed), Advances in Kernel Methods, Support Vector Learning. 1998. %
1402.2796.extracted_bib
{quinlan93Quinlan, R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers. 1993.
1402.2796.extracted_bib
{quercia.al11Quercia, D. and Kosinski, M. and Stillwell, D., and Crowcroft, J. Our Twitter Profiles, Our Selves: Predicting Personality with Twitter. In Proceedings of SocialCom2011. 2011. pp. 180--185. %
1402.2796.extracted_bib
{rammstedt.john07Rammstedt, B., and John, O. P. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41(1), 203-212. (2007). %
1402.2796.extracted_bib
{riggio.friedman83Riggio, R.E. and Friedman, H.S. Individual differences and Cues to Deception. Journal of Personality and Social Psychology, 45(4). 899-915. 1983. %
1402.2796.extracted_bib
{solan.tiersma04Solan, L. M. and Tiersma, P. M. Author identification in american courts. Applied Linguistics. 25(4):448�-465. 2004. %
1402.2796.extracted_bib
{stein.al07Stein, B., Koppel, M., and Stamatatos, E. Plagiarism analysis, authorship identification, and near-duplicate detection pan07. SIGIR Forum. 41:68-�71. 2007. %
1402.2796.extracted_bib
{strapparava.mihalcea09Strapparava, C. and Mihalcea, R. The Lie Detector: Explorations in the Automatic Recognition of Deceptive Language. In Proc. ACLShort09 - Proc. of the ACL-IJCNLP 2009 Conference. 1--4. 2009.
1402.2796.extracted_bib
{tausczik.pennebaker10Tausczik, Y. R., and Pennebaker, J. W. . The psychological meaning of words: LIWC and computerized text analysis methods. Journal of Language and Social Psychology, 29(1), 24-54. (2010). %
1402.2796.extracted_bib
{vaassen.daelemans11Vaassen, F. and Daelemans, W. Automatic emotion classification for interpersonal communication. In 2nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis. 1--7. 2011. %
1402.2796.extracted_bib
{vrij08Vrij, A. Detecting Lies and Deceit: Pitfalls and Opportunities. In Wiley Series in Psychology of Crime, Policing and Law. John Wiley \& Sons, 2nd edition. 2008. %
1402.2796.extracted_bib
{vrij05Vrji, A. Criteria-based content analysis - A Qualitative Review of the First 37 Studies. Psychology, Public Policy, and Law. 11(1):3�-41. 2005. %
1402.2796.extracted_bib
{verhoeven.al13Verhoeven B., Daelemans W. and De Smedt T. Ensemble Methods for Personality Recognition. In Proc. of WCPR13, in conjunction with ICWSM-13. 2013. %
1402.2796.extracted_bib
{witten.frank05Witten, I.H., and Frank, E. Data Mining. Practical Machine Learning Tools and Techniques with Java implementations. Morgan and Kaufman. 2005. %
1402.2796.extracted_bib
{zhou.al04Zhou, L., Burgoon, J. K., Twitchell, D., Qin, T., and Nunamaker, J. F. A comparison of classification models for predicting deception in computer-mediated communication. Journal of Management Information Systems. 20:139�-169. 2004.