Why We Read Wikipedia
\alignauthor Philipp Singer\textsuperscript{\textasteriskcentered 1
% \alignauthor Philipp Singer\thanks{Both authors contributed equally to this work.
Philipp Singer
Florian Lemmerich\footnotemark[1]
Robert West
Leila Zia
Ellery Wulczyn
Leila Zia
Ellery Wulczyn
Markus Strohmaier
Jure Leskovec
% \alignauthor % Philipp Singer*
% \affaddr{GESIS
Wikipedia is one of the most popular sites on the Web, with millions of users relying on it to satisfy a broad range of information needs every day. Although it is crucial to understand what exactly these needs are in order to be able to meet them, little is currently known about why users visit Wikipedia. The goal of this paper is to fill this gap by combining a survey of Wikipedia readers with a log-based analysis of user activity. Based on an initial series of user surveys, we build a taxonomy of Wikipedia use cases along several dimensions, capturing users' motivations to visit Wikipedia, the depth of knowledge they are seeking, and their knowledge of the topic of interest prior to visiting Wikipedia. Then, we quantify the prevalence of these use cases via a large-scale user survey conducted on live Wikipedia with almost 30,000 responses. Our analyses highlight the variety of factors driving users to Wikipedia, such as current events, media coverage of a topic, personal curiosity, work or school assignments, or boredom. Finally, we match survey responses to the respondents' digital traces in Wikipedia's server logs, enabling the discovery of behavioral patterns associated with specific use cases. For instance, we observe long and fast-paced page sequences across topics for users who are bored or exploring randomly, whereas those using Wikipedia for work or school spend more time on individual articles focused on topics such as science. Our findings advance our understanding of reader motivations and behavior on Wikipedia and can have implications for developers aiming to improve Wikipedia's user experience, editors striving to cater to their readers' needs, third-party services (such as search engines) providing access to Wikipedia content, and researchers aiming to build tools such as recommendation engines. %For this purpose, the present paper combines a survey of Wikipedia readers with a log-based analysis of user activity to understand why users visit Wikipedia. % PHILIPP: % Daily, millions of users visit and rely on Wikipedia making it one of the most popular and important websites on the Web. % For optimally meeting user needs, it is important to understand why and how they visit Wikipedia and read articles. Yet, surprisingly little is known about reader characteristics, motivations, and behavior. % The main purpose of this paper is to bridge this gap by a multi-faceted research study. We build a taxonomy of Wikipedia use cases along several dimensions capturing users' motivation to visit Wikipedia, their depth of knowledge they are seeking, and their knowledge of the topic at interest prior to visiting Wikipedia. % We quantify the prevalence of these use cases representative of Wikipedia's reader population by a large-scale and robust user survey conducted on live Wikipedia. % Results highlight the importance of a variety of use cases including, e.g., the motivation to read an article after hearing about the topic in media. % Additionally, we match and correlate survey responses with readers' digital traces extracted from Wikipedia's server logs enriched with underlying background information about consumed content, thus, we discover behavioral patterns associated with specific use cases. % This work advances our understanding about reader behavior on Wikipedia. Our findings have implications for developers of Wikipedia's user experience, editors striving to optimally cater to their readers' needs, third-party services (such as search engines) providing access to Wikipedia content, and researchers aiming to build tools such as article recommendation engines. % % ORIGINAL: % Wikipedia, one of the most visited websites on the Web, attracts millions of users every day. Yet, relatively little is known about reader characteristics, motivations, and behavior. To that end, we ran a robust survey on Wikipedia asking readers about their prior knowledge, their intent of information depth, and their main motivation to read the article. For gaining representative insights, we control for bias by utilizing propensity score weight adjustment compared to a random user population sample. We find certain motivational aspects to be popular, such as the motivation to read an article after hearing about it in media, as well as certain interesting response correlation such as that readers that are bored have a higher probability to gaining just an overview about a topic compared to in-depth knowledge. Additionally, we couple survey responses with underlying large-scale log data from Wikipedia, e.g., visited article's topological position, or a reader's session behavior. Here, we apply subgroup discovery aiming at detecting interesting behavioral rules, finding, e.g., that readers motivated by school or work have higher shares of visiting central articles, articles about mathematics or technology, and accessing Wikipedia by desktop compared to the overall reader population. % Overall, this work advances our understanding about the larges user group of Wikipedia---readers. Furthermore, generated insights might have implications on the development of Wikipedia, e.g., by editor recommendations.
000abstract.tex
\vspace{-7mm}
\begin{abstract}
Wikipedia is one of the most popular sites on the Web, with millions of users relying on it to satisfy a broad range of information needs every day.
Although it is crucial to understand what exactly these needs are in order to be able to meet them, little is currently known about why users visit Wikipedia.
The goal of this paper is to fill this gap by combining a survey of Wikipedia readers with a log-based analysis of user activity.
Based on an initial series of user surveys, we build a taxonomy of Wikipedia use cases along several dimensions, capturing users' motivations to visit Wikipedia, the depth of knowledge they are seeking, and their knowledge of the topic of interest prior to visiting Wikipedia.
Then, we quantify the prevalence of these use cases via a large-scale user survey conducted on live Wikipedia with almost 30,000 responses. Our analyses highlight the variety of factors driving users to Wikipedia, such as current events, media coverage of a topic, personal curiosity, work or school assignments, or boredom.
Finally, we match survey responses to the respondents' digital traces in Wikipedia's server logs, enabling the discovery of behavioral patterns associated with specific use cases.
For instance, we observe long and fast-paced page sequences across topics for users who are bored or exploring randomly, whereas those using Wikipedia for work or school spend more time on individual articles focused on topics such as science.
Our findings advance our understanding of reader motivations and behavior on Wikipedia and can have implications for developers aiming to improve Wikipedia's user experience, editors striving to cater to their readers' needs, third-party services (such as search engines) providing access to Wikipedia content, and researchers aiming to build tools such as recommendation engines.

%For this purpose, the present paper combines a survey of Wikipedia readers with a log-based analysis of user activity to understand why users visit Wikipedia. 

% PHILIPP:
% Daily, millions of users visit and rely on Wikipedia making it one of the most popular and important websites on the Web. 
% For optimally meeting user needs, it is important to understand why and how they visit Wikipedia and read articles. Yet, surprisingly little is known about reader characteristics, motivations, and behavior.
% The main purpose of this paper is to bridge this gap by a multi-faceted research study. We build a taxonomy of Wikipedia use cases along several dimensions capturing users' motivation to visit Wikipedia, their depth of knowledge they are seeking, and their knowledge of the topic at interest prior to visiting Wikipedia.
% We quantify the prevalence of these use cases representative of Wikipedia's reader population by a large-scale and robust user survey conducted on live Wikipedia.
% Results highlight the importance of a variety of use cases including, e.g., the motivation to read an article after hearing about the topic in media. 
% Additionally, we match and correlate survey responses with readers' digital traces extracted from Wikipedia's server logs enriched with underlying background information about consumed content, thus, we discover behavioral patterns associated with specific use cases. 
% This work advances our understanding about reader behavior on Wikipedia. Our findings have implications for developers of Wikipedia's user experience, editors striving to optimally cater to their readers' needs, third-party services (such as search engines) providing access to Wikipedia content, and researchers aiming to build tools such as article recommendation engines.
%
% ORIGINAL:
% Wikipedia, one of the most visited websites on the Web, attracts millions of users every day. Yet, relatively little is known about reader characteristics, motivations, and behavior. To that end, we ran a robust survey on Wikipedia asking readers about their prior knowledge, their intent of information depth, and their main motivation to read the article. For gaining representative insights, we control for bias by utilizing propensity score weight adjustment compared to a random user population sample. We find certain motivational aspects to be popular, such as the motivation to read an article after hearing about it in media, as well as certain interesting response correlation such as that readers that are bored have a higher probability to gaining just an overview about a topic compared to in-depth knowledge. Additionally, we couple survey responses with underlying large-scale log data from Wikipedia, e.g., visited article's topological position, or a reader's session behavior. Here, we apply subgroup discovery aiming at detecting interesting behavioral rules, finding, e.g., that readers motivated by school or work have higher shares of visiting central articles, articles about mathematics or technology, and accessing Wikipedia by desktop compared to the overall reader population.
%  Overall, this work advances our understanding about the larges user group of Wikipedia---readers. Furthermore, generated insights might have implications on the development of Wikipedia, e.g., by editor recommendations. 
\end{abstract}
010introduction.tex
%\newpage
\section{Introduction}
\label{sec:intro}

Wikipedia is the world's largest encyclopedia and one of the most popular websites, with more than 500 million pageviews per day. It attracts millions of readers from across the globe and serves a broad range of their daily information needs.
Despite this, very little is known about the motivations and needs of this diverse user group: why they come to Wikipedia, how they consume the content in the encyclopedia, and how they learn. Without this knowledge, creating more content, products, and services that ensure high levels of user experience remains an open challenge \cite{basu2003context,white2009characterizing,feild2010predicting,krug2014make}. 


\para{Background and objectives.}
%\mst{I would suggest moving the two paragraphs below to related work}
A rich body of work has investigated motivations and behavior patterns of users on the Web \cite{goel2012does,kumar2010characterization}. Specific attention has been cast on a few major sites including search engines \cite{broder2002taxonomy,rose2004understanding,weber2011uses}, and social networking sites such as Twitter \cite{java2007we,kwak2010twitter} and Facebook \cite{ryan2011uses}. Yet, surprisingly little is known about the motivations, needs, and behaviors of \WP readers, possibly keeping \WP from reaching its full potential. 
%\mst{can we state why it is important to know about WP motivations, rather than say that we know little right now?}

The vast literature on user behavior in \WP (\cf\ Okoli \etal\ \cite{okoli2012people} for an overview) has focused on content production. It mainly investigates editors' motivations \cite{arazy,nov2007motivates}, patterns of editing behavior \cite{jurgens2012temporal}, and the quality of content \cite{kittur2008harnessing,stvilia2008information}. Much less is known about content consumption, even though readers make up the majority of Wikipedia users. The limited work on readers has focused on topics such as content preferences \cite{spoerri2007popular,ratkiewicz2010characterizing,lehmann2014reader}, search queries leading to \WP \cite{waller2011search}, or navigation patterns \cite{paranjape2016improving,lamprecht2016evaluating,west2012human,singer2014detecting}. In contrast, the present work aims at understanding \emph{why we read Wikipedia}.


\para{Materials and methods.}
We present a robust taxonomy of use cases for reading Wikipedia, constructed through a series of surveys based on techniques from grounded theory \cite{strauss1998basics}. Initially, we administered a survey to elicit free text responses to the question, \emph{Why are you reading this article today?} Based on the responses, we designed a taxonomy covering three major dimensions that can be used to characterize the observed use cases.
After validating the robustness of our taxonomy, we study the prevalence of use cases as measured by a large-scale multiple\hyp choice survey on English Wikipedia. To correct for various forms of representation bias in our pool of respondents, we use inverse propensity score weighting adjustment. We then enrich the survey data by linking each survey response to the respondent's behavior traces mined from \WP's webrequest logs. An illustration of how both survey and log data are collected can be found in \Figref{fig:fig1}. Finally, we use the joined survey and log data to identify characteristic behavior patterns for reader groups with specific intentions via subgroup discovery \cite{kloessgen1996,herrera2010}.

\begin{figure}[t!]
	\centering
	\includegraphics[width=0.3\textwidth]{fig1_3_cropped.pdf}
	\caption{
	%\emph{Illustration.} Figure shows
	\emph{Example Wikipedia reading session.}
	The user arrives from a search engine and visits the article about \textit{Oceania;} she then navigates to \textit{Australia,} where she responds to our survey. Afterwards, the reader goes back to the search engine and finally visits the article about \textit{Nauru}. This paper studies survey responses as well as webrequest logs.}
	\label{fig:fig1}
	%\vspace{-1em}
\end{figure}


\para{Contributions and findings.}
The following are our three main contributions:
(i)~We present a robust taxonomy for character\-izing use cases for reading Wikipedia (\Secref{sec:taxonomy}), which captures users' motivations to visit Wikipedia, the depth of information they are seeking, and their familiarity with the topic of interest prior to visiting Wikipedia.
(ii)~We quantify the prevalence and interactions between users' motivations, information needs, and prior familiarity via a large-scale survey yielding almost 30,000 responses (\Secref{sec:survey_results}). 
(iii)~We enhance our understanding of the behavioral patterns associated with different use cases by combining survey responses with digital traces recorded in Wikipedia's server logs  (\Secref{sec:webrequest_results}). 

Our analysis lets us conclude that there is a variety of motivations bringing readers to Wikipedia, which can be characterized by distinct behavioral patterns. For example, users visiting \WP out of boredom view multiple, topically different articles in quick succession.
While no motivation clearly dominates, it is generally the case that Wikipedia is used for shallow information needs (fact look-up and overview) more often than for in-depth information needs. 
Also, prior to reading an 
article, readers are familiar with its topic about as often as not.





%\para{}
%\indent
%In terms of implications, 
The outcomes of this research can help Wikipedia's editor and developer communities, as well as the Wikimedia Foundation, to make more informed decisions about how to create and serve encyclopedic content in the future.


% \para{Structure of the paper. }
% The rest of this paper is organized as follows:
% We describe how we develop a taxonomy of Wikipedia readers in Section 2.
% Next, Section 3 presents our datasets and pre-processing.
% Afterwards, Section 4 features our main results on why user read Wikipedia.
% We reflect on our contribution, its implications and limitations in Section 5 before we conclude in Section 6.
% \note[Leila]{Add the structure of the paper.}
% \note[Florian]{I think we do not need this necessarily, added it anyway}
% This paper, is to the best of our knowledge, the first to capture, quantify, and study reader motivation and information needs on Wikipedia

020taxonomy.tex
% \begin{table}\label{tb:surveysForTaxonomy}
% \caption{xxx}
% \center
%   \begin{tabular}{l | l | l | p{1cm} | p{1cm} | c }
%       & \textbf{Project} & \textbf{Platform} & \textbf{Sampling Rate} & \textbf{Number of responses} & \textbf{Period} \\ 
%       \hline
%       1 & English & Desktop and Mobile & 5:100K & 5000 & 4 days\\
%       \hline
%       2 & English & Mobile & 5:10K & 1655 & 3 days\\
%       \hline
%       3 & English & Desktop and Mobile & 5:100K & 10,503 & 6 days\\
%       \hline
%       4 & Spanish & Desktop and Mobile & & & 2 days\\
%       \hline
%       5 & Persian & Desktop and Mobile & 75:100 & 3615 & 2 days\\
%       \hline
%   \end{tabular}
% \end{table}

% \begin{table}[b!]
% \center
%  \caption{xxx}
%   \begin{tabular}{l | p{4cm} }
%   \textbf{Trend} & \textbf{Categories}\\
%   \hline
%   Information depth & quick look-up, overview, in-depth\\
%   \hline
%   Familiarity & familiar, unfamiliar\\
%   \hline
%   Motivator & bored/random, conversation, current event, intrinsic learning, media, personal decision, work/school, other\\
%   \hline
%   \end{tabular}
%   \label{tb:taxonomy}
% \end{table}
%\vspace{-1mm}
\section{\hspace{-0.3mm}Taxonomy of Wikipedia readers}

\label{sec:taxonomy}
Our research relies on a taxonomy of Wikipedia readers, something that was previously absent from the literature. We designed and analyzed a series of surveys based on techniques from \emph{grounded theory} \cite{strauss1998basics} to build a robust categorization scheme for Wikipedia readers' motivations and needs.
In this section, we explain the individual steps taken and the resulting taxonomy.

% Any research that aims to understand Wikipedia readers' motivations relies on a taxonomy of Wikipedia readers but such a taxonomy was absent from the literature. Therefore, we used a series of surveys in three Wikipedia languages and the techniques in Grounded Theory \cite{strauss1998basics} to build a robust taxonomy for Wikipedia readers' motivations. In this section, we explain the steps and the resulting taxonomy.

\para{Building the initial taxonomy.}
We started with an initial questionnaire \emph{(Survey 1)}, where a randomly selected 
subgroup of English Wikipedia readers (sampling rate 1:200, desktop and mobile, 4 days, about 5,000 responses) saw a survey widget while browsing Wikipedia articles. If the reader chose to participate, she was taken to an external site (Google Forms) and asked to answer the question \textit{``Why are you reading this article today?"}\ in free-form text (100\hyp character limit). 

To arrive at categories for describing use cases of Wikipedia reading, five researchers performed three rounds of hand\hyp coding on a subset of the 5,000 responses, without discussing any expectations or definitions ahead of time.
In the first stage, all researchers worked together on 20 entries to build a common understanding of the types of response.
In the second stage, based on the discussions of the first stage, tags were generously assigned by each researcher individually to 100 randomly selected responses, for a total of 500 responses tagged. All 500 tagged responses were reviewed, and four main trends (motivation, information need, context, and source) were identified, alongside tags associated with each response.
In the third stage, each researcher was randomly assigned another 100 responses and assessed if they contained information about the four main trends identified in the previous stage and if the trends and tags should be reconsidered. 
The outcome of these stages revealed the following three broad ways in which users interpreted the question; we use them as orthogonal dimensions to shape our taxonomy: 
% \begin{itemize}[leftmargin=*, topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex,leftmargin=0ex,itemindent=\parindent]

\begin{itemize}[leftmargin=*, topsep=1pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]
\item \emph{Motivation}: work/school project, personal decision, current event, media, conversation, bored/random, intrinsic learning.
\item \emph{Information need}: quick fact look-up, overview, in-depth.
\item \emph{Prior knowledge}: familiar, unfamiliar.
\end{itemize}


%We use these as orthogonal dimensions to shape our taxonomy.

%\vspace{1pt}
%Similar smaller-scale surveys conducted on the Spanish and Persian Wikipedias led to similar observations.

% We started by survey 1 in Table \ref{tb:surveysForTaxonomy}. A randomly selected subgroup of English Wikipedia readers saw a widget while browsing Wikipedia. If the reader would accept to participate, she would be taken to an external site (Google Forms) where she was asked to answer one question in free-form text (with 100 character limit): \textit{Why are you reading this article today?}. A three stage hand-coding was done over the survey responses by five researchers. The goal of the iterative hand-coding was to arrive at categories that we could  use to describe Wikipedia reader subgroups. In the first stage, all researchers worked together on 20 entries to build a common understanding of the type of responses. In the second stage, tags were generously assigned by each researcher to 100 randomly selected responses, for a total of 500 responses tagged. All 500 tagged responses were reviewed and four main trends (motivator, information need, context, and source) along with tags associated with each responses were identified. In the third stage, each researcher randomly chose 100 new responses and assessed whether the responses contained information about each of the four main trends identified in the previous stage and whether the trends and tags should be reconsidered. The outcome of this stage was the following trends and tags: Motivator (work/school project, personal decision, current event, media, conversation, bored/random exploration, intrinsic learning), Information Depth (quick look-up, overview, in-depth), and Familiarity (familiar, unfamiliar).


%\note[Philipp]{I would argue to keep a short notice on other languages, but we would need exact numbers for sampling rate etc.}
%\vspace{-1.7em}
\para{Assessing the robustness of the taxonomy.}
We conducted two surveys similar to \emph{Survey 1} on the Spanish and Persian Wikipedias which resulted in similar observations and dimensions as above.
Additionally, we assessed the robustness of the above taxonomy in two follow-up surveys.
First, we ran a survey identical to Survey~1 to validate our categories on unseen data
(\emph{Survey 2;} sampling rate 1:2000, mobile, 3 days, 1,650 responses).
No new categories were revealed through hand\hyp coding. 


% The first question to address was whether the trends identified in survey 1 hold true through further surveys in English Wikipedia and in other Wikipedia languages. To this end, we repeated ran survey 2, 4 and 5 which were repeats of survey 1 English, Spanish, and Persian Wikipedia. Through sampling and hand-coding of the results in these three surveys we learned that no new trends or categories were communicated by survey participants.

Second, we crafted a multiple\hyp choice version of the free\hyp form surveys (\emph{Survey 3;} sampling rate 1:200, desktop and mobile, 6 days, about 10,500 responses).
It comprised three questions with the following answer options in random order (the first two questions also offered ``other'' as an answer, with the option to enter free\hyp form text):

%questioned whether identified categories of our taxonomy through Survey 1 were capturing what the survey participants had to communicate as responses to the following three questions:
\begin{itemize}[leftmargin=*, topsep=1pt,after=\vspace{1pt},itemsep=-1ex,partopsep=1ex,parsep=1ex]
\item \emph{I am reading this article because\dots}: I have a work or school-related assignment; I need to make a personal decision based on this topic (\eg, buy a  book, choose a travel destination); I want to know more about a current event (\eg, a soccer game, a recent earthquake, somebody's death); the topic was referenced in a piece of media (\eg, TV, radio, article, film, book); the topic came up in a conversation; I am bored or randomly exploring Wikipedia for fun; this topic is important to me and I want to learn more about it (\eg, to learn about a culture). 
Users could select multiple answers for this question.
\item \emph{I am reading this article to\dots}: look up a specific fact or to get a quick answer; get an overview of the topic; get an in-depth understanding of the topic. 
%Single choice responses allowed.
\item \emph{Prior to visiting this article\dots}: I was already familiar with the topic; I was not familiar with the topic, and I am learning about it for the first time. 
%Single choice responses allowed.
\end{itemize}


%To that end, we ran \emph{Survey 3} with the above three multiple choice questions (randomized order) and an ``other'' option (along with option to enter free form text) for the first and second questions (5:1000 sampling rate, desktop and mobile, 6 days, \mytilde10,500 responses). 
Only 2.3\% of respondents used the ``other'' option, and hand-coding of the corresponding free\hyp form responses did not result in new categories.
We thus conclude that our categories are robust and use the resulting classification as our \emph{taxonomy of Wikipedia readers} in the rest of this paper.  


%based on which we ran an additional large-scale survey (see Section~\ref{Survey}) and in the remainder of this paper. 

030data+methodology.tex

\input{feature_table.tex}



%\vspace{2mm}
\newpage
\section{Datasets and Preprocessing}
%\note[Florian]{Section title up for discussion. Before it was only "datasets", but this does not well capture the bias correction. Also, feature extraction from the data sources could be considered a pre-prop step.}
\label{sec:data+methodology}

% In this section, we describe the datasets we leverage, as well as how we have preprocessed them.
Here, we describe utilized datasets and preprocessing.

%In this section, we describe all data utilized throughout this work including necessary preprocessing.

\input{031survey}




\input{032webrequests}

\begin{figure*}[t!]
	\centering
	\subfloat{\includegraphics[width=0.99\textwidth]{survey_response.pdf}\label{<figure1>}}
		\vspace{-1em}
	\caption{\emph{Survey responses.} This figure visualizes the share of answers for the three parts of the user survey: motivation, information need, and prior knowledge. The blue bars (left) reflect the raw unweighted responses, and the green bars (right) depict the bias-corrected weighted responses
	(propensity score weight adjustment).
% 	as determined by propensity score weight adjustment against a population sample. 
	Error bars visualize the 95\% confidence intervals using effective sample size for the weighted responses. In general, results suggest popularity of both extrinsic and intrinsic motivation, as well as high, but balanced,  relevance for certain information needs and prior knowledge.
% 	the results suggest that readers often visit Wikipedia due to being motivated by media, intrinsic knowledge, conversational aspects, or also boredom or work/school  aspects. 
% 	In terms of information depth, both the aspect of gaining an overview or checking a fact are the most popular answers followed by the wish to gain in-depth knowledge. Readers appear to be familiar and unfamiliar with the topic read on Wikipedia at similar rates. 
	The results based on bias-correction weighting reflect minor changes in responses without drastically changing the general direction. }
	\label{fig:survey_response}
	\vspace{-1em}
\end{figure*}

\input{033backgroundData}

\input{034features}

\input{035bias}

%\todo[Bob]{The feature section needs to be integrated into the other sections}



031survey.tex
%\vspace{-0.3cm}
\subsection{Survey}
\label{sec:survey_description}
To quantify the prevalence of the driving factors specified by our taxonomy, we ran an additional \emph{large-scale survey} on English Wikipedia consisting of the same three questions on motivation, depth of information need, and prior knowledge as \emph{Survey 3} (\Secref{sec:taxonomy}).
%to gather reader data corresponding to each of the trends and categories identified in Section \ref{sec:taxonomy}, \ie, motivation, depth of information need, and prior knowledge. 
The survey was run at a sampling rate of 1:50
from 2016-03-01 to 2016-03-08 on all requests to English Wikipedia's mobile and desktop sites.
It was not shown on non\hyp article pages (discussion pages, search pages, \etc), on the main page of Wikipedia, and to browsers with \emph{Do not Track} enabled. Potential survey participants were identified by assigning a token to their browsers and eventually showing a widget with an invitation to participate in the survey. 
%At each page request by an eligible token, the reader would have a chance to see the survey (multiple surveys may run on Wikipedia at the same time and when that is the case, users are randomly assigned to specific surveys at the time of each request). 
Once shown, the reader could ignore it, dismiss it, or opt in to participate which would take the reader to an external site (Google Forms), where she would see the three questions described in \Secref{sec:taxonomy}. A unique, anonymous ID was passed to Google Forms for each user, which would later be used to link the survey responses to users' webrequest logs (\Secref{sec:webrequest_logs}).
A privacy and consent statement\footnote{\url{https://wikimediafoundation.org/wiki/Survey_Privacy_Statement_for_Schema_Revision_15266417}} providing details about the collection, sharing, and usage of the survey data was shown to all users prior to submitting their responses.
% All users were shown, and consented to, a privacy statement providing details about the collection, sharing, and usage of the survey data prior to submitting their responses.
Overall, our dataset consists of survey answers from 29,372 participants after basic data cleaning such as removing duplicate answers from the same users.
Whenever we write ``survey'' throughout the rest of this paper, we refer to the survey described here.

032webrequests.tex
%\newpage
\subsection{Webrequest logs}
\label{sec:webrequest_logs}

Ultimately, we aim to to understand how users' motivation, desired depth of knowledge, and prior knowledge (\ie, their answers to our survey) manifest themselves in their reading behavior.
The data collected through the survey alone, however, does not provide any information on the respondent's behavior beyond the single pageview upon which the survey was presented.

In order to be able to analyze respondents' reading behavior in context, we connect survey responses to the webrequest logs maintained by \WP's web servers, where every access to any \WP page is stored as a record that contains, among others, the requested URL, referer URL, timestamp, client IP address, browser version, and city\hyp level geolocation inferred from the IP address.
Since the logs do not contain unique user IDs, we construct approximate user IDs by concatenating the client IP address and browser version; \cf\ discussion in \Secref{sec:Reflections on methodology}.

As the information needs and reading behavior of the same user may change over time, we operate at an intermediate temporal granularity by decomposing a user's entire browsing history into \emph{sessions,} where we define a session as a contiguous sequence of pageviews with no break longer than one hour \cite{halfaker2015user}.
To reconstruct the session in which a user took our survey (\cf\ \Figref{fig:fig1}), we retrieved from the webrequest logs all records with the user's (approximate) ID, ordered them by time, chunked them into sessions according to the aforementioned one-hour rule, and returned the session that contains the record with the specific URL and timestamp of the survey response.

033backgroundData.tex
%\newpage
\subsection{Wikipedia article data}
\label{sec:wikipedia_background_data}

%Here, we describe utilized Wikipedia background data such as how we determined the underlying topological link network.

Different articles are consumed in different ways.
Hence, the properties of articles viewed by survey respondents play an important role in our analysis.
To extract these properties, we utilized the public dump of English \WP released on 2016-03-05,%
\footnoteurl{https://archive.org/details/enwiki-20160305}
such that article revisions closely match those seen by survey participants.

The dump contains wiki markup, whereas browsers receive HTML code generated from this markup. Since the markup may contain templates that are expanded only upon conversion to HTML, some page content is not immediately available in markup form.
In order to obtain a more complete representation, we retrieved the full HTML of the article contents using Wikimedia's public API. %
%\footnoteurl{https://www.mediawiki.org/wiki/API:Main_page}
 In addition to the textual article content, we extracted the network of articles (5.1M) connected by hyperlinks (370M).



% Throughout this work, we couple the survey and respective requests with features extracted from Wikipedia articles. Specifically, we are interested in obtaining article features from the underlying topological link network and from static text content. We focus on all articles contained in the main namespace of the English Wikipedia---matching the survey requests---extracted from the public XML dump from March, 2016.

% In this work, we focus on all articles contained in the main namespace of the English Wikipedia as extracted from the public XML dump from March, 2016.\footnoteurl{https://archive.org/details/enwiki-20160305}.
% To retrieve a more complete representation of the Wikipedia link network (e.g., the official Wikipedia dump does not contain links present in certain templates such as infoboxes), we additionally crawled all static HTML pages using Wikimedia's public API\footnoteurl{https://www.mediawiki.org/wiki/API:Main_page} specifically matching respective revision ID of the article in the dump. With this data, we then created the topological link network where articles are nodes and directed edges refer to unique links between two articles. We additionally rendered the articles to check if links are visible to the user.
% Overall, the link network contains \mytilde$5.1$ million articles connected by \mytilde$370$  million distinct links. We also extracted the plain text for all articles at hand from the HTML.

034features.tex


\subsection{Features}
\label{sec:features}

Throughout this work, we study features extracted 
directly from the survey responses (\Secref{sec:survey_description}), from underlying webrequest logs of article requests and extracted sessions (\Secref{sec:webrequest_logs}), and background Wikipedia article data associated with requested articles (\Secref{sec:wikipedia_background_data}). We list and describe all features utilized in this work in \Tabref{tab:features}.

For topic detection, we fit a Latent Dirichlet Allocation (LDA) \cite{blei2003latent} model on bag-of-words vectors  representing articles' textual content (with stopwords removed) using online variational Bayes.
% ; we removed standard English stopwords, all words occurring only once in an article, and all words occurring in more than 90\% of all articles. 
To find a balance between complexity and interpretability, we decided to work with 20 topics. We assigned labels to topics by manually inspecting the topics' word distributions and their top Wikipedia articles.

% from both the survey responses, as well as from respondents' mapped digital traces on Wikipedia---\ie visited articles and session information---supplemented with visited articles' content and topological features. We list all features in \Tabref{tab:features} and provide detailed description. 
% %The remainder of this section shortly summarizes the distinct feature categories and adds additional detail not covered in the table.
% (i) \emph{Response features} list answers for our three survey questions about a reader's motivation to visit Wikipedia, expected information need, and knowledge prior to reading the article. (ii) \emph{Survey features} cover all information directly extracted from survey responses including the page request associated with the survey---i.e., the request where the survey was presented to the user. (iii) \emph{Article features} capture information about content and link-network topology of articles requested by readers---i.e., those articles associated with digital traces of survey respondents. 
% For topic detection, we fit a Latent Dirichlet Allocation on bag-of-words vectors  of textual content (with removed stopwords) using an online variational Bayes algorithm.
% % ; we removed standard English stopwords, all words occurring only once in an article, and all words occurring in more than 90\% of all articles. 
% To find balance between complexity and interpretability, we decided to work with 20 distinct topics. For assigning labels to the topics, we manually studied the topics' word distributions and their top Wikipedia articles.
% Finally, for our \emph{session features}, we followed the approach 
% examined by Halfaker et al. \cite{halfaker2015user} and break sessions if there is a break longer than 60 minutes. Subsequently, we detect the single session associated with the request of the survey, for which most session features are calculated.

% For detecting topics,
% we decided to fit a Latent Dirichlet Allocation. To that end, we first built bag-of-word representations of the textual content; we removed standard English stopwords, all words occurring only once in an article, and all words occurring in more than 90\% of all articles. Based on these bow representations, we inferred topics with LDA using an online variational Bayes algorithm. To find balance between complexity and interpretability, we decided to work with 20 distinct topics. For assigning labels to the topics, we manually studied the topics' word distributions and their top Wikipedia articles.
% Finally, for our \emph{session features}, we followed the approach 
% examined by Halfaker et al. \cite{halfaker2015user} and break sessions if there is a break longer than 60 minutes. Subsequently, we detect the single session associated with the request of the survey, for which most session features are calculated.


% Throughout this work, we focus on the following features and feature groups.

% \para{Survey features.}
% This set of features covers all information utilized that directly stems from survey respondents.
% Here, we cover all features extracted from the survey. We start with the basic responses including the \emph{information depth}, \emph{prior knowledge} and \emph{motivation}. For motivation, multiple responses are possible. Additionally, we extracted the \emph{country} and \emph{continent} of the respondent, as well as the \emph{local time hour} capturing the time of the day of the response. By matching respective request on Wikipedia where the survey was conducted, we can also study the \emph{host} and \emph{referer class}. \phsi{layout options}

%\begin{figure*}
%	\centering
%	\subfloat{\includegraphics[width=0.95\textwidth]{figures/topics.pdf}\label{<figure1>}}
%	\caption{\textbf{Topics.}}
%	\label{fig:topics}
%\end{figure*}

%\begin{table*}
%\center
%  \begin{tabular}{ | l | l | l | }
%    \hline
%Topic 0: & air station class line railway aircraft ship service airport navy  & transportation, modern military\\ \hline
%Topic 1: & species family 10 doi genus pmid scientific classification order related  & biology, chemistry \\ \hline
%Topic 2: & ye al district olya sofla india ali deh rural sar  & south asia, middle east  \\ \hline
%Topic 3: & link page article articles change point disambiguation led internal directly  & mathematics, disambiguation\\ \hline
%Topic 4: & retrieved 2013 2011 2014 2015 2012 2010 2009 com january  & 21st century \\ \hline
%Topic 5: & film series time best man episode movie new story season  & tv, television, novels\\ \hline
%Topic 6: & john st london england south australian new australia british william  & common wealth\\ \hline
%Topic 7: & japanese japan chinese china hong kong na korean jazz kim  & east asia\\ \hline
%Topic 8: & saint la article sur le help wikipedia en stub expanding  & spanish (stubs)\\ \hline
%Topic 9: & war army ii battle military general king died pp royal  & war, history\\ \hline
%Topic 10: & south republic russian united africa russia states international islands france  & geography (unions, trade)\\ \hline
%Topic 11: & church isbn art university new history press book museum work  & literature, art\\ \hline
%Topic 12: & school university party government national education college state law president  & education, government, law \\ \hline
%Topic 13: & born 1992 1990 1994 1980 1999 1998 1986 1976 1996  & 20th century \\ \hline
%Topic 14: & team league football season cup club player 10 games championship  & sports \\ \hline
%Topic 15: & new state county states national york united american school city  & United states\\ \hline
%Topic 16: & 10 11 12 15 14 13 16 18 17 20  & numbers \\ \hline
%Topic 17: & used company power design use energy space data technology based  & technology, energy, power\\ \hline
%Topic 18: & music album song band released live love albums single chart  & music \\ \hline
%Topic 19: & river area road park population north south city lake lès  & geographical entities\\ \hline
%
%
%    \hline
%  \end{tabular}
%\end{table*}

% \para{Article features.}
% For each request matching the survey response, we can study feature associated with respective Wikipedia article. 
% %To that end, we derived article features based on Wikipedia's topological link network and the content of an article.
% %This covers all features associated with the article visited when conducting the survey, e.g., the article's pagerank.
% First, we derived features based on the underlying topological link network. 
% The \emph{in-degree} captures an article's number of incoming links from other articles, the \emph{out-degree} refers to the number of links to other articles, and the \emph{pagerank} of an article captures its topological position in the network. For calculating the pagerank, we used a damping factor of $0.85$ and an iterative computation with a tolerance of $0.01$; the pagerank is not normalized to sum to 1 for all articles. For obtaining a proxy of page popularity, we utilize the \emph{number of pageviews} and article accumulated over the span of our survey---cf. also \cite{lehmann2014reader} for a discussion on reader preferences with respect to pageview statistics.

% Second, we focused on article features obtained from the textual content of Wikipedia articles as extracted from our crawled HTML files. The \emph{text length} simply refers to the number of characters in an article. 
% Prominently, we also study topics associated with the textual content of Wikipedia articles throughout this paper.
% We decided to fit a Latent Dirichlet Allocation to the textual data. To that end, we first built bag-of-word representations of the textual content; we remove standard English stopwords, only considered words occurring at least twice in an article, and removed all words occurring in more than 90\% of all articles. Based on these bow representations, we inferred topics with LDA using an online variational Bayes algorithm. To find balance between complexity and interpretability, we decided to work with 20 distinct topics. 

% We manually studied the topics' word distribution and their top Wikipedia articles to come up with topic labels as follows: \phsi{additional material}
% (1) transportation, modern military, (2) biology, chemistry, (3) south asia, middle east, (4) mathematics, (5) 21st century, (6) tv, movies, novels, (7) Britain \& common wealth (8) east asia, (9) spanish (stubs), (10) war, history, (11) geography (unions, trade), (12) literature, art, (13) education, government, law, (14) 20th century, (15) sports, (16) United States, (17) numbers, (18) technology, energy, power, (19) music, and (20) geographical entities. We use the probabilistic \emph{topic distribution} as 20 individual features for each article. Additionally, we use an individual feature for capturing the \emph{top topic} of an article as well as the \emph{topic entropy} to detect the topic specificity of an article.


% \phsi{details about nr. of topics and maxit}. For simplicity, we then decided to assign a single topic to an article based on the highest probability. We also manually labeled topics \phsi{label details}

%Finally, to capture the popularity of articles, we captured the \emph{pageviews} of an article in respective month. \phsi{this is missing, details}

%\begin{enumerate}
%    \item article length
%    \item in-degree
%    \item out-degree
%    \item pagerank
%    \item TODO: pageviews
%    \item topic (from LDA)
%\end{enumerate}

% \para{Session behavior features.}
% This covers all features associated with the session conducted during the survey going beyond the single request we covered before. To that end, we followed the approach examined by Halfaker et al. \cite{halfaker2015user} and break sessions if there is a break longer than 60 minutes. Subsequently, we detect the single session associated with the request of the survey; we only consider requests that are pageviews. Base on this single survey session, we then derived a set of additional features.

% The \emph{session length} reflects the total number of requests made in respective session; similarly the \emph{session duration} captures the total time spent in the session in minutes. With the \emph{avg. time between requests} feature we then study the average time difference between requests in a session. Also, with the \emph{number of survey article} we reflect how frequently the article associated with the survey response occurs in the session. These features have been also discussed in \cite{lehmann2014reader} finding certain reading sub-patterns (cf. Section~\ref{sec:relwork}).
% %Additionally, requests in sessions are of different nature. For example, some are direct pageviews, while others are requests stemming from e.g., loading and looking at images. With the \emph{avg. num. of pageviews} we can capture this artefact. 
% Additionally, requests in sessions are of different nature; e.g., they have different referers. 
% Thus, we derive the \emph{number of external search referers}, \emph{number of external non-search referers}, \emph{number of internal refers} and \emph{number of no referers} within a session; for each of these features we also derived the ratio compared to the overall session length. 

% Additionally, we can map our individual article features to all requests within a session and derive additional features such as the \emph{average pagerank, outdegree, indegree and textlength of articles in a session}. This is compliment by an \emph{average pagerank difference} feature that captures whether users move to the periphery or core of Wikipedia within a session. We also derived topical features in the form of the \emph{average topic entropy} of articles in the session as well as \emph{the average topic distance} which is the average Manhattan distance between  topic distributions of subsequent articles in a session---this should reflect whether users frequently change the topic of interest within a session.

%For example, requests have different referers; some requests are based on internal navigation, others come from external resources such as search engines. Thus, we can derive a series of features from this, e.g., the \emph{avg. num. of external searchers} tells us how many requests within a session are based on the user's search behavior on sites like Google. \phsi{more}

%\begin{enumerate}
%    \item session length (in requests)
%    \item session duration
%    \item avg time between requests
%    \item avg\_survey\_session\_num\_pageviews (analogously, any avg. network measure)
%    \item number/ratio external searches
%\end{enumerate}

%\para{User features.}
%This covers all features associated with general information about the user participating in the survey.
%
%The \emph{country} and \emph{continent} features refer to the country and continent the user comes from. Additionally, we can capture device information from the user such as used \emph{browser}, \emph{operating system}, and \emph{device}.
%
%\begin{enumerate}
%    \item country
%    \item continent
%    \item TODO: Browser
%    \item TODO: OS
%    \item TODO: maybe device
%\end{enumerate}

%\para{Context features.}
%
%So far we have:
%\begin{enumerate}
%    \item time of survey(can be decomposed into "time\_of\_the\_day", "weekday")
%    \item survey\_time\_to\_click
%    
%\para{Feature ideas.}

%\begin{itemize}
%	\item Number of times the visited article occurs in the session; i.e., how many times does the reader return to the article \phsi{this is difficult though as I am not sure whether the motivation is directly associated with the article at hand} \phsi{this might be a general problem and should be discussed}
%\end{itemize}    
%
%\end{enumerate}

035bias.tex

\subsection{\hspace{-1mm}Correcting survey bias via webrequest logs}
\label{sec:correction}

The goal of this work is to study the motivations and behaviors representative of Wikipedia's entire reader population.
However, deducing properties of a general population from surveying a limited subpopulation is subject to different kinds of biases and confounders, including
\emph{coverage bias} (inability to reach certain subpopulations), \emph{sampling bias} (distortions due to sampling procedure),
and \emph{non\hyp response bias} (diverse likelihood of survey participation after being sampled as a participant).
%
%For example, we observe that more active users (\eg, more requests, longer sessions) are overrepresented among survey participants when the data is analyzed on a user session level (see~\Secref{sec:discussion} for a discussion of the unit of analysis).
%In addition, our survey is potentially subject to unit \emph{non-response bias}: some user groups are more likely to participate in the survey than others and is a significant issue for the interpretation of answers in our survey.
%, particularly by \emph{unit non-response}.

Consequently, an important step in our analysis is to account for potential biases in survey responses.
Finding suitable adjustments has been a decade\hyp long research effort in the survey methodology community \cite{brick2013unit}. Since strata methods such as poststratification~\cite{gelman2000poststratification} are less well suited to control for a large number of features, we opt for \emph{inverse propensity score weighting} \cite{austin2011introduction} as an alternative.
This technique assigns control weights to each survey response, thus correcting bias with respect to a control group (Wikipedia population). The rationale behind this procedure is that answers of users less likely to participate in the survey should receive higher weights, as they represent a larger part of the overall population with similar features. For determining participation probabilities \emph{(propensity scores),} we use gradient\hyp boosted regression trees on individual samples to predict if they belong to the survey \vs\ the control group, using all features of \Secref{sec:features}.
(We provide additional methodological details in the appendix.)
By using background features (\eg, country, time) plus digital traces (\eg, sessions),
and by building a representative control group,
we have an advantage over traditional survey design, which is often limited to few response features such as gender and age, as well as to small control groups.

When discussing results in the next section, we shall see (\Figref{fig:survey_response}) that our weight adjustment changes the relative shares of survey responses only slightly, with general trends staying intact.
Hence, we shall utilize only weighted survey responses for inference on statistical properties from this point on. Additionally, we use the so-called \emph{effective sample size} (\cf\ appendix) when calculating standard errors, confidence intervals, and statistical tests, in order to account for differing standard errors of weighted estimators.


% methods based on \emph{propensity scores} have been developed, as they are used in this paper for applying weights to individual survey responses to correct for bias influence factors \cite{austin2011introduction}. 
%In that directions, simple approaches such as poststratification methods have been proposed

% In detail, we employ so-called \emph{inverse propensity score weighting} assigning control weights to survey samples. 
% The propensity score of a single instance reflects the probability that an instance with these control features (cf. \Secref{sec:features}) participated in the survey and is not part of a control group which is a random sample of Wikipedia users from the log-data. We calculate the propensity score with gradient boosting and determine weights as the inverse of these probabilities. The rationale behind this procedure is that answers of users that are overall less likely to participate in the survey receive higher weights since they represent a larger part of the overall population with similar features. We evaluate the weights with respect to outliers and their ability to reduce the error when conclusions about feature distribution in the random sample are drawn from the distribution in the survey data. 

% In Figure~\ref{fig:survey_response} we show how the determined weights change the relative shares of survey responses. While we can see slight adjustments, the general direction is kept intact. From this point on, we always utilize weighted survey responses for inference on statistical properties. Additionally, we use the so-called \emph{effective sample size} when calculating standard errors, confidence intervals, and statistical tests for accounting for too small standard errors of weighted estimators.


% In that directions, simple approaches such as poststratification or cell adjustment cells divide the entire population into several strata (subgroups) using additionally available auxiliary variables with a known population distribution.
% To estimate mean or total values for a survey answer in the population, then the Horvitz-Thompson estimator~\cite{horvitz1952generalization} can be applied, which basically weights each instance in a stratum with the inverse probability of response in that stratum.
% As an alternative, methods based on \emph{propensity scores} have been developed, as they are used in this paper for applying weights to individual survey responses to correct for bias influence factors.


% For traditional survey \emph{coverage bias}, the inability to reach certain subpopulation with the survey, e.g., by landline phone calls, is a central problem. 
% In our scenario, this is not an issue per-se, as the design allows to reach the whole population at interest---i.e., Wikipedia readers.
% However, ideally, the answers from our survey should be representative for the overall Wikipedia viewer base. In our survey, each user request received an equal chance of showing the survey to a user.
% This, however, implies that e.g., users with longer viewing sessions in Wikipedia, i.e., user that view multiple pages, have an overall higher likelihood of being exposed to the survey.
% In addition, our survey is potentially subject to unit \emph{non-response bias}: some user groups are more likely to participate in the survey than others and is a significant issue for the interpretation of answers in our survey, particularly by \emph{unit non-response}.
% Finding suitable adjustments for non-responses is a decade long research  effort in the survey methodology community, see ~\cite{brick2013unit,lewis2012weighting} for recent overviews.
% In that directions, simple approaches such as poststratification or cell adjustment cells divide the entire population into several strata (subgroups) using additionally available auxiliary variables with a known population distribution.
% To estimate mean or total values for a survey answer in the population, then the Horvitz-Thompson estimator~\cite{horvitz1952generalization} can be applied, which basically weights each instance in a stratum with the inverse probability of response in that stratum.
% As an alternative, methods based on \emph{propensity scores} have been developed, as they are used in this paper for applying weights to individual survey responses to correct for bias influence factors.


%In addition, our survey is potentially subject to unit non-response bias: some user groups are more likely to participate in the survey than others and this possibly influences the survey results.
%As suggested by survey methodology, we apply weights to the individual survey answers to correct for such influence factors.
%
%
%
%
%By contrast, \emph{non-response bias} is a significant issue for the interpretation of answers in our survey, particularly by \emph{unit non-response}. 
%That means that participants that have been asked to fill in the survey refuse or neglect to answer any question.
%Finding suitable adjustments for non-responses is a decade long research  effort in the survey methodology community, see ~\cite{brick2013unit,lewis2012weighting} for recent overviews.
%In that directions, simple approaches such as poststratification or cell adjustment cells divide the entire population into several strata (subgroups) using additionally available auxiliary variables with a known population distribution.
%To estimate mean or total values for a survey answer in the population, then the Horvitz-Thompson estimator~\cite{horvitz1952generalization} can be applied, which basically weights each instance in a stratum with the inverse probability of response in that stratum.
%As an alternative, methods based on \emph{propensity scores} have been developed, as they are used in this paper as elaborated next.
%%Apart from the weighting by inverse propensities as applied in this paper, it has also been suggested to use the propensity scores to group instances for cell adjustments, cf.

% \para{Propensity score weighting method.}
% For this purpose, we employ \emph{inverse propensity score weighting}.
% The main idea is to compute for each user that participated in the survey a propensity score, i.e., the (relative) likelihood of survey participation for this user, with the feature set of ~\Cref{sec:features} as control features. \phsi{references in this section missing}

% To implement this in our scenario, we first drew a simple random sample of Wikipedia viewers from the Wikipedia logs of all user requests as a control group. We focused on users with at least one request in the first week of march in 2016. Then, we proceeded to sample one request for each selected user and marked it as an imaginary request reflecting a potential survey response. We only sampled requests that are desktop or mobile web page views on main English Wikipedia namespace articles to match the original survey.
% For each of the sampled request we compute then the features as described in~\Cref{sec:features}.

% Next, we join the datasets for the survey with the random sample. 
% The propensity score of a single instance (request) then reflects the probability that an instance with these control features participated in the survey and is not part of the control group.
% For that, a post-stratification approach is infeasible due to the large number of control features we consider.
% Instead, we model the group membership (survey participant or control group) with a statistical model.
% For that purpose, we consider two types of models: logistic regression and gradient boosting.
% Logistic regression is the established choice in survey methodology, but boosting method have been reported to competitive propensity score estimations \mst{needs rewriting}. 
% Given the features of an instance $x$, both models produce as an outcome a probability $p(x)$ that $x$ belongs to the survey group.
% We then set the weight $w$ for instance to $\frac{1}{p(x)}$.
% The rationale behind this procedure is that answers of users that are overall less likely to participate in the survey receive higher weights since they represent a larger part of the overall population with similar features.

% \para{Evaluating weights.}
% Checking if the applied weighting schemes have the intended correcting effect of making the user survey data more representative for the overall Wikipedia viewership is non-trivial.
% Consequently, we employ multiple evaluation scenarios for checking the weights:

% First, we investigate a typical problem for inverse propensity weighting that occurs especially if the fit of the models employed to compute the propensity score is bad; sometimes few instances receive overproportional high weights that completely dominate the results. 
% However, in our data the weights are relatively evenly distributed. The maximum weights are below $14$ (gradient boosting) or $21$ (logistic regression), respectively. The vast majority of weights (93\% or 99.9\%) are below $5$. \phsi{should we show the dist?}
% In Figure~\ref{fig:survey_response} we also show how the determined weights change the relative shares of survey responses. While we can see slight adjustments, the general direction is kept intact.

% Additionally, we evaluate the weights by their ability to reduce the error when conclusions about feature distribution in the random sample are drawn from the distribution in the survey data.
% For that purpose, we compute for each numeric feature the mean and standard deviation in the random sample as ground truth. 
% Then, we compare this with estimations for that value from the survey dataset using (a) no weights, (b) weights obtained from gradient boosting, and (c) weights from logistic regression.
% For each feature, we measure the bias as the number of standard deviations the estimated mean deviates from the true mean.
% Averaging these errors over all features leads to a mean error of $0.19$ standard deviations for the unweighted case, $0.96$ standard deviations for the gradient boosted weights, and $0.106$ for the logistic regression weights. \phsi{gbc seems wrong}
% That means that the inverse propensity score correction reduces the prediction error by about $50\%$.
% The reduction is strongly significant ($p \leq 0.001$ according to a Wilcoxon signed rank test).
% Results also imply that for most of the features, the bias in the survey data is very small, i.e., less than $0.1$ standard deviations for article features, but is significantly higher for session behavior features such as session length, or the session time. \phsi{should we somehow show that this error reduction holds for all features?} \phsi{mb also use the other evaluation technique with weights:see next comment by flo} \fle{Maybe add reduced accuracy of the weighted survey/sample classifiers?}


% The results presented in the remainder of the paper focus on the weights obtained from gradient boosting due to their slightly lower error in recovering feature means as well as their higher flexibility in model specification.
% However, the respective results for the logistic regression weights deviate only little from that, also because there is a strong correlation ($\rho = 0.71$) between the two weighting vectors.

%\note[Philipp]{Maybe we can remove the whole concept of design effect and only talk about effective sample size}

% \para{Effective sample size.}
% Throughout this work, we employ a variety of statistical techniques on the survey data. However, the introduction of sample weights for correcting bias in survey responses leads to violations of the IID assumptions \cite{mukhopadhyay2016complex}.  
% Consequently, standard errors of estimators are estimated as too small, which in turn leads to confidence intervals being too narrow and statistical tests asserting significance too often if standard procedures are applied. 
% %The effect that our survey design and correction has on the variance is known as \emph{design effect} (deff), first introduced by Kish \cite{kish1965survey}, capturing the extent to which the sampling error in the survey for some parameter $\theta$ deviates from the expected error from an IID sample. 
% %T \emph{design effect} (deff), first introduced by Kish , 
% The extent to which the sampling error in the survey for some parameter $\theta$ deviates from the expected error from an IID sample due to survey design and correction, is known as the \emph{design effect} \cite{kish1965survey}. 
% If the design effect deviates from $1$---as it is the case in our survey---then our understanding of sample size for calculating standard errors becomes incorrect. 
% %The propensity score weighting distorts our previous survey sample size which has an impact on statistical inference and tests that mostly assume complete random sampling; specifically, it affects the standard error of estimates which varies with sample size. 
% To that end, we consider the \emph{effective sample size} estimating the required sample size of a random sampling survey for achieving the same error as the weighted sample---it is defined as $n_{\text{eff}}=n/\text{deff}$. As we cannot directly calculate $\text{deff}$ without knowing the expected sampling error, we use Kish's approximation formula with weights $w_i$ \cite{kish1965survey}:

% \begin{equation}
%   n_{\text{eff}} = \frac{(\sum_{i=1}^n w_i)^2}{\sum_{i=1}^n w_i^2} 
% \end{equation}

% For our complete survey data, the effective sample size is $n_{\text{eff}}=8601$.
% We use this effective sample size throughout this article for calculating standard errors, confidence intervals, and statistical tests. Note that this makes reported confidence interval and statistical hypothesis tests overly careful. For further details, please refer to \cite{mukhopadhyay2016complex}.

%Our weighting results in an overall effective sample size of $n_{\text{eff}}=$ \phsi{fill in}. We utilize it for statistical measures and tests where mentioned.



%Here, we describe how we check and account for bias in the survey. Specifically, we are interested in first determining whether our data of users who answered the survey is different from a random population sample. Then, we aim at correcting for that.
%
%\subsection{Sample}
%To compare our survey to the general population, we drew random request samples in two different ways: user and request based.
%
%\para{User-based sample.}
%
%Here, 
%
%\para{Request-based sample.}
%THIS GOES TO THE DISCUSSION
%First, we randomly labeled requests from the first week of march in 2016 as requests where an imaginary person would have answered the survey. To match the original survey, we only considered requests that are desktop or mobile web page views on main English Wikipedia namespace articles. Of all requests, we chose a sampling rate of $1:10,000$ leading to $166,432$ requests. We then proceeded as for our original survey data and merged the sampled requests with complete request data for respective user.


%\subsection{Correcting for bias}
%Deducing properties of a general population from surveying a limited subpopulation is subject to different kinds of biases if not accounted for correctly. 
%For traditional survey \emph{coverage bias}, the inability to reach certain subpopulation with the survey, e.g., by landline phone calls, is a central problem. 
%In our scenario, this is not an issue since by design we were able to reach the whole studied population, i.e., the Wikipedia readership.
%By contrast, \emph{non-response bias} is a significant issue for the interpretation of answers in our survey, particularly by \emph{unit non-response}. 
%That means that participants that have been asked to fill in the survey refuse or neglect to answer any question.
%Finding suitable adjustments for non-responses is a decade long research  effort in the survey methodology community, see ~\cite{brick2013unit,lewis2012weighting} for recent overviews.
%In that directions, simple approaches such as poststratification or cell adjustment cells divide the entire population into several strata (subgroups) using additionally available auxiliary variables with a known population distribution.
%To estimate mean or total values for a survey answer in the population, then the Horvitz-Thompson estimator~\cite{horvitz1952generalization} can be applied, which basically weights each instance in a stratum with the inverse probability of response in that stratum.
%As an alternative, methods based on \emph{propensity scores} have been developed





%\section{Predicting reader behavior}
%Here, we maybe can predict reader behavior based on our empirical insights from before.

040results.tex



%\section{Results}
%\label{sec:results}

%This section discusses results on our survey, and webrequest-logs.

\section{Results: Why we read Wikipedia}

%In this section, we use the data described in the last section to study why users read Wikipedia.
This section discusses results on why users read Wikipedia.

%What follows is a study on why we read Wikipedia.

\input{041surveyResults}

\input{042webrequestResults}

\input{043resultsummary}

041surveyResults.tex
\subsection{Survey results}
\label{sec:survey_results}

We start with a discussion of the responses to our survey. 
%From this point on, we only focus on the weighted responses as elaborated in Section~\ref{sec:correction}.

\para{Survey responses.}
First, we examine the percentages of survey respondents with specific motivations, information needs, and prior knowledge.
We visualize the results in \Figref{fig:survey_response}, focusing on the green (right) bars representing weighted survey responses (sorted by popularity).



With respect to \emph{motivation,} we find that Wikipedia is consulted in a large spectrum of use cases and that no clearly dominant motivation can be identified. Prominently, extrinsic situations trigger readers to visit Wikipedia to look up a topic that was referenced in the media (30\%), came up in a conversation (22\%), is work or school\hyp related (16\%), or corresponds to a current event (13\%). At the same time, readers have intrinsic motivations, such as wanting to learn something (25\%), being bored (20\%), or facing a personal decision (10\%).
We also find that the ``other'' option was only rarely selected, further confirming the robustness of the taxonomy of readers introduced in \Secref{sec:taxonomy}. 


% all motivations have been selected at least in $10\%$ while the option of a motivation not covered by our survey only got selected seldom. This further confirms the usefulness of our developed taxonomy of reader motivations as elaborated in \Secref{sec:taxonomy}. 
% In detail, we find both extrinsic and intrinsic motivations to be highly popular. Prominently, extrinsic situations motivate readers to visit Wikipedia to look up a topic that was referenced in media (around 30\%), because the topic came up in conversations (around 22\%), it is work/school related (around 16\%), or a current event (around 13\%). At the same time, readers have intrinsic motivations, such as that they want to learn something (25\%), they are bored (20\%), or based on personal decisions (10\%).

% The most popular response choice is the reason to visit Wikipedia to look up a topic that was referenced in media. 

% For motivation---note that multiple answers were allowed for this question---results reveal that most respondents (around 30\%) visit Wikipedia to look up a topic that was referenced in media. At the same time, readers have intrinsic learning motivations, i.e., they find the topic important and want to learn more about it. A different motivational category build the group of users who visit because they are motivated by conversation.
% Additionally, a group of readers appear to be bored and randomly explore Wikipedia for fun; i.e., do not appear to have a clear externally driven motivation for visiting Wikipedia. This list if followed by work/school, current event, and personal decision motivations.

%\subpara{Information need.} 
The results also show that Wikipedia is visited to satisfy different kinds of \emph{information needs}. 
Interestingly, shallow information needs (overview [39\%] and quick fact-checking [38\%]) appear to be more common than deep information needs (21\%).
As for \emph{prior knowledge,} we observe nearly identical shares of readers being familiar (50\%) \vs\ unfamiliar (47\%) with the topic of interest.

% On one hand, readers visit just to get an overview of an article, but also to fact-check information, and a bit less prominently for gaining in-depth knowledge about a topic.

% \subpara{Prior knowledge.} Interestingly, we can observe near similar shares of users that visit Wikipedia already having prior knowledge about consumed topic (familiar) and those that are unfamiliar.


% With respect to Wikipedia readers' anticipated information depth, we can observe a pretty balanced ratio of users aiming at getting an overview of the topic, opposed to those aiming at quick fact or answer checking. The goal of gaining in-depth insights in a topic have a clear lower share of around 20\%. For prior knowledge, the two possible responses of already being familiar vs. being unfamiliar with the topic prior to visiting Wikipedia show similar response rates.

\begin{table}[b!]
\vspace{-1.5em}
\small
	\centering
	%	\small
	\caption{\emph{Survey response correlations.} 
		%The crosstables show co-occurrence rates between survey responses. 
		Each cell depicts the row-normalized share of responses that have also selected a given column as answer (without ``other'' and non\hyp responses). The bottom rows highlight the overall share of responses for a given column as expectation. Values in brackets reflect the lift ratio of observed \vs\ expected frequency.
		%For example, the fourth row in (a) states that of all respondents selecting bored/random as motivation, around 31\% have also selected fact-checking preferences (below expectation), 23\% in-depth (above expectation), and the much higher share of readers (45\%) just aim at getting an overview of the topic (above expectation). 
		%The arrows in each cell indicate whether respective share is larger (up-arrow) or lower (down-arrow) compared to global expectation (bottom row). 
		The last column indicates significance (***~$<$~0.001, **~$<$~0.01, *~$<$~0.05) for the hypothesis test of independence of observed frequencies (contingency table with row frequencies and complement of all other rows) and expected frequencies (as in the last table row) using a $\chi^2$ test using the effective sample size.}
		%\vspace{-1em}
	\subfloat[Motivation vs. information need]{\input{survey_crosstable_motivation_need.tex}\label{table:cross1}} \\
	\subfloat[Motivation vs. prior knowledge]{\input{survey_crosstable_motivation_prior.tex}\label{table:cross2}} \\
	\subfloat[Prior knowledge vs. 
	information need ]{\input{survey_crosstable_prior_need.tex}\label{table:cross3} }
	\label{table:cross}
	\vspace{-1em}
\end{table}






\para{Survey response correlations.}
Next, in \Tabref{table:cross}, we study whether certain combinations of motivations, information needs, and prior knowledge occur more frequently than expected, quantified by the \emph{lift,} \ie, the ratio between observed and expected frequencies.
%We show certain results in \Tabref{table:cross} and discuss them next.

%\subpara{Motivation vs. information need.} 
%Results in 
\Tabref{table:cross1} suggests that 
different \emph{motivations} are coupled with different \emph{information depths}.
Specifically, in-depth information needs prevail when readers are driven by intrinsic learning (lift $1.62$);
quick fact look-ups are associated more strongly with conversation and work\slash school motivations than one would expect \textit{a priori;}
and gaining an overview of a topic appears to be especially important for readers motivated by media coverage and for the bored.


% For instance, readers wanting to learn about a topic, have much higher needs for gaining in-depth knowledge than what one would expect (ratio of $1.61$) in comparison to e.g., readers initiated by conversation where the frequent aim is to fact-check. The need to gain an overview over a topic is specifically correlated with media motivation (ratio of $1.12$) and readers being bored (ratio of $1.17$).

% \subpara{Motivation vs. prior knowledge.}
% Overall, 

In \Tabref{table:cross2}, we find weaker correlations between \emph{motivations} and levels of \emph{prior knowledge,} apparent from lifts closer to 1 and a lack of significance. However, certain trends still emerge; \eg, when readers research a topic from the media, they are more likely to be unfamiliar with the topic (lift $1.22$). In contrast, readers whose goal is learning are more likely to be familiar with the topic (lift $1.14$).

%\subpara{Prior knowledge vs. information need.} 
As a corollary of the above correlations, we also observe patterns when contrasting \emph{prior knowledge} with \emph{information need} (\Tabref{table:cross3}). We find that familiar readers are more likely to look up quick facts (lift $1.13$) and aim at getting in-depth knowledge about a topic (lift $1.15$) than one would expect. Contrarily, unfamiliar readers are more likely to first aim at getting an overview of the topic (lift $1.22$) instead of directly going into depth (lift $0.87$).

% Next, we study how these different responses correlate with each other. To that end, Table~\ref{table:cross} presents three crosstables: Table~\ref{table:cross1} depicts motivation vs. information need, Table~\ref{table:cross2} motivation vs. prior knowledge, and Table~\ref{table:cross3} depicts prior knowledge vs. information need. Each row-normalized cell shows the relative share of responses; the total share of columns is depicted in the bottom row.
% For each depicted row, we perform a significance test of independence between observed frequencies compiled from the contingency table of the frequencies in respective row and its compliment from all other rows against the expected marginal sum frequencies shown in the last row of the tables. We utilize a classic chi2 contingency table test with effective sample size correction. 
% Note, that we only present and discuss a selection of correlations; also, higher orders (e.g., if someone has bored/random motivation what is the most likely combination of prior knowledge \emph{and} information depth) are not shown.

% Amongst others, results in Table~\ref{table:cross1} highlight that for readers being bored and doing random navigation, there is a higher share of users just looking for an overview compared to those interested in fact-checking and in gaining in-depth knowledge; this is specifically imminent when compared against the overall response rates (bottom row). Similar results can be observed for e.g., readers being motivated by media. Opposite results are highlighted for readers having intrinsic learning motivations where the share of readers being interested in gaining an overview is lower than the overall response rate, while gaining an in-depth knowledge is clearly overrepresented. Table~\ref{table:cross2} correlates motivation with prior knowledge; the general tendency is that we can overall less interesting behavioral outliers compared to the global share---see significance and proportion differences. An interesting artifact is though that readers with intrinsic learning motivation appear to already be familiar with the topic with higher probability; contrary, readers motivated by media are more likely to be unfamiliar and thus, read up on the topic on Wikipedia. Finally, we correlated prior knowledge with information need in Table~\ref{table:cross3}. Here, we find that readers being already familiar, have a higher share to fact-check as compared to expectation. Users being unfamiliar, on the other hand, have a much higher tendency to just aiming for getting an overview of the topic.

\begin{figure}[t!]
	\centering
	\subfloat{\includegraphics[width=0.49\textwidth]{figlegend_motivation_small.pdf}} \\ \vspace{-1.4em}
	\addtocounter{subfigure}{-1}
% 	\subfloat[Weekday]{\includegraphics[width=0.45\textwidth]{figures/motivation_weekday.pdf}\label{fig:weekday}}\hspace{1em}
% 	\subfloat[Daytime]{\raisebox{-0.4em}{\includegraphics[width=0.45\textwidth,height=1.99in]{figures/motivation_hour.pdf}\label{fig:hour}}} \hspace{1em}
\subfloat{\includegraphics[width=0.49\textwidth]{motivation_weekday_hour.pdf}\label{fig:weekday_hour}}
	\vspace{-1em}
	\caption{\emph{Motivation day and time.} This figure visualizes how the relative share of motivation (y-axes) changes over the course of a week and the course of a day (x-axes). 
	%In \Figref{fig:weekday}, the x-axis depicts the individual day of a week, and the y-axis shows the fraction of survey responses selecting a certain motivation on a given day. For instance, on Monday, around 30\% of the respondents selected media as motivation. 
	Error bars depict the 95\% confidence interval with effective sample size. 
	%Moitivations appear to be stable over a week with the exceptions of media having a higher share on weekends, conversation peaking on Friday and Saturday, and work/school declining on the weekend. \Figref{fig:hour} can be read similarly with the distinction that the x-axis depicts grouped hours throughout a day (independent of weekday). Being motivated by media appears to play a higher role throughout the evening and night, while work/school peaks throughout the midday.
	}
	\vspace{-2em}
	\label{fig:time}
\end{figure}

\para{Survey responses over time.}
%As highlighted in \Secref{sec:intro}
Next, we study how the prevalence of motivations, information needs, and prior knowledge changes over time. 
%Primarily, we focus on \emph{motivation} and show trends for weekdays and hours of a day in \Figref{fig:time}.
%For \emph{weekdays}, 
For \emph{motivations,} shown in \Figref{fig:time}, we find relatively stable trends over the course of a week or day.
Three notable exceptions, however, emerge.
First, on weekends (Saturday, Sunday) and at night, there is a higher share of readers who are led to Wikipedia by media coverage; this is potentially due to a higher likelihood of being exposed to media during these time periods.
Similarly, conversations play a more important role on Fridays and Saturdays, possibly since people go out, meet with friends, and are involved in conversations that lead to consulting Wikipedia.
By contrast, reading an article for work or school reasons has a relatively lower share towards the weekend, but peaks at daytime hours, probably because people work and go to school on working days and during daytime hours.

% As shown, readers visit Wikipedia for different reasons, with different needs, and varying prior knowledge. Intuitively, their motivations might also change in various contexts such as the time of the day or location we are in. Consequently, we now study whether the share of readers with certain motivations, needs, and knowledge is higher on specific days of the week and at different times of a day.

% results are shown in \Figref{fig:time}. In \Figref{fig:weekday}, we study how prevalence of motivation changes over the course of a week. While the relative share of most motivations appears to be stable over a week, we can find three interesting sub-artifacts. First, on weekends (Saturday, Sunday), there is a higher share of readers reporting to visit Wikipedia after being motivated by media. Similarly, conversation plays a more important role on Friday and Saturday, maybe evoked by people going out, meeting with friends and being involved in conversations that lead to visiting Wikipedia. Contrary, reading an article for work or school reasons, has a relatively lower share towards the weekend, most likely because people work and go to school throughout working days.

% For \emph{hours}, results suggest that media has higher shares throughout the night, with a clear peak at prime time; this might be reflecting that readers more frequently watch TV at night compared to daytime hours. In contrast, work/school motivation is prominent throughout daytime hours, also being regular school and working hours.


% In \Figref{fig:hour}, we depict a similar visualization, but now split for different times throughout a day (ignoring which day it is). Here, results suggest that media has higher shares throughout the night, with a clear peak at prime time; this might be reflecting that readers have higher probability to watch TV at night compared to daytime hours, and thus, also select this motivation more frequently at these hours. In contrast, work/school motivation is prominent throughout daytime hours, also being regular school and working hours.

% Overall, these analysis show that the relative popularity of motivations is quite stable throughout the day and week with interesting exceptions such as media, conversation, and work/school which me might also expect to play certain roles in the real-life of respondents such as that they are more likely to work throughout daytime and more likely to watch TV at night.

Additionally, results on \emph{information need} show overall quite stable trends over a week and over a day
without clear outliers,
%; no clear outliers can be detected, 
also due to larger confidence intervals (results not visualized). For \emph{prior knowledge,} we identify small upward trends on weekends and evening hours for already being familiar with the topic, compared to being unfamiliar. However, error bars are again too large to justify stronger claims.


%\para{Summary.}
%\begin{table}[h!]
%	\centering
%	\caption{\textbf{Motivation/information depth crosstable.} Amongst others, results highlight that readers are more aiming for gaining an overview about topics when they are bored and do random surfing. Contrary, fact checking is mostly related to conversational, event-specific, or work/school motivations. For intrinsic learning motivations, readers aim for gaining in-depth knowledge.}
%	\input{tables/survey_crosstable_motivation_depth.tex}
%	\label{table:cross2}
%\end{table}
%
%\begin{table}[h!]
%	\centering
%	\caption{\textbf{Motivation/prior knowledge crosstable.} Results e.g., show that intrinsic learning motivations are more frequently coupled with being already familiar with the topic. A contrary example is based on motivation by media where Wikipeda readers appear more frequently to be unfamiliar.}
%	\input{tables/survey_crosstable_motivation_prior.tex}
%	\label{table:cross3}
%\end{table}

042webrequestResults.tex

\input{_sg_results.tex}

\subsection{Webrequest\hyp log results}
\label{sec:webrequest_results}
Our previous results suggest that Wikipedia is visited for a variety of use cases that differ not only in their motivation triggers, but also in the depth of information needs, and readers' prior familiarity with the topic.
%We observed certain correlation of the survey answers with each other.
In this section, we investigate correlations of survey responses with behavioral patterns based on request, article, and session features (\Secref{sec:features}).
%While motivations are mostly stable over time, certain behavioral patterns can be identified, \eg that bored readers are more frequently interested in just gaining an overview compared to readers with intrinsic learning motivations who aim at getting an in-depth understanding of the topic. 
In doing so, we reveal characteristic differences and develop stereotypes for motivational groups.
%These observations suggest that different motivations, information needs, and prior knowledge might also reflect specific behavioral reading patterns on Wikipedia. Consequently, this section aims at untangling potential behavioral sub-patterns by contrasting survey responses with underlying webrequest data as elaborated in \Secref{sec:data+methodology}.

% The survey results suggest a variety and popularity of both intrinsic, as well as extrinsic Wikipedia use cases. At the same time, we find that readers are more frequently visiting Wikipedia for shallow information needs compared to in-depth needs, and that they are both familiar and unfamiliar with the consumed topic with similar frequency. Interesting sub-patterns emerge, such as that deep information needs prevail mostly for readers with intrinsic learning motivations


% In this, we investigate user groups of our taxonomy more in-depth.
% In that direction, we find characteristic features that correlate with a certain motivation, a certain information need, or certain prior knowledge.


% \todo[Bob]{This `method' section should be shifted to }
% \todo[Florian]{Well, to where? Section 3.5? Seems also not to be a good place.}
% \todo[Florian]{In the end, we use a rather simplistic form of subgroup discovery, thus we could also rephrase the whole thing and just say that we sort features and remove non-significant ones. Might be easier to understand. Opinions?}


\para{Methodology. }
Due to our large set of features at interest (\Secref{sec:features}), we investigate behavioral reader patterns based on %For that purpose, we investigate the individual features 
 rule mining techniques, specifically \emph{subgroup discovery}~\cite{kloessgen1996,herrera2010}.
The general goal of subgroup discovery is to find descriptions of subsets of the data that show an interesting (\ie, significantly different) distribution with respect to a predefined target concept from a large set of candidates.
In our scenario, we perform a series of subgroup searches, each using one survey answer option as the target.
To create the search space of candidate subgroup descriptions, we use all features described in~\Secref{sec:features}.
For the topic features, we consider a topic as present in an article viewed by a user if our topic model provided a probability for this topic above $20\%$.
Other numeric features are binarized in five intervals
using equal-frequency discretization. Due to missing values and multiple occurrences of values, bin sizes can significantly deviate from 20\% of the dataset for some features.
To select the most interesting subgroups, we use the \emph{lift} as a quality function \cite{geng2006interestingness}. 
This measure is computed as the ratio between the likelihood of a survey answer in the subgroup and the respective likelihood in the overall dataset.
As an example, a lift of $1.3$ means that the respective survey answer is $30\%$ more likely to occur in the subgroup than in the overall data.
Additionally, we apply a filter to remove all subgroups that could not be shown to be significant by a $\chi^2$ test with a Bonferroni\hyp corrected threshold of $\alpha = 0.05$.

As a result, we obtain a list with the top $k$ interesting subgroups for each survey answer $T$.
For each subgroup $S$, we can compute various statistics: 
the (relative) \emph{size} $P(S)$ of the subgroup, \ie, the share of users that are covered by the subgroup description, 
the share $P(S|T)$ of subgroup users among those who answered with $T$ in the survey,
the \emph{target share} $P(T|S)$ in the subgroup, \ie, the share of users within the subgroup that reported the respective answer, and the \emph{lift,} which is defined as %\emph{lift (S)} =
$P(T|S)/P(T) = P(S|T) / P(S)$.
Note that the absence of a feature in the discussion does not mean that it was not considered, but that it is not among the most significant subgroups.

\para{Motivation. }
We start with characterizing groups with specific \emph{motivations} as reported in the survey.
In particular, we provide detailed results for two exemplary motivational groups (work/school and bored/random; \Tabref{tab:sgResults}) and only shortly summarize results for other motivations.

%When  Wikipedia  is  used  for  work  or  school  assignments, users tend to use a desktop computer to engage in long page views and sessions; sessions tend to be topically coherent and predominantly involve central, “serious” articles, rather than entertainment-related ones; search engine usage is increased, and sessions tend to traverse from the core to the periphery of the article network.

Users who intend to use Wikipedia for work or school are more frequently observed for specific topics of articles, namely
war \& history, mathematics, technology, biology \& chemistry, and literature \& arts.
For the first two of these topics, users are more than twice as often motivated by work or school tasks as on average.
While these topics cover a wide range of different areas, all of them are more related to academic or professional activities than for leisure.
Additionally, this type of motivation is more often reported by users accessing Wikipedia's desktop version. This could be expected since many work\slash school activities are performed in office settings.
Furthermore, we can see that this motivation occurs more often for users who are referred by external search engines multiple times in a session, and by users who stay longer on an individual page, which can be seen as a potential indicator for intensive studying.

By contrast, users who describe their motivation as\linebreak bored/random, are more likely to use internal navigation within Wikipedia and to spend only little time on the individual articles.
Also, they tend to switch topics between the individual articles more often (as indicated by the subgroup with a high average topic distance).
These are telltales for less focused browsing behavior.
Bored users also view more articles on Wikipedia both within the survey session and overall during the respective week.
Finally, this motivation can also be observed more frequently for articles that cover specific topics, such as sports, 21st century, and TV, movies,~\& novels. Clearly, these topics are more leisure\hyp oriented and are in stark contrast to the previously discussed topics favored by users who use Wikipedia for work or school.

%For other motivations, also interesting correlations can be observed, which we only outline due to limited space.
Due to limited space, we only outline findings for other motivations:
For example, motivation via media is significantly more often observed for the topics TV, movies,~\& novels (lift $1.37$) and 21st century (lift $1.26$), for popular articles, \ie, articles with a high number of pageviews (lift $1.17$), and for articles in the periphery of the Wikipedia link network according to pagerank (lift $1.14$).
The motivation of looking up something that came up in a conversation is more frequently reported for users with a single Wikipedia article request within a session (lift $1.08$) and for users of the mobile version of Wikipedia (lift $1.08$).
The current\hyp event motivation is more likely for articles about sports (lift $1.97$), 21st century (lift $1.49$), and education, government,~\& law (lift $1.49$). It is also more common for articles with many page views (lift $1.68$), possibly because articles on current events are trending.
Users who aim at intrinsic learning show a topic preference for more scholarly topics such as literature \& art  (lift $1.30$), mathematics (lift $1.24$), and technology (lift $1.21$).
Finally, the geographical origin of a user also has an effect:
the motivations personal decision, current event, and intrinsic learning are reported significantly more often for users from Asia (mostly India; lifts $1.46$, $1.44$, and $1.20$).

\para{Information need. }
Overall, the investigated subgroups are more homogeneous with respect to the reported information need.
We can, however, find some notable (anecdotal) exceptions:
Users from Asia describe their \emph{information needs} significantly more often as acquiring in-depth information (lift $1.51$).
For users who want to obtain an overview of a topic, using the desktop version of Wikipedia is more common than for the average user (lift $1.13$)
Also, topics play a certain role: fact look-ups, for example, are more often observed for the sports topic (lift $1.08$).
Session features that describe user behavior across multiple page visits do not lead to any significant differences in information need.

\para{Prior knowledge. }
Regarding readers' \emph{prior knowledge,} we can observe that users feel familiar with topics that are more spare-time oriented, such as sports (lift $1.21$), 21st century (lift $1.08$), and TV, movies,~\& novels (lift $1.07$).
They also feel more familiar about articles that are popular, \ie,  have many pageviews (lift $1.11$), are longer (lift $1.10$), and are more central in the link network (out-degree, in-degree, or pagerank; lifts $1.11$, $1.09$, and $1.08$).
Naturally, the answer ``unfamiliar'' is more often reported for the exact opposite of these subgroups.
Features that describe a user behavior over multiple article views do not lead to significant deviations.



043resultsummary.tex
\subsection{Summary of results}


\para{Prevalence of use cases.}
We have shown that Wikipedia is read in a wide variety of use cases that differ in their motivation triggers, the depth of information needs, and readers' prior familiarity with the topic. There are no clearly dominating use cases, and readers are familiar with the topic they are interacting with as often as they are not. Wikipedia is used for shallow information needs (fact look-up and overview) more often than for deep information needs. While deep information needs prevail foremost when the reader is driven by intrinsic learning, and fact look-ups are triggered by conversations, we saw that overviews are triggered by bored/random exploration, media coverage, or the need for making a personal decision. 

\para{Use cases over time.}
Motivations appear to be mostly stable over time (days of the week and hours of the day), with a few exceptions: motivations triggered by the media are increased over the weekends and at nights, conversation triggers are increased over the weekends, and work/school triggers are increased on week days and during the day.
%We also showed that motivations are associated with distinct behavioral patterns. For example, when Wikipedia is used for work or school assignments, users tend to use a desktop computer to engage in long page views and sessions; sessions tend to be topically coherent and predominantly involve central, ``serious'' articles, rather than entertainment-related ones; search engine usage is increased, and sessions tend to traverse from the core to the periphery of the article network.
%By contrast, people who use Wikipedia out of boredom or in order to explore randomly tend to be power users; they navigate Wikipedia on long, fast-paced, topically diverse link chains and they often visit popular articles on entertainment-related topics, less so on science-related topics

\para{Behavioral patterns.} By connecting survey responses with webrequest logs, we identified certain behavioral patterns:


%%%OLD VERSION OF BEHAVIORAL PATTERNS
\begin{itemize}[leftmargin=*, topsep=0pt,itemsep=-1ex,partopsep=1ex,parsep=1ex]
%\denselist
    \item When Wikipedia is used for work or school assignments, users tend to use a desktop computer to engage in long pageviews and sessions; sessions tend to be topically coherent and predominantly involve central, ``serious'' articles, rather than entertainment\hyp related ones; search engine usage is increased; and sessions tend to traverse from the core to the periphery of the article network.   
    \item Media-driven usage is directed toward popular, entertainment\hyp related articles that are frequently less well embedded into the article network.
    \item  Intrinsic learning tends to involve arts and science articles with no significant navigational features; conversations bring infrequent users to Wikipedia, who engage in short interactions with the site, frequently on mobile devices. 
    \item People who use Wikipedia out of boredom or in order to explore randomly tend to be power users; they navigate Wikipedia on long, fast-paced, topically diverse link chains; and they often visit popular articles on entertainment-related topics, less so on science-related topics. 
    \item Current events tend to drive traffic to long sports and politics-related articles; the articles tend to be popular, likely because the triggering event is trending.
\item When Wikipedia is consulted to make a personal decision, the articles are often geography and technology-related, possibly due to travel or product purchase decisions.
\end{itemize}
050discussion+relwork.tex
\section{Discussion}
\label{sec:discussion}

Every day, Wikipedia articles are viewed more than 500 million times, but so far,
very little has been known about the motivations and behaviors of the people behind these pageviews.
The present study is the first comprehensive attempt to help us understand this group of users by combining a survey with a log\hyp based analysis.

The work most closely related to ours is by Lehmann \etal\ \cite{lehmann2014reader}, who extracted \WP navigation traces from Yahoo!\ toolbar logs (which may be considered a biased sample of the complete logs we have access to) with the goal of discovering a set of usage patterns according to which articles are consumed.
Using clustering techniques, they concluded that there are four types of articles:
trending articles, articles read in a focused manner, articles read by exploring users, and articles users just quickly pass through.
Lehmann \etal's work is entirely ``unsupervised'', in the sense that they have no ground truth of the actual underlying user motivations and needs.

We, on the contrary, have elicited the ground truth through our survey and can thus arrive at stronger and more actionable conclusions, which we discuss next.
We do so by first highlighting implications and directions for future work (\Secref{sec:Implications and future directions}), and then reflecting on our methodology and pointing out its limitations (\Secref{sec:Reflections on methodology}).


\subsection{Implications and future directions}
\label{sec:Implications and future directions}
This research has already had considerable impact within the Wikimedia Foundation, where it has informed several items on the product development agenda, and we hope that it will further inspire Wikimedia developers, academic researchers, and volunteers to build tools for improving the user experience on \WP.

%Wikipedia editors, the Wikimedia Foundation, and third\hyp party developers
%to offer tools and services that satisfy the needs of Wikipedia readers more closely.
%They also can empower Wikipedia editors to shape the content in Wikipedia in ways that can reinforce what they consider an online encyclopedia is, given that for many people on the Web learning about a topic involves reading Wikipedia.
%In this section, we explain some of these opportunities and discuss areas that require further research.
%Today many volunteer editors contribute content to Wikipedia without deeper knowledge of the users that read their content which has potential impact on what depth and type of content is being added to Wikipedia.
%First, editors currently contribute to content on \WP without much knowledge of the users who will eventually read this content. By imparting on editors an understanding of what brings readers to \WP and what depth of knowledge they are seeking, our findings can thus improve editors' efficacy by helping them make their contributions maximally targeted and useful.

\para{Predicting motivation and desired depth of knowledge.}
A tool immediately suggested by our results could involve statistical models for real-time inference of user session motivations from behavioral traces as captured in the webrequest logs.
Such models could be trained in a supervised fashion with features of \Secref{sec:webrequest_logs} as input, and survey responses as output, and could form the basis for products and services for supporting the needs of \WP readers more proactively.
For instance, if an editor working on an article could be shown an estimate of the distribution of the motivations and desired depths of knowledge on behalf of the readers of the article, she can take this information into account to tailor the content to the needs of the audience or attempt to change the distribution of the audience's motivation by creating specific types of content in the article. 
Such a tool could have large impact, considering that, currently, editors contribute to content on \WP without much knowledge of the users who will eventually read it.

Similarly, predicting the distribution over depths of knowledge sought by the readers of an article could offer opportunities for creating different versions the article, \eg, for those who are interested in quick look-ups \vs\ in-depth readers.
This could enhance the usability of Wikipedia articles particularly on mobile devices with smaller screens and low\hyp bandwidth connections.

The above task of using digital traces to predict survey responses has been called \emph{amplified asking,} and it is known to be difficult \cite{salganik2017bit}.
This has been confirmed by our preliminary attempts, where we have achieved accuracies only slightly better than simple baselines.
This may be partly explained by the fact that user motivations may change during a session, and while the survey captures the motivations at the article level accurately, it fails to capture possible transitions between motivations during a session.
For instance, a session might start with a school or work project in mind, but the user might then transition to procrastinating by exploring \WP randomly, which would not be captured in our current setting.
Also, prediction is complicated by the fact that, even for a fixed article, user motivations might vary widely.
For instance, of the $222$ users taking the survey upon reading the article about Donald Trump, $38\%$ read the article out of boredom, $32\%$ in response to media coverage, $24\%$ because of a conversation, $23\%$ due to current events, $17\%$ because the topic was personally important to them, \etc

% motivation_bored/random: 85
% motivation_conversation: 53
% motivation_current event: 52
% motivation_intrinsic learning: 38
% motivation_media: 71
% motivation_no response: 4
% motivation_other: 41
% motivation_personal decision: 24
% motivation_work/school: 19

Despite these difficulties, future work should investigate the problem of predicting user intentions in more depth.

%Second, search engines play a major role in driving traffic toward \WP. Since a principal objective of search engines is to serve results that help users efficiently achieve their goals, an understanding of the goals that drive users to \WP may be useful for improving the ranking of \WP specific articles among search results.





%\note[Leila]{There are a few features that the WMF has developed in the past and now is justifying its development based on this study. These are: related pages (in the bottom of the articles, they recommend what to be read next.), Link Preview on mobile site (that will allow you to see the first few sentences of the leading paragraph of an article when a hyperlink is clicked, and the feed: which suggests articles to be read. I'd be very careful about highlighting these in the paper. There is something that is not understood/acknowledged widely and that's the value of serendipity in exploring Wikipedia. Many of the features that are developed or may be developed based on the result of this research may take away serendipity from Wikipedia. Wikipedia is one of the few places left on the internet where you can go and read without thinking that you are being watched or content is being adapted to you. This is potentially a good thing, something that needs to be consciously preserved and very carefully changed. I'll think about how/if we want to say this in the discussion section.}






%- understand specific motivations better: \eg, intrinsic learning (how can we help users in effectively understanding complex topics over longer periods of time?); bored/random (how can we turn this potential waste of time into a worthwhile activity?); current events, media (what aspects of a current event are most sought-after?)

%- intrinsic learning; school: can we combine reading and editing to achieve better pedagogical results? \citemissing

% \xhdr{Future studies}
% - survey: fairly light intervention
% - might justify more costly studies and experiments in the future
% - TODO: come up with a couple of experiments (\eg, A/B tests), surveys, or focus groups that would be justified by our results.
% (\eg, a survey setup where users are asked at the beginning of the session what they're trying to achieve, and at the end of the session, whether they were successful)
% also: can we design a study to determine to what extent WP usage is constrained by the current structure and interaction modes of WP?
% (\eg, it might be the case that more people would use WP for work/school assignments if WP offered specific functionality for these purposes).



\subsection{Methodological limitations}
\label{sec:Reflections on methodology}

We discuss certain limitations of present research next.

\xhdr{Survey selection bias}
A general caveat with surveys is that one typically cannot guarantee that whether a subject participates or not is a fully random choice.
Certain covariates may be associated with both participation rates and responses given, leading to biased conclusions.
We made a best effort to correct for this bias by adjusting survey responses based on a random sample of all \WP pageviews drawn from \WP's webrequest logs (\Secref{sec:correction}).
However, if the bias\hyp inducing covariates are hidden, one cannot fully correct for the bias.
For instance, young users might be both more prone to use \WP for work or school and to participate in our survey; this would over\hyp represent the work\slash school motivation in our raw survey results, and since we have no information about users' age, we could not correct for this bias.
%%%%%%%%%%% The following two items may be discarded if we need space.
Apart from that, survey answers might be biased by social desirability~\cite{demaio1984social};
\eg, even in an anonymous survey, users might be reluctant to admit they are visiting Wikipedia out of boredom.
%Furthermore, since we are investigating a global readership, answers might be influenced by different notions and associations of the survey phrasing across cultures \cite{harkness2003cross}.

% TODO for camera-ready: sensitivity analysis. Here's a method for assessing the sensitivity of inverse-probability-to-treat weighting (the bias correction method we use): https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3777387/


\xhdr{Unique visitors and level of analysis}
Wikipedia does not require users to log in, nor does it use cookies in webrequest logs to maintain a notion of unique clients.
Hence, we need to rely on an approximate notion of user IDs based on IP addresses and browser versions (\Secref{sec:webrequest_logs}), which makes the attribution of pageviews to users and the construction of sessions imperfect.
In particular, we might not recognize that two pageviews are by the same user if they use several devices or if their IP address changes for other reasons; and we might conflate several users if they share the same device or IP address (\eg, via a proxy).
Currently, we limit the impact of such errors by 
analyzing the data on a session level and
operating at relatively short time scales (an inactivity of more than one hour ends the session being studied).
If the attribution of pageviews to unique users becomes more precise in the future, we could study user behavior at longer time scales, which would, \eg, allow us to understand and support long-term learning needs.
Also, our current method aims at giving each user session equal weight. An alternative approach would be to analyze the data on a request level, which would put more emphasis on the motivations and needs of power users.

\xhdr{Cultural issues}
The results discussed here pertain to the English edition of Wikipedia.
Even within this limited scope, our behavioral analysis hints at subtle cultural and geographical differences; \eg, the bored\slash random motivation is particularly frequent in the U.S., whereas current events are a stronger motivator in India.
Survey answers might also be influenced by different notions and associations of the survey phrasing across cultures \cite{harkness2003cross}.
Since \WP strives to reach beyond cultural and linguistic boundaries, it is important to further investigate these cultural issues.
As part of this effort, we are planning to repeat our study in additional language versions of \WP to elicit cultural differences on a larger scale.


% there is also no notion of a unique device which means that building sessions for a device relies on using heuristics that rely on IP address, user agent, and X-Forwarded-For headers.
% This means that the behavior of Wikipedia readers at the person level cannot be studied at the moment (each person may access Wikipedia via multiple devices during the day/week and each of these accesses, at best, are considered from separate from each other).


% \textbf{The depth of content in Wikipedia.} Only 20\% of the survey participants have reported the need for in-depth knowledge, when compared to quick look-up or overview. This can be due to the fact that those who are seeking in-depth knowledge are less likely to accept interruptions and participate in a survey. However, if future research can confirm that this number is as low as reported in this study, this information can be used to reprioritize editing on Wikipedia. 
% \inote{I would consider this as another methodological limitation: there might be biases that we cannot correct for, e.g., that users with a certain information need are inherently less likely to answer the survey; We should join his with the age/gender argument -flo}


%As all surveys, our survey might be subject to different fundamental kinds of biases.
%While we do correct for non-responses with respect to a wide variety of features based on the user behavior within Wikipedia, we cannot control for other influence factors such as age or gender due to lack of information.
%Finally, it might also occur that there is direct influence of the motivation on the likelihood to answer a survey that does not manifest in any other observable properties, which is a general limitation of survey-based approaches.
%\inote{So far that sound too negative, but may serve as a bridge to future work}

%- general difficulty of introspection in surveys

%- bias correction: sampling users \vs sampling sessions \vs sampling page views

%- level of analysis: user level, session level, page level


%- India?! different understanding of what answers mean?


% \subsection{Related work}
% \label{sec:relwork}

% - Understanding patterns and motivations of various user behaviors on the Web:

% Web search:
% A taxonomy of web search
% \cite{broder2002taxonomy},
% Who uses web search for what: and how
% \cite{weber2011uses},
% Understanding user goals in web search
% \cite{rose2004understanding}

% Web browsing:
% A characterization of online browsing behavior
% \cite{kumar2010characterization},
% Who Does What on the Web: A Large-Scale Study of Browsing Behavior
% \cite{goel2012does}

% Twitter:
% Why we twitter: understanding microblogging usage and communities
% \cite{java2007we},
% What is Twitter, a social network or a news media?
% \cite{kwak2010twitter}

% Wikipedia:
% Overview: \cite{okoli2012people}

% Behavior of editors within WP \citemissing \cite{arazy},
% and beyond WP \cite{west2012drawing}.

% Reader behavior:
% What is popular? \cite{spoerri2007popular}
% Search queries leading to WP \cite{waller2011search}
% Navigation in the wild \cite{paranjape2016improving},
% papers on clickstream dataset \citemissing,
% navigation in game context \cite{west2012human,singer2015hyptrails,helic2013models}.

% Most closely related: \cite{lehmann2014reader}
% different: no-one has combined survey and logs

% \todo[Bob]{@Leila: Maybe you could check at WMF if we're missing any major pieces of work? For instance on surveys about WP usage, reader/editor demographics, etc.}




% - additional properties to include in analysis
% While we have studied a large set of features based on different characteristics within this paper, future work can extend this arsenal in order to further advance our understanding of reader behavior on Wikipedia. Exemplary features not considered in this paper are: edit features of visited articles such as the number of edits (i.e., revisions), quality of visited articles as assessed by Wikipedia's internal assessment tool, reading time of individual articles (we capture this with sessions though)

% Wikipedia has been studied in a multitude of different settings, e.g., contrasting collaborative editing efforts and content quality \cite{kittur2008harnessing,stvilia2008information}, leveraging these rich and large-scale content structures \cite{gabrilovich2007computing}, or understanding emerging underlying topological link networks \cite{capocci2006preferential}. However, as emphasized also in \cite{lehmann2014reader} and \cite{okoli2012people}, relatively little is known about the huge population of readers and their behavior. Previous work in that direction has mostly focused on studying (i) aggregated pageview statistics, and (ii) human navigation on Wikipedia. 

% \para{Reader article popularity on Wikipedia.}
% In \cite{lehmann2014reader}, the authors studied reader preferences and behavior on Wikipedia as measured by article view popularity supplemented by navigational data. The behavioral aspects were then contrasted with   underlying content and editor behavior. Results reveal that the most popular articles do not always match with the most edited and high content quality articles. Also, the paper suggests that popular articles are read according to focus, trending, exploration and passing mechanisms, but that how an article is read might change over time. Anselm Spoerri \cite{spoerri2007popular} found that the most visited articles on Wikipedia are mostly related to topics about entertainment (TV, celebrities, etc.). Additionally, popularity of Wikipedia articles is related to Web search query popularity. These observations were also confirmed in \cite{lehmann2014reader,waller2011search}, also highlighting the popularity of history, technology, and health related topics. This suggests that article popularity on Wikipedia is more focused on ``lighter'' topics instead of typical encyclopedian topics.

% The work in \cite{reinoso2011temporal} studied reading and editing activity in different language editions of Wikipedia. Results suggest temporal periodicity in visits, searches and requests to edit. For some language editions such as English or German Wikipedia, there appears to be a strong correlation between visits and edits, while this is not the case in other editions like the Japanese. Temporal popularity patterns have also been studied specifically with respect to bursty behavior \cite{kampf2012fluctuations}, which also can be utilized for tasks like movie popularity prediction \cite{mestyan2013early}.

% In this work, we have gone beyond previous work focusing on article popularity by carefully conducting and studying reader behavior with a survey focusing on capturing motivational aspects. By coupling survey responses with specific requests on Wikipedia, we have been able to get further insights into reader behavior on Wikipedia.

% \para{Human navigation on Wikipedia.}
% Human reader behavior has also been studied within the context of navigation. Humans move through Wikipedia's large-scale underlying topological link network when successively accessing articles. 
% Prominently, traces emerging from Wikipedia navigational game data has been of interest \cite{west2012human,singer2013,singer2015hyptrails,west2009wikispeedia,helic2013models}. Results e.g., have shown that humans have a preference towards navigating semantically and topical similar articles on Wikipedia \cite{west2012human,singer2013,singer2015hyptrails}, that trade-offs between similarity and degree exist \cite{west2009wikispeedia}, or that different navigational phases---exploration and exploitation---are present \cite{helic2013models}. Insights in browsing behavior have then also been utilized to inferring missing links \cite{west2015mining} or improving the link structure of Wikipedia \cite{paranjape2016improving} .
% In this work, we have focused on navigational behavior when contrasting survey responses with behavioral aspects.
060conclusion.tex
\section{Conclusions}
\label{sec:conclusions}
In this work, we study why users read Wikipedia.
We use survey data to develop a taxonomy of Wikipedia usage along three dimensions: \emph{motivation}, \emph{information need}, and \emph{prior knowledge}.
In a large-scale survey with almost 30,000 participants, we quantify the share of readership for these driving factors.
The bias\hyp corrected survey results reveal a broad range of usage scenarios, interdependencies between survey answers, and temporal trends.
Combining the survey responses with webrequest logs allows us to  characterize motivational groups with behavioral patterns.
The outcomes of this study are currently being discussed in the Wikimedia Foundation as a stimulus for developing specialized tools for readers and editors.
%\todo[fle]{This is a first draft only, feel free to edit}

%In this work, we have studied reader characteristics, motivations, and behavior. To that end, we conducted a robust survey capturing readers' motivations, information needs, and prior knowledge. By accounting for response biases by utilizing propensity score weight adjustment, we have discussed representative responses and have identified certain trends, such as that a high share of users is motivated to read Wikipedia based on extrinsic triggers such as media coverage, conversation, or work/school. Another group of readers have intrinsic motivations, such as learning or boredom. These motivational aspects are coupled with certain specific information needs and prior knowledge; e.g., intrinsic learning motivations have a higher share of readers that are already familiar compared to readers motivated by media who are more likely to be unfamiliar with the topic at hand.

%Additionally, we coupled survey responses with rich underlying log-data, and content and topological article features from Wikipedia. We applied subgroup discovery (a variant of rule mining) that aims at identifying interesting behavioral subgroups. For instance, we found that readers motivated by work or school have higher tendencies to visit articles in the core of the topological Wikipedia link network, articles about mathematical topics, or that they have preferences of accessing Wikipedia by desktop compared to the overall population of readers. These and similar insights do not only advance our understanding of reader behavior, but might also have impact of aspects like the development of Wikipedia, by e.g., characterizing articles by motivational reader proportions and using this information for informing decisions of Wikipedia editors. In future work, we aim at advancing into this research direction, specifically, with a focus on tool-assisted editing, or even building recommendation structures for different behavioral aspects with the ultimate goal to further improve the general quality of Wikipedia adhering to behavioral reader aspects.






% \tikzstyle{decision} = [diamond, draw, fill=blue!20, 
% text width=6em, text badly centered, node distance=3cm, inner sep=0pt]
% \tikzstyle{data} = [circle, draw, fill=blue!20, 
% text width=6em, text badly centered, node distance=3cm]
% \tikzstyle{block} = [rectangle, draw, fill=blue!20, 
% text width=9em, text centered, rounded corners, minimum height=4em]
% \tikzstyle{block2} = [rectangle, draw, fill=yellow!20, 
% text width=9em, text centered, rounded corners, minimum height=4em]
% \tikzstyle{line} = [draw, -latex']
% \tikzstyle{cloud} = [draw, ellipse,fill=red!20, node distance=3cm,
% minimum height=4em]

%\begin{figure*}[!ht]
%	\centering
%	%\hspace*{-135pt}
%	\begin{tikzpicture}[node distance = 5cm, auto]
%	\node [block] (survey) {\Large Conduct survey on Wikipedia};
%	\node [data, below of = survey, node distance=3cm] (survey_data) {\Large Survey data};
%	\node [block, right of = survey] (requests) {\Large Prepare requests from Wikipedia logs};
%	\node [data, below of = requests, node distance=3cm] (requests_data) {\Large Requests data};
%	\node [block, right of = requests] (sample) {\Large Sample requests (like survey)};
%	\node [data, below of = sample, node distance=3cm] (sample_data) {\Large Sample data};
%	\coordinate (middle) at ($(survey_data)!0.5!(requests_data)$);
%	\node [block, below of = middle, node distance=3cm] (survey_requests_join) {\Large Join survey and request data};
%	\node [data, below left of = survey_requests_join, node distance=4cm] (survey_requests_data) {\Large Survey and request data};
%	\coordinate (middle) at ($(requests_data)!0.5!(sample_data)$);
%	\node [block, below of = middle, node distance=3cm] (sample_requests_join) {\Large Join sample and request data};
%	\node [data, below right of = sample_requests_join, node distance=4cm] (sample_requests_data) {\Large Sample and request data};
%	\coordinate (middle) at ($(survey_requests_join)!0.5!(sample_requests_join)$);
%	\node [block, below of = middle, node distance=2cm] (features) {\Large Generate features};
%	\path[->] (features) edge [in=270,out=280,loop] node[auto,align=center] {survey\\ features} ();
%	\path[->] (features) edge  [in=310,out=320,loop] node[auto,align=center] {article\\ features} ();
%	\path[->] (features) edge  [in=220,out=230,loop] node[auto,align=center] {session\\ features} ();
%	\path[->] (features) edge  [in=170,out=180,loop] node[auto,align=center] {request\\ features} ();
%	\coordinate (middle) at ($(survey_requests_data)!0.5!(sample_requests_data)$);
%	\node [block, below of = middle, node distance=3cm] (bias_compare) {\Large Check for survey bias};
%	\node [decision, below of = bias_compare, node distance=3cm] (bias_compare_check) {\Large Bias?};
%	\coordinate (middle) at
%	($(bias_compare)!0.5!(bias_compare_check)$);
%	\node [block, left of = middle, node distance=3cm] (correct_bias) {\Large Correct for bias};
%	\node [block2, left of = correct_bias, node distance=4cm] (empirical) {\Large Empirical analysis};
%	\node [block2, left of = survey_requests_join, node distance=4.5cm] (predict) {\Large Predict survey};
%	
%	%
%	\path [line] (survey) -- (survey_data);
%	\path [line] (requests) -- (requests_data);
%	\path [line] (sample) -- (sample_data);
%	\path [line] (requests_data) -- (sample);
%	\path [line] (survey_data) -- (survey_requests_join);
%	\path [line] (requests_data) -- (survey_requests_join);
%	\path [line] (sample_data) -- (sample_requests_join);
%	\path [line] (requests_data) -- (sample_requests_join);
%	\path [line] (survey_requests_join) -- (survey_requests_data);
%	\path [line] (sample_requests_join) -- (sample_requests_data);
%	\path [line] (features) -- (sample_requests_data);
%	\path [line] (features) -- (survey_requests_data);
%	\path [line] (survey_requests_data) -- (bias_compare);
%	\path [line] (sample_requests_data) -- (bias_compare);
%	\path [line] (bias_compare) -- (bias_compare_check);
%	\path [line] (bias_compare_check) -- node {\Large yes}(correct_bias);
%	\path [line] (correct_bias) -- (survey_requests_data);
%	\path [line] (survey_requests_data) -- (empirical);
%	\path [line] (survey_requests_data) -- (predict);
%	\path [line] (predict) -- (survey_data);
%	\end{tikzpicture}
%	\caption{Flow diagram}
%\end{figure*}
070appendix.tex
%\appendix
%\section{Survey bias correction}
\section*{APPENDIX: SURVEY BIAS CORRECTION}
\label{ap:bias}

This appendix covers the details of the survey bias correction.

\para{Propensity score weight adjustment.}
We use \emph{inverse propensity score weighting} to adjust for potential biases in survey response data with respect to control data \cite{austin2011introduction,lunceford2004stratification}.
Specifically, we want to infer unbiased estimates of survey answers for the whole Wikipedia readership. Thus, we randomly sampled a large set of Wikipedia readers (25 times the number of survey responses) from the webrequests logs in the survey period. Then, we proceeded to sample one request for each selected user and marked it as an imaginary request reflecting a potential survey response; we also deduced the same set of features as for our survey (except responses). We only sampled requests that are desktop or mobile pageviews in English Wikipedia's main namespace and applied bot-filtering in order to match the original survey.


% The main idea is to compute for each user that participated in the survey a propensity score, i.e., the (relative) likelihood of survey participation for this user, with the feature set of ~\Cref{sec:features} as control features. 
%\phsi{references in this section missing}
The propensity score of a single instance then reflects the probability that an instance with these control features (\Secref{sec:features}) participated in the survey. We approximate it using our control group.
For that, a post-stratification approach \cite{lunceford2004stratification} is infeasible due to the large number of control features we consider.
Instead, we model the group membership (survey participant or control group) using gradient boosted regression trees showing promising results in the past in comparison to traditional approaches like logistic regression \cite{lee2010improving}.
% For that purpose, we consider two types of models: logistic regression and gradient boosting.
% Logistic regression is the established choice in survey methodology, but boosting method have been reported to competitive propensity score estimations \mst{needs rewriting}. 
Given the features of an instance $x$, the model predicts a probability $p(x)$ that $x$ belongs to the survey group.
We then set the weight $w$ for instance to
$1/p(x)$.
%$\frac{1}{p(x)}$.
The rationale behind this procedure is that answers of users that are overall less likely to participate in the survey receive higher weights since they represent a larger part of the entire population with similar features.


% \para{Control group.}
% We want to induce representative interpretations about the reader population on Wikipedia making it necessary to adjust for confounders in our survey responses as discussed. Thus, for our control group, we randomly sampled Wikipedia readers from the Wikipedia logs of all user requests in the same March period as our treatment group (survey).
% Then, we proceeded to sample one request for each selected user and marked it as an imaginary request reflecting a potential survey response; we also deduced the same set of features as for our survey (except responses). We only sampled requests that are desktop or mobile web page views on main English Wikipedia namespace articles to match the original survey. The propensity score then reflects the probability whether a sample with certain features belongs to the original survey responses or our control group.

% For implementing propensity score weight adjustment, we need a control group that reflects the overall reader population on Wikipedia for which we aim at deducing representative interpretations.
% To that end, we drew a simple random sample of Wikipedia readers from the Wikipedia logs of all user requests. We focused on users with at least one request in the first week of march in 2016. Then, we proceeded to sample one request for each selected user and marked it as an imaginary request reflecting a potential survey response. We only sampled requests that are desktop or mobile web page views on main English Wikipedia namespace articles to match the original survey.
% For each of the sampled request we compute then the features as described in~\Cref{sec:features}.
% The propensity score then reflects the probability whether a sample with certain features belongs to the original survey responses or our control group.
%Next, we join the datasets for the survey with the random sample. 


\para{Evaluating weights.}
To evaluate if applied weighting schemes have the intended correcting effect of making the user survey data more representative for the overall Wikipedia, we resort to two scenarios.

First, we check that the resulting weights do not contain drastic outliers dominating subsequent results, which would warrant so-called \emph{trimming} \cite{lee2011weight}. In that regard, we observe that weights are sufficiently homogeneous distributed with a minimum of $1$, a maximum of $190$, 
a mean of $17.6$, and a standard deviation of $26.9$.

% First, we investigate a typical problem for inverse propensity weighting that occurs especially if the fit of the models employed to compute the propensity score is bad; sometimes few instances receive overproportional high weights that completely dominate the results. 
% However, in our data the weights are relatively evenly distributed. The maximum weights are below $14$ (gradient boosting) or $21$ (logistic regression), respectively. The vast majority of weights (93\% or 99.9\%) are below $5$. \phsi{should we show the dist?}
% In Figure~\ref{fig:survey_response} we also show how the determined weights change the relative shares of survey responses. While we can see slight adjustments, the general direction is kept intact.

Additionally, we evaluate how well we can recover the mean value of features in the overall population from observed survey response features 
%from the users that participated in the survey 
and our weighting scheme.
For that purpose, we compute weighted and unweighted averages of the observed values for the survey users and compare them with the mean of a different random sample as a ground truth.
As a result, the average of relative errors is reduced by $86\%$, from $0.556$ in the unweighted case to $0.079$ in the weighted case.
The reduction is strongly significant ($p \leq 0.001$ according to a Wilcoxon signed rank test).
If weighting is applied, then the mean recovered from the weighted survey is never more than $0.2$ standard deviations off compared to the actual feature mean in the sample.


%Results also imply that for most of the features, the bias in the survey data is very small, i.e., less than $0.1$ standard deviations for article features, but is significantly higher for session behavior features such as session length, or the session time. 

%\phsi{should we somehow show that this error reduction holds for all features?} \phsi{mb also use the other evaluation technique with weights:see next comment by flo} \fle{Maybe add reduced accuracy of the weighted survey/sample classifiers?}


% The results presented in the remainder of the paper focus on the weights obtained from gradient boosting due to their slightly lower error in recovering feature means as well as their higher flexibility in model specification.
% However, the respective results for the logistic regression weights deviate only little from that, also because there is a strong correlation ($\rho = 0.71$) between the two weighting vectors.

%\note[Philipp]{Maybe we can remove the whole concept of design effect and only talk about effective sample size}

\para{Effective sample size.}
Int this work, we employ a variety of statistical techniques on the survey data. Yet, the introduction of sample weights for correcting bias in survey responses leads to violations of IID assumptions \cite{mukhopadhyay2016complex}.  
Thus, standard errors of estimators are estimated as too small, which in turn leads to confidence intervals being too narrow and statistical tests asserting significance too often if standard procedures are applied. 
%The effect that our survey design and correction has on the variance is known as \emph{design effect} (deff), first introduced by Kish \cite{kish1965survey}, capturing the extent to which the sampling error in the survey for some parameter $\theta$ deviates from the expected error from an IID sample. 
%T \emph{design effect} (deff), first introduced by Kish , 
The extent to which the sampling error in the survey for some parameter $\theta$ deviates from the expected error from an IID sample due to survey design and correction, is known as the \emph{design effect} (deff) \cite{kish1965survey}. 
If the design effect deviates from $1$---as it is the case in our survey---then our understanding of sample size for calculating standard errors becomes incorrect. 
%The propensity score weighting distorts our previous survey sample size which has an impact on statistical inference and tests that mostly assume complete random sampling; specifically, it affects the standard error of estimates which varies with sample size. 
To that end, we consider the \emph{effective sample size} estimating the required sample size of a random sampling survey for achieving the same error as the weighted sample---it is defined as $n_{\text{eff}}=n/\text{deff}$. As we cannot directly calculate $\text{deff}$ without knowing the expected sampling error, we use Kish's approximation formula with weights $w_i$ \cite{kish1965survey}:

%\vspace{-1em}
\begin{equation*}
   n_{\text{eff}} =
   \frac{\left(\sum_{i=1}^n w_i\right)^2}{\sum_{i=1}^n w_i^2} 
\end{equation*}

For our complete survey data, $n_{\text{eff}}=8839$.
We use this effective sample size throughout this article for calculating standard errors, confidence intervals, and statistical tests. Note that this makes reported confidence interval and statistical hypothesis tests overly careful. For further details, please refer to \cite{mukhopadhyay2016complex}.

\para{Acknowledgements.}
We thank Dario Taraborelli from Wikimedia Foundation who was indispensable to the early phases of the project. We also thank Jonathan Morgan for helping us with the hand-coding; Jon Katz and Toby Negrin for helping us shape the direction of the research and supporting us throughout; Anne Gomez, Jeff Hobson, Bahodir Mansurov, Jon Robson, and Sam Smith for running the surveys on Wikipedia; and Aeryn Palmer for creating the privacy statements for this research. This research has been supported in part by NSF IIS-1149837, ARO MURI, DARPA NGS2, and Stanford Data Science Initiative.
feature_table.tex
\begin{table*}[t!]
%\vspace{-1em}
\centering
\small
\caption{\emph{Features.} This table describes all features utilized in this work. \emph{Survey features} capture responses to our survey questions; \emph{request features} capture background information about the respondent mined from webrequest logs; \emph{article features} describe the requested Wikipedia article; and \emph{session\slash activity features} are derived from the entire reading session and beyond\hyp session activity.
% Responses denote the survey responses, survey features refer to all features directly extracted from survey data, article features reflect content and topological features associated with visited article of the survey respondent, and session features extend this notion by looking at features associated with the whole session (partly also with all requests) of a reader participating in the survey.
}
\vspace{-1em}
\begin{tabularx}{0.99\textwidth}{l|l|X} 
\toprule 
 & feature & description \\ 
\midrule
%\multirow{9}{*}{\rotatebox[origin=c]{90}{survey features}}
& motivation & Type of motivation for reading an article, as selected by respondent in survey. As multiple responses were allowed, we work with boolean dummy variables for each motivation.\\
\raisebox{.8\normalbaselineskip}[0pt][0pt]{\rotatebox[origin=c]{90}{survey~}} & information need & Information need for reading an article, as selected by respondent in survey.\\ 
& prior knowledge & Prior knowledge about the topic before visiting the article, as selected by respondent in survey.\\ \hline
& country & Country code of respondent in survey (e.g., USA) derived from the IP address. \\
  & continent & Continent of respondent in survey (e.g., North America) derived from the IP address. \\ 
& local time weekday & Local weekday of survey request detected by timezone information (Mon-Sun).  \\
\raisebox{1\normalbaselineskip}[0pt][0pt]{\rotatebox[origin=c]{90}{request~~~}} & local time hour & Local hour of survey request detected by timezone information (0-24).\\
 & host & Requested Wikipedia host: ``desktop'' (en.wikipedia.org), or ``mobile web'' (en.m.wikipedia.org). \\
& referer class & Referer class of request (none, internal, external, external search engine, or unknown).\\ 
\hline
& article in-degree & The topological in-degree of an article. \\
& article out-degree & The topological out-degree of an article. \\
& article pagerank & The unnormalized pagerank of an article; calculated with damping factor of 0.85. \\
& article text length & The text length of an article as extracted from HTML---number of characters. \\
& article pageviews & The sum of pageviews for the article in same time period as survey. \\
\raisebox{0\normalbaselineskip}[0pt][0pt]{\rotatebox[origin=c]{90}{article~~~}} & article topics & Probability vector for 20 topics as extracted by LDA from a bag-of-words representation. Topics were manually labeled as follows: (1) transportation \& modern military, (2) biology \& chemistry, (3) South Asia, Middle East, (4) mathematics, (5) 21st century, (6) TV, movies, \& novels, (7) Britain \& Commonwealth, (8) East Asia, (9) Spanish (stubs), (10) war, history, (11) geography (unions, trade), (12) literature, art, (13) education, government, law, (14) 20th century, (15) sports, (16) United States, (17) numbers, (18) technology, energy, \& power, (19) music, and (20) geographical entities. We use the probabilistic topic distribution as 20 individual features for each article. \\
& article topic entropy & Measures the topic specificity of an article from LDA probability vector. \\ \hline
& session length & The number of requests within the session. \\
& session duration & Total time spent in the session in minutes. \\
 & avg. time difference & Average time difference between subsequent session requests (i.e., dwelling time). \\
& avg. pagerank difference & Average pagerank difference between subsequent session requests (i.e., stating whether readers move to periphery or core).\\
 & avg. topic distance & Average Manhattan distance between topic distributions for subsequent session requests (i.e., capturing topical changes). \\
\raisebox{-0\normalbaselineskip}[0pt][0pt]{\rotatebox[origin=c]{90}{~~~~~session/activity}} & referer class frequency & For each referer class (see survey features): frequency in session. \\
& session article frequency & The number of times requested article for survey response occurs within the session. \\
& session position & Relative position inside a session when answering the survey. \\
& num. of sessions & The total number of sessions for respondent in survey time period. \\
& num. of requests & The total number of requests for respondent in survey time period. \\
\bottomrule 
\end{tabularx} 
\label{tab:features}
\vspace{-1em}
\end{table*}
macros.tex
%\newcommand{\sg}{sg}

\newcommand{\phsi}[1]{\textbf{\color{black}/* #1 (phsi) */}}
\newcommand{\fle}[1]{\textbf{\color{black}/* #1 (fle) */}}
\newcommand{\jure}[1]{\textbf{\color{red}/* #1 (jure) */}}
\newcommand{\leila}[1]{\textbf{\color{blue}/* #1 (leila) */}}
\newcommand{\bob}[1]{\textbf{\color{green}/* #1 (bob) */}}
\newcommand{\mst}[1]{\textbf{\color{red}/* #1 (mst) */}}


\newcommand{\footnoteurl}[1]{\footnote{\url{#1}}}
\newcommand{\para}[1]{\smallskip\noindent\textbf{#1}}
%\newcommand{\subpara}[1]{\smallskip\noindent\emph{#1}}
\newcommand{\subpara}[1]{\emph{#1}}

\newcommand{\mytilde}{\raise.30ex\hbox{$\scriptstyle\mathtt{\sim}$}}

\newcommand{\more}[1]{{\small\color{gray}[[--#1--]]}}
\newcommand*\rot{\rotatebox{90}}

\newcommand{\WP}{Wikipedia\xspace}



\newcommand{\uptrend}{$\uparrow$}
\newcommand{\downtrend}{$\downarrow$}

\newcommand{\ie}{\textit{i.e.}\xspace}
\newcommand{\eg}{\textit{e.g.}\xspace}
\newcommand{\cf}{\textit{cf.}\xspace}
\newcommand{\etal}{\textit{et al.}\xspace}
\newcommand{\vs}{\textit{vs.}\xspace}
\newcommand{\etc}{\textit{etc.}\xspace}

% References for sections and tables:
\newcommand{\Secref}[1]{Sec.~\ref{#1}}
\newcommand{\secref}[1]{Sec.~\ref{#1}}
\newcommand{\eqnref}[1]{Eq.~\ref{#1}}
\newcommand{\Eqnref}[1]{Eq.~\ref{#1}}
\newcommand{\dashsecref}[2]{Sections~\ref{#1}--\ref{#2}}
\newcommand{\dblsecref}[2]{Sections~\ref{#1} and \ref{#2}}
\newcommand{\Tabref}[1]{Table~\ref{#1}}
\newcommand{\tabref}[1]{Table~\ref{#1}}
\newcommand{\Figref}[1]{Fig.~\ref{#1}}
\newcommand{\figref}[1]{Fig.~\ref{#1}}

\newcommand{\xhdrX}[1]{\vspace{0mm}{\bf #1.}}
\newcommand{\xhdr}[1]{\vspace{1.7mm}\noindent{{\bf #1.}}}
\newcommand{\xhdrNoPeriod}[1]{\vspace{0mm}{\bf #1}}

\newcommand{\denselist}{ \itemsep -3pt\topsep-10pt\partopsep-10pt }

\newcommand\blfootnote[1]{%
  \begingroup
  \renewcommand\thefootnote{}\footnote{#1}%
  \addtocounter{footnote}{-1}%
  \endgroup
}

\hyphenation{
Wi-ki-pe-dia
Wi-ki-pe-dias
Wi-ki-da-ta
Wi-ki-Proj-ect
Wi-ki-Proj-ects
Page-Rank
prog-ress
web-page
web-pages
where-as
name-space
}

%\begin{comment}
%\renewcommand{\phsi}[1]{}
%\end{comment}
main.tex
% THIS IS SIGPROC-SP.TEX - VERSION 3.1
% WORKS WITH V3.2SP OF ACM_PROC_ARTICLE-SP.CLS
% APRIL 2009
%
% It is an example file showing how to use the 'acm_proc_article-sp.cls' V3.2SP
% LaTeX2e document class file for Conference Proceedings submissions.
% ----------------------------------------------------------------------------------------------------------------
% This .tex file (and associated .cls V3.2SP) *DOES NOT* produce:
%       1) The Permission Statement
%       2) The Conference (location) Info information
%       3) The Copyright Line with ACM data
%       4) Page numbering
% ---------------------------------------------------------------------------------------------------------------
% It is an example which *does* use the .bib file (from which the .bbl file
% is produced).
% REMEMBER HOWEVER: After having produced the .bbl file,
% and prior to final submission,
% you need to 'insert'  your .bbl file into your source .tex file so as to provide
% ONE 'self-contained' source file.
%
% Questions regarding SIGS should be sent to
% Adrienne Griscti ---> griscti@acm.org
%
% Questions/suggestions regarding the guidelines, .tex and .cls files, etc. to
% Gerald Murray ---> murray@hq.acm.org
%
% For tracking purposes - this is V3.1SP - APRIL 2009

%\documentclass{acm_proc_article-sp}
%\documentclass[preprint]{acm_proc_article-sp} %Remove preprint for camera-ready 


%\documentclass{sig-alternate-05-2015}
\documentclass{sig-alternate-2013}
%\documentclass[format=sigconf]{acmart}
%\pagenumbering{arabic} % Remove for camera-ready
%\usepackage{color}% Remove for camera-ready
%\usepackage[usenames,dvipsnames]{xcolor}% Remove for camera-ready

%\usepackage{hyperref}
\usepackage[hyperfootnotes=false]{hyperref}
\usepackage{graphicx}
\usepackage{cleveref}
\usepackage{verbatim}
\usepackage{amsmath}
\usepackage{xfrac}
\usepackage{subfig}
\usepackage[font=bf,skip=\baselineskip]{caption}
%\usepackage{tikz}
%\usetikzlibrary{arrows,shapes,positioning,calc,shadows}
\usepackage{balance}
\usepackage{tabularx}
\usepackage{url}

\usepackage{xspace}

%\usepackage[bottom]{footmisc}

%\usepackage[show]{chato-notes} % set to ``hide'' for submission
\usepackage[hide]{chato-notes} % set to ``hide'' for submission

% Bob's package for nice Greek letters within a Times math environment.
\usepackage{cmm-greek}


\newenvironment{psmallmatrix}
  {\left(\begin{smallmatrix}}
  {\end{smallmatrix}\right)}

\usepackage{longtable}

\widowpenalty100000
\clubpenalty100000

\setlength{\pdfpagewidth}{8.5in}
\setlength{\pdfpageheight}{11in}


%\setlength{\abovedisplayskip}{0pt}
%\setlength{\belowdisplayskip}{0pt}
%\setlength{\abovedisplayshortskip}{0pt}
%\setlength{\belowdisplayshortskip}{0pt}

\usepackage{layouts}
\usepackage{textcomp}
\usepackage{dblfloatfix}
\usepackage{placeins}
\usepackage[numbers,sort]{natbib}
\DeclareRobustCommand{\VAN}[3]{#2}
\setlength{\bibsep}{2pt plus 0.1ex}

%\usepackage{mathptmx}
\usepackage{microtype}

%for rotated table lines:
\usepackage{adjustbox}
\usepackage{array}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage{tabularx}
%\usepackage{multicol}
\usepackage{enumitem}

%\usepackage{todonotes}

\usepackage{tikz}
\usetikzlibrary{shapes,arrows,calc,positioning}



%Silbentrennung
\usepackage{hyphenat}
%\hyphenation{}
\input{macros}

\newcommand{\Koblenz}{U. of Koblenz-Landau \xspace}
\newcommand{\GESIS}{GESIS - Leibniz Institute for the Social Sciences, Cologne \xspace}

% Copyright
%\setcopyright{none}
%\setcopyright{acmcopyright}
%\setcopyright{acmlicensed}
%\setcopyright{rightsretained}
%\setcopyright{usgov}
%\setcopyright{usgovmixed}
%\setcopyright{cagov}
%\setcopyright{cagovmixed}


% DOI
% \acmDOI{10.475/123_4}

% % ISBN
% \acmISBN{123-4567-24-567/08/06}

% %Conference
% \acmConference[WOODSTOCK'97]{ACM Woodstock conference}{July 1997}{El
%   Paso, Texas USA} 
% \acmYear{1997}
% \copyrightyear{2016}

% \acmPrice{15.00}



\begin{document}

%\title{Characterizing, Understanding and Modeling Reader Behavior on Wikipedia}

%\newcommand{\helv}[1]{{\fontfamily{helv}\selectfont #1}}

\title{Why We Read Wikipedia} %A Survey and Request-log based Study of Reader Motivations and Behavior}


\makeatletter
\def\@copyrightspace{\@float{copyrightbox}[b]
\begin{center}
\setlength{\unitlength}{1pc}
\begin{picture}(20,4.5) %Space for copyright notice
\put(0,-0.95){\crnotice{\@toappear}}
\end{picture}
\end{center}
\end@float}
\makeatother

%\permission{Copyright is held by the International World Wide Web Conference Committee (IW3C2). IW3C2 reserves the right to provide a hyperlink to the author's site if the Material is used in electronic media.}
%\conferenceinfo{WWW 2017,}{Perth.} 
%\copyrightetc{ACM \the\acmcopyr}
%\crdata{978-1-4503-3469-3/15/05. \\
%http://dx.doi.org/10.1145/2736277.2741080}

\permission{\copyright 2017 International World Wide Web Conference Committee \\ (IW3C2), published under Creative Commons CC BY 4.0 License.}
\conferenceinfo{WWW 2017,}{April 3--7, 2017, Perth, Australia.}
\copyrightetc{ACM \the\acmcopyr}
\crdata{978-1-4503-4913-0/17/04. \\
	http://dx.doi.org/10.1145/3038912.3052716 \\
	\includegraphics[scale=0.8]{cc.png}}

\author{
\alignauthor
Philipp Singer\textsuperscript{\textasteriskcentered 1},
Florian Lemmerich\textsuperscript{\textasteriskcentered 1},
Robert West\textsuperscript{\textdagger 2},\\
Leila Zia\textsuperscript{3},
Ellery Wulczyn\textsuperscript{3},
Markus Strohmaier\textsuperscript{1},
Jure Leskovec\textsuperscript{4}\\
\vspace{2mm}
\affaddr{\textsuperscript{1}GESIS \& University of Koblenz-Landau},
\affaddr{\textsuperscript{2}EPFL},
\affaddr{\textsuperscript{3}Wikimedia Foundation},
\affaddr{\textsuperscript{4}Stanford University}\\
%\vspace{1mm}
%\small
\textsuperscript{1}firstname.lastname@gesis.org,
\textsuperscript{2}robert.west@epfl.ch,
\textsuperscript{3}firstname@wikimedia.org,
\textsuperscript{4}jure@cs.stanford.edu\\
}

% \numberofauthors{7}
% \author{
%   \alignauthor Philipp Singer\thanks{Both authors contributed equally to this work.}\\
%      \affaddr{GESIS \& U. of Koblenz-Landau}\\
%      %\affaddr{Graz, Austria}\\
%      \email{philipp.singer@gesis.org}\\
%      \and
%  \and
%   \alignauthor Florian Lemmerich\footnotemark[1]\\%\samethanks \\
%      \affaddr{GESIS \& U. of Koblenz-Landau}\\
%      %\affaddr{Graz, Austria}\\x<
%      \email{florian.lemmerich@gesis.org}\\
%      \and
%   \alignauthor Robert West\\
%      \affaddr{EPFL}\\
%           \email{robert.west@epfl.ch}\\
%      \and
%      \alignauthor Leila Zia\\
%      \affaddr{Wikimedia Foundation}\\
%           \email{leila@wikimedia.org}\\
%      \and
%       \alignauthor Ellery Wulczyn\\
%      \affaddr{Wikimedia Foundation}\\
%           \email{ellery@wikimedia.org}\\
%      \and
%      \alignauthor Jure Leskovec\\
%      \affaddr{Stanford University}\\
%           \email{jure@cs.stanford.edu}\\
%      \and
% \and
%   \alignauthor Markus Strohmaier\\
%      \affaddr{GESIS \& U. of Koblenz-Landau}\\
%      %\affaddr{Graz, Austria}\\
%      \email{markus.strohmaier@gesis.org}\\
% }

% \author{Philipp Singer}
% \authornote{Equal contribution}
% %\authornote{Dr.~Trovato insisted his name be first.}
% %\orcid{1234-5678-9012}
% \affiliation{%
%   \institution{GESIS \& U. of Koblenz-Landau}
% %   \streetaddress{P.O. Box 1212}
% %   \city{Dublin} 
% %   \state{Ohio} 
% %   \postcode{43017-6221}
% }
% \email{philipp.singer@gesis.org}


%\textsuperscript{\textasteriskcentered}
% \author{Florian Lemmerich\footnotemark[1]}
% %\authornote{Equal contribution}
% %\authornote{The secretary disavows any knowledge of this author's actions.}
% \affiliation{%
%   \institution{GESIS \& U. of Koblenz-Landau}
% %   \streetaddress{P.O. Box 1212}
% %   \city{Dublin} 
% %   \state{Ohio} 
% %   \postcode{43017-6221}
% }
% \email{florian.lemmerich@gesis.org}

% \author{Robert West}
% %\authornote{This author is the
% %  one who did all the really hard work.}
% \affiliation{%
%   \institution{EPFL}
% %   \streetaddress{1 Th{\o}rv{\"a}ld Circle}
% %   \city{Hekla} 
% %   \country{Iceland}}
% }
% \email{robert.west@epfl.ch}

% \author{Leila Zia}
% \affiliation{
%   \institution{Wikimedia Foundation}
% %   \streetaddress{P.O. Box 5000}}
% }
% \email{leila@wikimedia.org}

% \author{Ellery Wulczyn}
% \affiliation{%
%   \institution{Wikimedia Foundation}
% %   \streetaddress{P.O. Box 5000}}
% }
% \email{ellery@wikimedia.org}

% \author{Leila Zia, Ellery Wulczyn}
% \affiliation{
%   \institution{Wikimedia Foundation}
% %   \streetaddress{P.O. Box 5000}}
% }
% \email{firstname@wikimedia.org}

% \author{Markus Strohmaier}
% \affiliation{%
%   \institution{GESIS \& U. of Koblenz-Landau}
% %   \streetaddress{P.O. Box 1212}
% %   \city{Dublin} 
% %   \state{Ohio} 
% %   \postcode{43017-6221}
% }
% \email{markus.strohmaier@gesis.org}


% \author{Jure Leskovec}
% \affiliation{%
%   \institution{Stanford University}
% %   \streetaddress{P.O. Box 1212}
% %   \city{Dublin} 
% %   \state{Ohio} 
% %   \postcode{43017-6221}
% }
% \email{jure@cs.stanford.edu}

% \renewcommand{\authors}{Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulczyn, Markus Strohmaier, and Jure Leskovec}
% \renewcommand{\shortauthors}{P. Singer et al.}

% \numberofauthors{6}
% \author{
% \alignauthor
% Philipp Singer*\\
%       \affaddr{GESIS}\\
%       \email{philipp.singer@gesis.org}
% \and
% \alignauthor
% Florian Lemmerich*\\
%       \affaddr{GESIS}\\
%       \email{florian.lemmerich@gesis.org}
% \and
% \alignauthor
% Robert West\\
%       \affaddr{EPFL}\\
%       \email{west@cs.stanford.edu}
% \and
% \alignauthor
% Leila Zia\\
%       \affaddr{Wikimedia Foundation}\\
%       \email{leila@wikimedia.org}
% \and
% \alignauthor
% Jure Leskovec\\
%       \affaddr{Stanford University}\\
%       \email{jure@cs.stanford.edu}
% \and
% \alignauthor
% Markus Strohmaier\\
%       \affaddr{GESIS}\\
%       \email{markus.strohmaier@gesis.org}
% }



% There's nothing stopping you putting the seventh, eighth, etc.
% author on the opening page (as the 'third row') but we ask,
% for aesthetic reasons that you place these 'additional authors'
% in the \additional authors block, viz.
%\additionalauthors{Additional authors: John Smith (The Th{\o}rv{\"a}ld Group,
%email: {\texttt{jsmith@affiliation.org}}) and Julius P.~Kumquat
%(The Kumquat Consortium, email: {\texttt{jpkumquat@consortium.net}}).}
%\date{}
% Just remember to make sure that the TOTAL number of authors
% is the number that will appear on the first page PLUS the
% number that will appear in the \additionalauthors section.

%\maketitle

%\note[Bob]{This is how you leave a note}

%\inote{This is how you leave an inline note}




\maketitle

\blfootnote{\textsuperscript{\textasteriskcentered}Both authors contributed equally to this work.}
\blfootnote{\textsuperscript{\textdagger}Robert West is a Wikimedia Foundation Research Fellow.}
\input{000abstract}

%\vspace{2mm}


\vspace{1mm}
\noindent
{\bf Keywords:} Wikipedia; survey; motivation; log analysis

%\noindent
%{\bf Keywords:} Behavioral Analysis; Sequences; Subgroups; Exceptional Transition Models; Descriptive Patterns; Web; Markov Chains
%\vfill\eject
%THE ACTUAL CONTENTS:
% \input{introduction}
% \input{data}
% \input{bias}
% \input{survey_results}
% \input{subgroups}
% \input{discussion}
% \input{relatedWork}
% \input{conclusion}
%\blfootnote{*Equal contribution.}
\input{010introduction}
\input{020taxonomy}
\input{030data+methodology}
\input{040results}
\input{050discussion+relwork}
\input{060conclusion}
\input{070appendix}
%
%ACKNOWLEDGMENTS are optional
%\section{Acknowledgments}
%

%\todo[Bob]{-- Unify spelling for `pageview', `dataset' (instead of `page view').

%-- `Web' \vs\ `web'

%-- Use ``'' rather than "" as quotation marks (DONE)}

%\todo[Florian]{Check capitalization of (sub-)section headers (DONE)}
%\todo[Florian]{Do not forget to remove pagenumbers and comments!}
%\todo[Florian]{Right now, the Figure 1 feels disconnected from the introduction. Either we take more time/space for a full explanation what we see, or (preferably) remove it altogether. (suggestion got outvoted)}


%\input{results_material}

%\newpage

%
% The following two commands are all you need in the
% initial runs of your .tex file to
% produce the bibliography for the citations in your paper.

\clearpage
\begingroup
%\vspace{-1em}
\small
\raggedright
\sloppy
\balance
%\bibliographystyle{ACM-Reference-Format
\bibliographystyle{abbrv}
% here we change the meaning of \VAN
\DeclareRobustCommand{\VAN}[3]{#3}
\bibliography{bib}  % sigproc.bib is the name of the Bibliography in this case
\endgroup
% You must have a proper ".bib" file
%  and remember to run:
% latex bibtex latex latex
% to resolve all references
%
%\vspace{-0.5em}
%\input{appendix}

% ACM needs 'a single self-contained file'!
%\balancecolumns
% That's all folks!


\end{document}

results_material.tex
\newpage
\section{Full Results, not to be included}




\begin{table*}[b!]
	\small
	\centering
	\caption{\emph{Subgroup Discovery Results for motivations.} Each of the tables shows the subgroups that have significantly different share of users with a certain motivation. 
	For each result subgroup S, we display the size of the subgroup $P(S)$, i.e., the share of users that are covered by the subgroup description,  the coverage of this motivation by the subgroup ($P(S|T)$), the target share in the subgroup (\emph{P(T|S)}), i.e., the share of users within the subgroup that reported the respective motivation.
	For comparison, it also shows the respective share of users with this motivation in the entire user population (\emph{$P(T|S)$}), which is constant for all subgroups.}
	\subfloat[motivation = work/school]{\input{tables/sg_all_motivationworkschool_table}\label{table:sg3}} 
	\subfloat[motivation = media]{\input{tables/sg_all_motivationmedia_table.tex}\label{table:sg5}} \\
	\subfloat[motivation = personal decision]{\input{tables/sg_all_motivationpersonaldecision_table.tex}\label{table:sg4}} 
	\subfloat[motivation = intrinsic learning]{\input{tables/sg_all_motivationintrinsiclearning_table.tex}\label{table:sg6}}
\end{table*}	
	
\begin{table*}[b!]
	\small
	\centering
	\caption{\emph{Subgroup Discovery Results for motivations. continued.} Each of the tables shows the subgroups that have significantly different share of users with a certain motivation. 
	For each result subgroup S, we display the size of the subgroup $P(S)$, i.e., the share of users that are covered by the subgroup description,  the coverage of this motivation by the subgroup ($P(S|T)$), the target share in the subgroup (\emph{P(T|S)}), i.e., the share of users within the subgroup that reported the respective motivation.
	For comparison, it also shows the respective share of users with this motivation in the entire user population (\emph{$P(T|S)$}), which is constant for all subgroups.}	
	\subfloat[motivation = current event]{\input{tables/sg_all_motivationcurrentevent_table.tex}\label{table:sg7}}
	\subfloat[motivation = bored/random]{\input{tables/sg_all_motivationboredrandom_table.tex}\label{table:sg9}} \\
	\subfloat[motivation = conversation]{\input{tables/sg_all_motivationconversation_table.tex}\label{table:sg8}}
	\label{table:sgs_motivation}
\end{table*}

\begin{table*}[b!]
	\small
	\centering
	\caption{\emph{Subgroup Discovery Results for information need.}
	Each of the tables shows the 5 subgroups that have the most significantly increased share of users with a certain information need. 
	For each finding, we display the size of the subgroup, i.e., the share of users that are covered by the subgroup description, and the target share in the subgroup (\emph{Target(SG)}), i.e., the share of users within the subgroup that reported the respective information need.
	For comparison, it also shows the respective share of users with this motivation in the entire user population (\emph{Target(data)}), which is constant for all subgroups.} 
	\subfloat[information need = in-depth]{\input{tables/sg_all_informationdepth=indepth_table.tex}\label{table:sg11}} 
	\subfloat[information need = overview]{\input{tables/sg_all_informationdepth=overview_table.tex}\label{table:sg12}} \\
	\subfloat[information need = fact]{\input{tables/sg_all_informationdepth=fact_table}\label{table:sg13}} 
\end{table*}

\begin{table*}[b!]
	\small
	\centering
	\caption{\emph{Subgroup Discovery Results for prior knowledge.}
	Each of the tables shows the 5 subgroups that have the most significantly increased share of users with a prior knowledge. 
	For each finding, we display the size of the subgroup, i.e., the share of users that are covered by the subgroup description, and the target share in the subgroup (\emph{Target(SG)}), i.e., the share of users within the subgroup that reported the respective prior knowledge.
	For comparison, it also shows the respective share of users with this motivation in the entire user population (\emph{Target(data)}), which is constant for all subgroups.}
	\subfloat[prior knowledge = familiar]{\input{tables/sg_all_priorknowledge=familiar_table.tex}\label{table:sg21}} 
	\subfloat[prior knowledge = unfamiliar]{\input{tables/sg_all_priorknowledge=unfamiliar_table.tex}\label{table:sg22}} \\
\end{table*}
sg_all_informationdepth=fact_table.tex
\begin{tabular}{lrrrrl}
\toprule
                                Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
                topic (tv, movies, novels) & 34.1\% &  36.8\% &  40.6\% &  1.08 &   ** \\
 \# referer ext. nonsearch (session): [0:1[ & 96.2\% &  97.4\% &  38.1\% &  1.01 &  *** \\
                       topic entropy>=2.69 & 20.0\% &  17.9\% &  33.7\% &  0.90 &    * \\
        topic (education, government, law) & 16.9\% &  14.6\% &  32.4\% &  0.86 &  *** \\
                            continent=Asia & 14.9\% &  11.9\% &  30.2\% &  0.80 &  *** \\
                           country code=IN & 13.5\% &  10.7\% &  29.8\% &  0.79 &  *** \\
                         referer: external &  3.2\% &   2.2\% &  25.8\% &  0.68 &   ** \\
     \# referer ext. nonsearch (session)>=1 &  3.8\% &   2.6\% &  25.2\% &  0.67 &  *** \\
\bottomrule
\end{tabular}

sg_all_informationdepth=indepth_table.tex
\begin{tabular}{lrrrrl}
\toprule
                 Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
            country code=IN & 13.5\% &  20.9\% &  33.1\% &  1.55 &  *** \\
             continent=Asia & 14.9\% &  22.5\% &  32.4\% &  1.51 &  *** \\
 topic (tv, movies, novels) & 34.1\% &  29.9\% &  18.8\% &  0.88 &   ** \\
            country code=GB & 11.8\% &   9.2\% &  16.8\% &  0.79 &    * \\
           continent=Europe & 16.1\% &  12.5\% &  16.6\% &  0.77 &  *** \\
\bottomrule
\end{tabular}

sg_all_informationdepth=overview_table.tex
\begin{tabular}{lrrrrl}
\toprule
                                Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
                         referer: external &  3.2\% &   4.3\% &  52.4\% &  1.36 &  *** \\
     \# referer ext. nonsearch (session)>=1 &  3.8\% &   5.1\% &  51.4\% &  1.33 &  *** \\
        topic (education, government, law) & 16.9\% &  19.3\% &  43.9\% &  1.14 &  *** \\
                              host=desktop & 35.5\% &  40.1\% &  43.5\% &  1.13 &  *** \\
 \# referer ext. nonsearch (session): [0:1[ & 96.2\% &  94.9\% &  38.1\% &  0.99 &  *** \\
                           host=mobile web & 64.5\% &  59.9\% &  35.9\% &  0.93 &  *** \\
                            continent=Asia & 14.9\% &  12.4\% &  32.1\% &  0.83 &  *** \\
                           country code=IN & 13.5\% &  11.2\% &  32.0\% &  0.83 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationboredrandom_table.tex
\begin{tabular}{lrrrrl}
\toprule
                                Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
                         referer: internal &  9.4\% &  14.0\% &  29.0\% &  1.49 &  *** \\
                 \# viewed pages in week>=8 & 11.8\% &  16.6\% &  27.5\% &  1.41 &  *** \\
                            topic (sports) &  5.9\% &   8.0\% &  26.1\% &  1.34 &   ** \\
             \# internal refers(session)>=1 & 17.1\% &  22.7\% &  25.9\% &  1.33 &  *** \\
             ses rel position: [0.33:0.75[ &  7.5\% &   9.8\% &  25.6\% &  1.31 &   ** \\
       avg. topic distance (session)>=1.08 &  7.5\% &   9.8\% &  25.2\% &  1.29 &    * \\
                      topic (21st century) & 25.1\% &  32.1\% &  25.0\% &  1.28 &  *** \\
              \# viewed pages in session>=3 & 22.2\% &  28.3\% &  24.8\% &  1.27 &  *** \\
 avg. time per page (session): [0.68:1.56[ &  7.7\% &   9.7\% &  24.7\% &  1.27 &    * \\
                      \# no referer(ses)>=2 &  9.7\% &  12.2\% &  24.5\% &  1.26 &    * \\
                topic (tv, movies, novels) & 34.1\% &  41.4\% &  23.7\% &  1.21 &  *** \\
                \# article pageviews>=63606 & 19.8\% &  23.5\% &  23.1\% &  1.19 &   ** \\
                   continent=North America & 63.4\% &  69.3\% &  21.3\% &  1.09 &  *** \\
                           country code=US & 57.1\% &  62.3\% &  21.3\% &  1.09 &  *** \\
         \# internal refers(session): [0:1[ & 82.9\% &  77.3\% &  18.2\% &  0.93 &  *** \\
          \# viewed pages in session: [1:2[ & 61.7\% &  57.0\% &  18.0\% &  0.92 &   ** \\
              session duration: [0.0:0.02[ & 62.0\% &  57.1\% &  18.0\% &  0.92 &  *** \\
         referer: external (search engine) & 62.6\% &  55.4\% &  17.3\% &  0.89 &  *** \\
                  article text length<6233 & 20.0\% &  16.7\% &  16.3\% &  0.84 &    * \\
                topic (biology, chemistry) &  8.6\% &   5.2\% &  11.8\% &  0.60 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationconversation_table.tex
\begin{tabular}{lrrrrl}
\toprule
                       Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
    \# viewed pages in week: [1:2[ & 48.3\% &  53.1\% &  24.4\% &  1.10 &  *** \\
 \# viewed pages in session: [1:2[ & 61.7\% &  66.7\% &  24.0\% &  1.08 &  *** \\
     session duration: [0.0:0.02[ & 62.0\% &  66.9\% &  23.9\% &  1.08 &  *** \\
                  host=mobile web & 64.5\% &  69.4\% &  23.9\% &  1.08 &  *** \\
                     host=desktop & 35.5\% &  30.6\% &  19.1\% &  0.86 &  *** \\
 \# external searches (session)>=2 & 20.5\% &  17.1\% &  18.6\% &  0.84 &   ** \\
     ses rel position: [0.0:0.33[ & 15.2\% &  12.5\% &  18.3\% &  0.82 &    * \\
     \# viewed pages in session>=3 & 22.2\% &  18.0\% &  17.9\% &  0.81 &  *** \\
        \# viewed pages in week>=8 & 11.8\% &   9.4\% &  17.7\% &  0.80 &    * \\
           session duration>=6.60 & 18.0\% &  13.7\% &  16.9\% &  0.76 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationcurrentevent_table.tex
\begin{tabular}{lrrrrl}
\toprule
                         Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
                  topic (east asia) &  1.8\% &   3.6\% &  25.7\% &  2.03 &  *** \\
                     topic (sports) &  5.9\% &  11.7\% &  25.0\% &  1.97 &  *** \\
         \# article pageviews>=63606 & 19.8\% &  33.3\% &  21.3\% &  1.68 &  *** \\
               topic (21st century) & 25.1\% &  37.4\% &  18.9\% &  1.49 &  *** \\
 topic (education, government, law) & 16.9\% &  25.3\% &  18.9\% &  1.49 &  *** \\
                     continent=Asia & 14.9\% &  21.3\% &  18.2\% &  1.44 &  *** \\
                    country code=IN & 13.5\% &  19.3\% &  18.1\% &  1.43 &  *** \\
             article outdegree>=390 & 20.0\% &  26.3\% &  16.6\% &  1.31 &  *** \\
         article text length>=44860 & 20.0\% &  24.9\% &  15.8\% &  1.24 &   ** \\
         topic entropy: [2.42:2.69[ & 20.0\% &  24.1\% &  15.3\% &  1.21 &    * \\
               article outdegree<50 & 19.8\% &  15.7\% &  10.0\% &  0.79 &    * \\
                 topic entropy<1.87 & 20.0\% &  15.6\% &   9.9\% &  0.78 &   ** \\
           \# article pageviews<2519 & 19.8\% &  14.4\% &   9.2\% &  0.73 &  *** \\
                 topic (technology) & 13.2\% &   8.9\% &   8.5\% &  0.67 &  *** \\
            topic (literature, art) & 10.1\% &   5.9\% &   7.4\% &  0.58 &  *** \\
         topic (biology, chemistry) &  8.6\% &   4.6\% &   6.8\% &  0.53 &  *** \\
                topic (mathematics) &  7.9\% &   3.9\% &   6.2\% &  0.49 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationintrinsiclearning_table.tex
\begin{tabular}{lrrrrl}
\toprule
                 Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
    topic (literature, art) & 10.1\% &  13.1\% &  32.8\% &  1.30 &  *** \\
        topic (mathematics) &  7.9\% &   9.8\% &  31.2\% &  1.24 &    * \\
         topic (technology) & 13.2\% &  16.0\% &  30.5\% &  1.21 &   ** \\
             continent=Asia & 14.9\% &  17.8\% &  30.1\% &  1.20 &   ** \\
            country code=IN & 13.5\% &  16.1\% &  30.0\% &  1.19 &   ** \\
       topic (21st century) & 25.1\% &  21.9\% &  22.0\% &  0.87 &   ** \\
           continent=Europe & 16.1\% &  13.8\% &  21.5\% &  0.85 &    * \\
 \# article pageviews>=63606 & 19.8\% &  16.6\% &  21.1\% &  0.84 &   ** \\
            country code=GB & 11.8\% &   8.6\% &  18.4\% &  0.73 &  *** \\
 topic (tv, movies, novels) & 34.1\% &  24.1\% &  17.8\% &  0.71 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationmedia_table.tex
\begin{tabular}{lrrrrl}
\toprule
                    Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
    topic (tv, movies, novels) & 34.1\% &  46.9\% &  43.4\% &  1.37 &  *** \\
          topic (21st century) & 25.1\% &  31.6\% &  39.9\% &  1.26 &  *** \\
    \# article pageviews>=63606 & 19.8\% &  23.1\% &  36.9\% &  1.17 &  *** \\
         article pagerank<0.50 & 20.0\% &  22.9\% &  36.1\% &  1.14 &  *** \\
      \# no referer(ses): [1:2[ & 21.5\% &  24.1\% &  35.5\% &  1.12 &   ** \\
                 referer: none & 24.8\% &  27.3\% &  34.8\% &  1.10 &    * \\
               host=mobile web & 64.5\% &  68.7\% &  33.7\% &  1.07 &  *** \\
      \# no referer(ses): [0:1[ & 68.8\% &  65.7\% &  30.2\% &  0.95 &   ** \\
                  host=desktop & 35.5\% &  31.3\% &  27.8\% &  0.88 &  *** \\
        article outdegree>=390 & 20.0\% &  17.2\% &  27.1\% &  0.86 &  *** \\
               country code=IN & 13.5\% &  11.4\% &  26.8\% &  0.85 &    * \\
                continent=Asia & 14.9\% &  12.5\% &  26.7\% &  0.84 &   ** \\
      \# article pageviews<2519 & 19.8\% &  16.5\% &  26.4\% &  0.83 &  *** \\
         article indegree>=731 & 19.9\% &  16.5\% &  26.2\% &  0.83 &  *** \\
        article pagerank>=9.98 & 20.0\% &  14.9\% &  23.6\% &  0.75 &  *** \\
         topic (United States) &  4.9\% &   3.3\% &  21.5\% &  0.68 &  *** \\
           topic (mathematics) &  7.9\% &   5.2\% &  20.8\% &  0.66 &  *** \\
 topic (geographical entities) &  5.4\% &   3.5\% &  20.5\% &  0.65 &  *** \\
    topic (biology, chemistry) &  8.6\% &   5.4\% &  19.8\% &  0.63 &  *** \\
            topic (technology) & 13.2\% &   8.1\% &  19.4\% &  0.61 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationother_table.tex
\begin{tabular}{lrrrrl}
\toprule
              Subgroup S &  P(S) &  P(S|T) &  P(T|S) &  P(T) & sig. \\
\midrule
 \# article pageviews<795 &  8.3\% &   12.7\% &   12.2\% &  7.9\% &    * \\
\bottomrule
\end{tabular}

sg_all_motivationpersonaldecision_table.tex
\begin{tabular}{lrrrrl}
\toprule
                    Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
 topic (geographical entities) &  5.4\% &   8.4\% &  14.8\% &  1.54 &   ** \\
               country code=IN & 13.5\% &  19.9\% &  14.3\% &  1.48 &  *** \\
                continent=Asia & 14.9\% &  21.8\% &  14.1\% &  1.46 &  *** \\
            topic (technology) & 13.2\% &  17.4\% &  12.7\% &  1.32 &    * \\
          article outdegree<50 & 19.8\% &  24.8\% &  12.1\% &  1.25 &    * \\
       continent=North America & 63.4\% &  57.5\% &   8.8\% &  0.91 &    * \\
                topic (sports) &  5.9\% &   2.4\% &   3.8\% &  0.40 &  *** \\
\bottomrule
\end{tabular}

sg_all_motivationworkschool_table.tex
\begin{tabular}{lrrrrl}
\toprule
                                Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
                       topic (mathematics) &  7.9\% &  17.1\% &  34.8\% &  2.17 &  *** \\
                      topic (war, history) &  4.4\% &   9.6\% &  34.7\% &  2.16 &  *** \\
                        topic (technology) & 13.2\% &  23.7\% &  28.8\% &  1.79 &  *** \\
                topic (biology, chemistry) &  8.6\% &  14.0\% &  26.2\% &  1.63 &  *** \\
                              host=desktop & 35.5\% &  57.8\% &  26.1\% &  1.63 &  *** \\
                    article pagerank>=9.98 & 20.0\% &  32.4\% &  26.1\% &  1.62 &  *** \\
        avg. time per page (session)>=9.40 &  7.7\% &  11.5\% &  24.0\% &  1.50 &  *** \\
         ses avg pagerank difference<-4.35 &  7.6\% &  11.2\% &  23.6\% &  1.47 &  *** \\
                   topic (literature, art) & 10.1\% &  14.7\% &  23.5\% &  1.46 &  *** \\
 avg. time per page (session): [3.60:9.40[ &  7.7\% &  11.0\% &  23.1\% &  1.44 &  *** \\
          \# external searches (session)>=2 & 20.5\% &  28.5\% &  22.4\% &  1.39 &  *** \\
                    session duration>=6.60 & 18.0\% &  24.2\% &  21.6\% &  1.34 &  *** \\
                        topic entropy<1.87 & 20.0\% &  25.7\% &  20.6\% &  1.28 &  *** \\
                     article indegree>=731 & 19.9\% &  25.3\% &  20.4\% &  1.27 &  *** \\
      \# article pageviews: [2519.0:8740.0[ & 19.8\% &  24.2\% &  19.6\% &  1.22 &  *** \\
         referer: external (search engine) & 62.6\% &  75.1\% &  19.3\% &  1.20 &  *** \\
        topic (education, government, law) & 16.9\% &  20.3\% &  19.2\% &  1.20 &    * \\
                    article outdegree>=390 & 20.0\% &  24.0\% &  19.2\% &  1.20 &   ** \\
     \# article pageviews: [8740.0:22514.0[ & 19.8\% &  23.4\% &  19.0\% &  1.18 &    * \\
             article pagerank: [3.22:9.98[ & 20.0\% &  23.5\% &  18.9\% &  1.18 &    * \\
                  \# no referer(ses): [0:1[ & 68.8\% &  79.0\% &  18.4\% &  1.15 &  *** \\
 \# referer ext. nonsearch (session): [0:1[ & 96.2\% &  98.0\% &  16.4\% &  1.02 &   ** \\
                   continent=North America & 63.4\% &  58.6\% &  14.9\% &  0.92 &   ** \\
                           country code=US & 57.1\% &  52.0\% &  14.6\% &  0.91 &   ** \\
            article indegree: [36.0:120.0[ & 19.9\% &  16.3\% &  13.1\% &  0.82 &    * \\
                topic entropy: [1.87:2.17[ & 20.0\% &  15.4\% &  12.4\% &  0.77 &  *** \\
                \# article pageviews>=63606 & 19.8\% &  14.1\% &  11.4\% &  0.71 &  *** \\
             article pagerank: [0.50:1.30[ & 20.0\% &  14.1\% &  11.3\% &  0.70 &  *** \\
                      \# no referer(ses)>=2 &  9.7\% &   6.5\% &  10.9\% &  0.68 &   ** \\
                  \# no referer(ses): [1:2[ & 21.5\% &  14.4\% &  10.8\% &  0.67 &  *** \\
                           host=mobile web & 64.5\% &  42.2\% &  10.5\% &  0.65 &  *** \\
                     article pagerank<0.50 & 20.0\% &  13.0\% &  10.4\% &  0.65 &  *** \\
                             referer: none & 24.8\% &  15.2\% &   9.9\% &  0.61 &  *** \\
      \# external searches (session): [0:1[ & 27.4\% &  16.1\% &   9.4\% &  0.59 &  *** \\
     \# referer ext. nonsearch (session)>=1 &  3.8\% &   2.0\% &   8.3\% &  0.52 &   ** \\
                topic (tv, movies, novels) & 34.1\% &  15.7\% &   7.4\% &  0.46 &  *** \\
                         referer: external &  3.2\% &   1.4\% &   7.1\% &  0.44 &   ** \\
                      topic (21st century) & 25.1\% &  10.5\% &   6.7\% &  0.42 &  *** \\
                             topic (music) &  9.3\% &   3.5\% &   6.0\% &  0.37 &  *** \\
                            topic (sports) &  5.9\% &   2.2\% &   5.8\% &  0.36 &  *** \\
\bottomrule
\end{tabular}

sg_all_priorknowledge=familiar_table.tex
\begin{tabular}{lrrrrl}
\toprule
                                 Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
                             topic (sports) &  5.9\% &   7.2\% &  60.8\% &  1.21 &  *** \\
 avg. topic distance (session): [0.24:0.47[ &  7.6\% &   8.6\% &  56.8\% &  1.13 &    * \\
                     article outdegree>=390 & 20.0\% &  22.2\% &  55.6\% &  1.11 &  *** \\
                 \# article pageviews>=63606 & 19.8\% &  21.9\% &  55.5\% &  1.11 &  *** \\
                 article text length>=44860 & 20.0\% &  22.0\% &  55.1\% &  1.10 &  *** \\
                      article indegree>=731 & 19.9\% &  21.6\% &  54.4\% &  1.09 &   ** \\
                     article pagerank>=9.98 & 20.0\% &  21.6\% &  54.3\% &  1.08 &    * \\
                       topic (21st century) & 25.1\% &  27.0\% &  54.0\% &  1.08 &   ** \\
                 topic (tv, movies, novels) & 34.1\% &  36.7\% &  53.8\% &  1.07 &  *** \\
                    continent=North America & 63.4\% &  65.6\% &  51.9\% &  1.04 &   ** \\
                            country code=US & 57.1\% &  59.1\% &  51.9\% &  1.03 &    * \\
                        article indegree<36 & 19.7\% &  17.9\% &  45.4\% &  0.91 &   ** \\
                   article text length<6233 & 20.0\% &  18.0\% &  45.1\% &  0.90 &  *** \\
                       article outdegree<50 & 19.8\% &  17.5\% &  44.3\% &  0.88 &  *** \\
                        topic entropy>=2.69 & 20.0\% &  17.6\% &  44.2\% &  0.88 &  *** \\
\bottomrule
\end{tabular}

sg_all_priorknowledge=unfamiliar_table.tex
\begin{tabular}{lrrrrl}
\toprule
                 Subgroup S &  P(S) & P(S|T) & P(T|S) &  lift & sig. \\
\midrule
        topic entropy>=2.69 & 20.0\% &  22.5\% &  53.2\% &  1.12 &  *** \\
       article outdegree<50 & 19.8\% &  22.2\% &  53.1\% &  1.12 &  *** \\
        article indegree<36 & 19.7\% &  21.7\% &  52.1\% &  1.10 &   ** \\
   article text length<6233 & 20.0\% &  21.8\% &  51.5\% &  1.09 &    * \\
       topic (21st century) & 25.1\% &  23.3\% &  43.9\% &  0.93 &    * \\
 topic (tv, movies, novels) & 34.1\% &  31.6\% &  43.8\% &  0.93 &  *** \\
     article pagerank>=9.98 & 20.0\% &  18.3\% &  43.4\% &  0.92 &    * \\
      article indegree>=731 & 19.9\% &  18.1\% &  43.0\% &  0.91 &   ** \\
 article text length>=44860 & 20.0\% &  18.1\% &  42.7\% &  0.90 &   ** \\
 \# article pageviews>=63606 & 19.8\% &  17.8\% &  42.6\% &  0.90 &   ** \\
     article outdegree>=390 & 20.0\% &  17.9\% &  42.2\% &  0.89 &  *** \\
             topic (sports) &  5.9\% &   4.6\% &  36.8\% &  0.78 &  *** \\
\bottomrule
\end{tabular}

_sg_results.tex
\begin{table*}[t!]
	\small
	\centering
%\vspace{-1em}
\caption{\emph{Top subgroups for the motivations ``work/school'' and ``bored/random''.} Each  table shows the top subgroups with significantly different shares of users with a certain motivation $T$. 
For each subgroup $S$, we display
the relative size $P(S)$ of the subgroup (\ie, the share of users covered by the subgroup description),
the share $P(S|T)$ of the subgroup among those with motivation $T$,
the target share $P(T|S)$ in the subgroup,
%\ie, the share of users within the subgroup that reported the respective motivation,
and the \emph{lift} measure, defined as $P(T|S)/P(T) = P(S|T)/P(S)$.
Rows are ranked by lift.
The last column indicates significance (***~$<$~0.001, **~$<$~0.01, *~$<$~0.05) for the hypothesis test of independence between subgroup and target motivation using a $\chi^2$ test with effective sample size and Bonferroni correction.}
\vspace{-1em}
	\fontsize{6.5}{\baselineskip}\selectfont
\subfloat[$T$: ``motivation = work/school''; $P(T) = 16.1\%$]{
\begin{tabularx}{0.5\textwidth}{l|XXXX|l} 
\toprule
                                Subgroup $S$ &  $P(S)$ & $P(S|T)$ & $P(T|S)$ &  lift & sig. \\
\midrule
topic (mathematics) &  7.9\% &  17.1\% &  34.8\% &  2.17 &  *** \\
topic (war, history) &  4.4\% &   9.6\% &  34.7\% &  2.16 &  *** \\
topic (technology) & 13.2\% &  23.7\% &  28.8\% &  1.79 &  *** \\
topic (biology, chemistry) &  8.6\% &  14.0\% &  26.2\% &  1.63 &  *** \\
host = desktop & 35.5\% &  57.8\% &  26.1\% &  1.63 &  *** \\
article pagerank $\geq$ 9.98 & 20.0\% &  32.4\% &  26.1\% &  1.62 &  *** \\
avg. time difference $\geq$ 9.40 &  7.7\% &  11.5\% &  24.0\% &  1.50 &  *** \\
avg. pagerank difference < -4.35 &  7.6\% &  11.2\% &  23.6\% &  1.47 &  *** \\
topic (literature, art) & 10.1\% &  14.7\% &  23.5\% &  1.46 &  *** \\
avg. time difference: [3.60:9.40[ &  7.7\% &  11.0\% &  23.1\% &  1.44 &  *** \\
num. (referer=search) $\geq$ 2 & 20.5\% &  28.5\% &  22.4\% &  1.39 &  *** \\
session duration $\geq$ 6.60 & 18.0\% &  24.2\% &  21.6\% &  1.34 &  *** \\
\bottomrule
\end{tabularx}
}
\subfloat[$T$: ``motivation = bored/random''; $P(T) = 19.5\%$] {
	%	\fontsize{6.5}{\baselineskip}\selectfont
\begin{tabularx}{0.5\textwidth}{l|XXXX|l} 
\toprule
Subgroup $S$ &  $P(S)$ & $P(S|T)$ & $P(T|S)$ &  lift & sig. \\
\midrule
referer class: internal &  9.4\% &  14.0\% &  29.0\% &  1.49 &  *** \\
num. of requests $\geq$ 8 & 11.8\% &  16.6\% &  27.5\% &  1.41 &  *** \\
topic (sports) &  5.9\% &   8.0\% &  26.1\% &  1.34 &   ** \\
num. (referer=internal) $\geq$ 1 & 17.1\% &  22.7\% &  25.9\% &  1.33 &  *** \\
session position: [0.33:0.75[ &  7.5\% &   9.8\% &  25.6\% &  1.31 &   ** \\
avg. topic distance  $\geq$ 1.08 &  7.5\% &   9.8\% &  25.2\% &  1.29 &    * \\
topic (21st century) & 25.1\% &  32.1\% &  25.0\% &  1.28 &  *** \\
session length $\geq$ 3 & 22.2\% &  28.3\% &  24.8\% &  1.27 &  *** \\
avg. time difference: [0.68:1.56[ &  7.7\% &   9.7\% &  24.7\% &  1.27 &    * \\
num. (referer=none) $\geq$ 2 &  9.7\% &  12.2\% &  24.5\% &  1.26 &    * \\
topic (tv, movies, novels) & 34.1\% &  41.4\% &  23.7\% &  1.21 &  *** \\
\# article pageviews $\geq$ 63606 & 19.8\% &  23.5\% &  23.1\% &  1.19 &   ** \\
\bottomrule
\end{tabularx}
}
\label{tab:sgResults}
\vspace{-2em}
\end{table*}
survey_crosstable_depth_prior.tex
\begin{tabular}{lrrr}
\toprule
prior knowledge &  familiar &  no response &  unfamiliar \\
information depth &           &              &             \\
\midrule
fact              &    0.5729 &       0.0098 &      0.4173 \\
in-depth          &    0.5765 &       0.0120 &      0.4115 \\
no response       &    0.1544 &       0.7046 &      0.1410 \\
overview          &    0.4183 &       0.0070 &      0.5747 \\
\bottomrule
\end{tabular}

survey_crosstable_motivation_depth.tex
\begin{tabularx}{0.45\textwidth}{l|XXX|l} 
\toprule 
information depth & fact & in-depth & overview & sig. \\ 
motivation &  &  &  &  \\ 
\midrule 
bored/random & 0.3077 & 0.2146 & 0.4653 & *** \\ 
conversation & 0.4153 & 0.1966 & 0.3776 & *** \\ 
current event & 0.3711 & 0.2672 & 0.349 & *** \\ 
intrinsic learning & 0.2918 & 0.3396 & 0.3565 & *** \\ 
media & 0.3682 & 0.1893 & 0.4358 & *** \\ 
other & 0.4001 & 0.2022 & 0.3279 & *** \\ 
personal decision & 0.341 & 0.2783 & 0.3646 & *** \\ 
work/school & 0.3854 & 0.2459 & 0.3543 & *** \\ 
\hline 
response perc. & 0.3664 & 0.2205 & 0.389 & \\ 
\bottomrule 
\end{tabularx} 

survey_crosstable_motivation_need.tex
\begin{tabularx}{0.49\textwidth}{l|XXX|l} 
\toprule 
information need & fact & in-depth & overview & sig. \\ 
motivation &  &  &  &  \\ 
\midrule 
media & 0.38 (1.00) & 0.19 (0.87) & 0.43 (1.12) & *** \\ 
intrinsic learning & 0.29 (0.76) & 0.35 (1.62) & 0.35 (0.92) & *** \\ 
conversation & 0.43 (1.13) & 0.20 (0.93) & 0.36 (0.94) & *** \\ 
bored/random & 0.31 (0.83) & 0.23 (1.05) & 0.45 (1.17) & *** \\ 
work/school & 0.39 (1.04) & 0.23 (1.09) & 0.36 (0.93) &  \\ 
current event & 0.36 (0.95) & 0.28 (1.30) & 0.35 (0.92) & *** \\ 
personal decision & 0.32 (0.85) & 0.29 (1.35) & 0.38 (0.97) & *** \\ 
\hline 
response perc. & 0.38 & 0.21 & 0.39 & \\ 
\bottomrule 
\end{tabularx} 

survey_crosstable_motivation_prior.tex
\begin{tabularx}{0.45\textwidth}{l|XX|l} 
\toprule 
prior knowledge & familiar & unfamiliar & sig. \\ 
motivation &  &  &  \\ 
\midrule 
media & 0.42 (0.83) & 0.58 (1.22) & *** \\ 
intrinsic learning & 0.57 (1.14) & 0.41 (0.87) & *** \\ 
conversation & 0.49 (0.98) & 0.49 (1.04) & *** \\ 
bored/random & 0.53 (1.07) & 0.45 (0.95) &  \\ 
work/school & 0.52 (1.04) & 0.46 (0.97) &  \\ 
current event & 0.52 (1.03) & 0.46 (0.98) &  \\ 
personal decision & 0.50 (0.99) & 0.48 (1.02) &  \\ 
\hline 
response perc. & 0.50 & 0.47 & \\ 
\bottomrule 
\end{tabularx} 

survey_crosstable_prior_depth.tex
\begin{tabularx}{0.45\textwidth}{l|XXX|l} 
\toprule 
information depth & fact & in-depth & overview & sig. \\ 
prior knowledge &  &  &  &  \\ 
\midrule 
familiar & 0.4174 & 0.2522 & 0.323 & *** \\ 
unfamiliar & 0.3249 & 0.1929 & 0.4757 & *** \\ 
\hline 
response perc. & 0.3664 & 0.2205 & 0.389 & \\ 
\bottomrule 
\end{tabularx} 

survey_crosstable_prior_need.tex
\begin{tabularx}{0.45\textwidth}{l|XXX|l} 
\toprule 
information need & fact & in-depth & overview & sig. \\ 
prior knowledge &  &  &  &  \\ 
\midrule 
familiar & 0.43 (1.13) & 0.25 (1.15) & 0.32 (0.83) & *** \\ 
unfamiliar & 0.34 (0.90) & 0.19 (0.87) & 0.47 (1.22) & *** \\ 
\hline 
response perc. & 0.38 & 0.21 & 0.39 & \\ 
\bottomrule 
\end{tabularx} 

main.bbl
\begin{thebibliography}{10}

\bibitem{arazy}
O.~Arazy, H.~Lifshitz-Assaf, O.~Nov, J.~Daxenberger, M.~Balestra, and
  C.~Cheshire.
\newblock On the ``how'' and ``why'' of emergent role behaviors in {W}ikipedia.
\newblock In {\em Conference on Computer-Supported Cooperative Work and Social
  Computing}, 2017.

\bibitem{austin2011introduction}
P.~C. Austin.
\newblock An introduction to propensity score methods for reducing the effects
  of confounding in observational studies.
\newblock {\em Multivariate Behavioral Research}, 46(3):399--424, 2011.

\bibitem{basu2003context}
A.~Basu.
\newblock Context-driven assessment of commercial web sites.
\newblock In {\em International Conference On System Sciences}, 2003.

\bibitem{blei2003latent}
D.~M. Blei, A.~Y. Ng, and M.~I. Jordan.
\newblock Latent {D}irichlet allocation.
\newblock {\em Journal of Machine Learning Research}, 3(Jan):993--1022, 2003.

\bibitem{brick2013unit}
J.~M. Brick.
\newblock Unit nonresponse and weighting adjustments: A critical review.
\newblock {\em Journal of Official Statistics}, 29(3):329--353, 2013.

\bibitem{broder2002taxonomy}
A.~Broder.
\newblock A taxonomy of web search.
\newblock In {\em ACM SIGIR Forum}, 2002.

\bibitem{demaio1984social}
T.~J. DeMaio.
\newblock Social desirability and survey.
\newblock {\em Surveying Subjective Phenomena}, 2:257, 1984.

\bibitem{feild2010predicting}
H.~A. Feild, J.~Allan, and R.~Jones.
\newblock Predicting searcher frustration.
\newblock In {\em International Conference on Research and Development in
  Information Retrieval}, 2010.

\bibitem{gelman2000poststratification}
A.~Gelman and J.~B. Carlin.
\newblock Poststratification and weighting adjustments.
\newblock In {\em CiteSeerX}, 2000.

\bibitem{geng2006interestingness}
L.~Geng and H.~J. Hamilton.
\newblock Interestingness measures for data mining: A survey.
\newblock {\em ACM Computing Surveys}, 38(3):9, 2006.

\bibitem{goel2012does}
S.~Goel, J.~M. Hofman, and M.~I. Sirer.
\newblock Who does what on the {W}eb: A large-scale study of browsing behavior.
\newblock In {\em International Conference on Web and Social Media}, 2012.

\bibitem{halfaker2015user}
A.~Halfaker, O.~Keyes, D.~Kluver, J.~Thebault-Spieker, T.~Nguyen, K.~Shores,
  A.~Uduwage, and M.~Warncke-Wang.
\newblock User session identification based on strong regularities in
  inter-activity time.
\newblock In {\em International Conference on World Wide Web}, 2015.

\bibitem{harkness2003cross}
J.~A. Harkness, F.~J. Van~de Vijver, P.~P. Mohler, et~al.
\newblock {\em Cross-cultural survey methods}.
\newblock Wiley-Interscience Hoboken, 2003.

\bibitem{herrera2010}
F.~Herrera, C.~J. Carmona, P.~Gonz{\'a}lez, and M.~J. Del~Jesus.
\newblock An overview on subgroup discovery: Foundations and applications.
\newblock {\em Knowledge and Information Systems}, 29(3):495--525, 2010.

\bibitem{java2007we}
A.~Java, X.~Song, T.~Finin, and B.~Tseng.
\newblock Why we twitter: Understanding microblogging usage and communities.
\newblock In {\em Workshop on Web Mining and Social Network Analysis}, 2007.

\bibitem{jurgens2012temporal}
D.~Jurgens and T.-C. Lu.
\newblock Temporal motifs reveal the dynamics of editor interactions in
  {W}ikipedia.
\newblock In {\em International Conference on Web and Social Media}, 2012.

\bibitem{kish1965survey}
L.~Kish.
\newblock {\em Survey sampling}.
\newblock John Wiley and Sons, 1965.

\bibitem{kittur2008harnessing}
A.~Kittur and R.~E. Kraut.
\newblock Harnessing the wisdom of crowds in {W}ikipedia: Quality through
  coordination.
\newblock In {\em Conference on Computer Supported Cooperative Work}, 2008.

\bibitem{kloessgen1996}
W.~Kl\"{o}sgen.
\newblock Explora: A multipattern and multistrategy discovery assistant.
\newblock In {\em Advances in Knowledge Discovery and Data Mining}, pages
  249--271. American Association for Artificial Intelligence, 1996.

\bibitem{krug2014make}
S.~Krug.
\newblock {\em Don't Make Me Think, Revisited: A Common Sense Approach to Web
  Usability}.
\newblock New Riders, 2014.

\bibitem{kumar2010characterization}
R.~Kumar and A.~Tomkins.
\newblock A characterization of online browsing behavior.
\newblock In {\em International Conference on World Wide Web}, 2010.

\bibitem{kwak2010twitter}
H.~Kwak, C.~Lee, H.~Park, and S.~Moon.
\newblock What is {T}witter, a social network or a news media?
\newblock In {\em International Conference on World Wide Web}, 2010.

\bibitem{lamprecht2016evaluating}
D.~Lamprecht, D.~Dimitrov, D.~Helic, and M.~Strohmaier.
\newblock Evaluating and improving navigability of {W}ikipedia: A comparative
  study of eight language editions.
\newblock In {\em International Symposium on Open Collaboration}, 2016.

\bibitem{lee2010improving}
B.~K. Lee, J.~Lessler, and E.~A. Stuart.
\newblock Improving propensity score weighting using machine learning.
\newblock {\em Statistics in Medicine}, 29(3):337--346, 2010.

\bibitem{lee2011weight}
B.~K. Lee, J.~Lessler, and E.~A. Stuart.
\newblock Weight trimming and propensity score weighting.
\newblock {\em PloS One}, 6(3):e18174, 2011.

\bibitem{lehmann2014reader}
J.~Lehmann, C.~M{\"u}ller-Birn, D.~Laniado, M.~Lalmas, and A.~Kaltenbrunner.
\newblock Reader preferences and behavior on {W}ikipedia.
\newblock In {\em Conference on Hypertext and Social Media}, 2014.

\bibitem{lunceford2004stratification}
J.~K. Lunceford and M.~Davidian.
\newblock Stratification and weighting via the propensity score in estimation
  of causal treatment effects: A comparative study.
\newblock {\em Statistics in Medicine}, 23(19):2937--2960, 2004.

\bibitem{mukhopadhyay2016complex}
P.~Mukhopadhyay.
\newblock {\em Complex Surveys: Analysis of Categorical Data}.
\newblock Springer, 2016.

\bibitem{nov2007motivates}
O.~Nov.
\newblock What motivates {W}ikipedians?
\newblock {\em Communications of the ACM}, 50(11):60--64, 2007.

\bibitem{okoli2012people}
C.~Okoli, M.~Mehdi, M.~Mesgari, F.~{\AA}. Nielsen, and A.~Lanam{\"a}ki.
\newblock The people's encyclopedia under the gaze of the sages: A systematic
  review of scholarly research on {W}ikipedia.
\newblock {\em SSRN 2021326}, 2012.

\bibitem{paranjape2016improving}
A.~Paranjape, R.~West, L.~Zia, and J.~Leskovec.
\newblock Improving website hyperlink structure using server logs.
\newblock In {\em International Conference on Web Search and Data Mining},
  2016.

\bibitem{ratkiewicz2010characterizing}
J.~Ratkiewicz, S.~Fortunato, A.~Flammini, F.~Menczer, and A.~Vespignani.
\newblock Characterizing and modeling the dynamics of online popularity.
\newblock {\em Physical Review Letters}, 105(15):158701, 2010.

\bibitem{rose2004understanding}
D.~E. Rose and D.~Levinson.
\newblock Understanding user goals in web search.
\newblock In {\em International Conference on World Wide Web}, 2004.

\bibitem{ryan2011uses}
T.~Ryan and S.~Xenos.
\newblock Who uses {F}acebook? {A}n investigation into the relationship between
  the {Big Five}, shyness, narcissism, loneliness, and {F}acebook usage.
\newblock {\em Computers in Human Behavior}, 27(5):1658--1664, 2011.

\bibitem{salganik2017bit}
M.~J. Salganik.
\newblock {\em Bit by Bit: Social Research in the Digital Age}.
\newblock Princeton University Press, 2017.

\bibitem{singer2014detecting}
P.~Singer, D.~Helic, B.~Taraghi, and M.~Strohmaier.
\newblock Detecting memory and structure in human navigation patterns using
  {M}arkov chain models of varying order.
\newblock {\em PloS One}, 9(7):e102070, 2014.

\bibitem{spoerri2007popular}
A.~Spoerri.
\newblock What is popular on {W}ikipedia and why?
\newblock {\em First Monday}, 12(4), 2007.

\bibitem{strauss1998basics}
A.~Strauss and J.~Corbin.
\newblock {\em Basics of Qualitative Research: Techniques and Procedures for
  Developing Grounded Theory}.
\newblock Sage Publications, 1998.

\bibitem{stvilia2008information}
B.~Stvilia, M.~B. Twidale, L.~C. Smith, and L.~Gasser.
\newblock Information quality work organization in {W}ikipedia.
\newblock {\em Journal of the American Society for Information Science and
  Technology}, 59(6):983--1001, 2008.

\bibitem{waller2011search}
V.~Waller.
\newblock The search queries that took {A}ustralian {I}nternet users to
  {W}ikipedia.
\newblock {\em Information Research}, 16(2), 2011.

\bibitem{weber2011uses}
I.~Weber and A.~Jaimes.
\newblock Who uses web search for what: and how.
\newblock In {\em International Conference on Web Search and Data Mining},
  2011.

\bibitem{west2012human}
R.~West and J.~Leskovec.
\newblock Human wayfinding in information networks.
\newblock In {\em International Conference on World Wide Web}, 2012.

\bibitem{white2009characterizing}
R.~W. White and S.~T. Dumais.
\newblock Characterizing and predicting search engine switching behavior.
\newblock In {\em Conference on Information and Knowledge Management}, 2009.

\end{thebibliography}
