Wikiwhere: An Interactive Tool for Studying the Geographical Provenance of Wikipedia References
Martin K\"{o
abstract.tex
% run as pdflatex --shell-escape main.tex for eps-to-pdf problems
\documentclass[a4paper]{llncs}
\usepackage{etex}
\usepackage{amsmath, verbatim, amssymb, graphics, tabularx,  amsfonts, fixltx2e,
  color, rotate, listings, float, theorem, curves, cite, latexsym,  hyperref, fancyvrb, alltt, todonotes}
\usepackage{subcaption}
\usepackage{url, breakurl, paralist}
\usepackage{graphicx, multirow, multicol, framed, longtable, array} 
\usepackage[outdir=./]{epstopdf}
\usepackage[ruled, lined, boxed, longend, linesnumbered]{algorithm2e}
\usepackage[multiple]{footmisc}
\usepackage[htt]{hyphenat}
\newcommand{\dcat}{{\scshape dcat}\xspace}
\newcommand\footnoteurl[1]{\footnote{\scriptsize\url{#1}}}
\newcommand{\prov}{{\scshape prov-o}\xspace}
\newcommand{\ckan}{{\scshape ckan}\xspace}
\newcommand{\prop}[1]{{{\texttt{#1}}}}
\newcommand{\adms}{{\scshape adms}\xspace}
\usepackage{booktabs}
\lstset{basicstyle=\footnotesize\ttfamily,breaklines=true,frame=single}
%\usepackage{geometry}
%\geometry{a4paper, margin=1.5in}
\begin{document}
\mainmatter
\title{Wikiwhere: An Interactive Tool for Studying the Geographical Provenance of Wikipedia References} 


\author{Martin K\"{o}rner\inst{1} \and
  Tatiana Sennikova\inst{1} \and Florian Windh\"{a}user \inst{1} \and Claudia Wagner\inst{1,2} \and Fabian Fl\"{o}ck \inst{2}}
  \authorrunning{Martin K\"{o}rner et al.}
 \institute{University of Koblenz-Landau, Germany \and GESIS - Leibniz Institute for the Social Sciences, Germany
%
}
\maketitle

%\todo{max. 1000 words, 2 pages}

\input{intro.tex}

\bibliographystyle{abbrv}
\bibliography{bibliography}
\end{document}

intro.tex
\begin{comment} %▼▼▼▼ everything below is commented! ▼▼▼▼
  \abstract
Wikipedia articles about the same topic in different language editions are built around different sources of information. For example, one can find very different news articles linked as references in the English Wikipedia article titled ``Annexation of Crimea by the Russian Federation'' than in its German counterpart (determined via Wikipedia's language links). Some of this difference can of course be attributed to the different language proficiencies of readers and editors in separate language editions; yet, although including English-language news sources seems to be no issue in the German edition, English references that are listed do not overlap highly with the ones in the article's English version. 
Such patterns could be an indicator of bias towards certain national contexts when referencing facts and statements in Wikipedia. However, determining for each reference which national context it can be traced back to, and comparing the link distributions to each other is infeasible for  casual readers or scientists with non-technical backgrounds.
Wikiwhere} answers the question where Web references stem from by analyzing and visualizing the geographic location of external reference links that are included in a given Wikipedia article. Instead of relying solely on the IP location of a given URL our machine learning models consider several features.

\end{comment} %▲▲▲▲ end of commented section!▲▲▲▲




\section{Introduction and problem statement}
\label{sec:introduction}
Wikipedia articles about the same topic in different language editions are built around different sources of information. For example, one can find very different news articles linked as references in the English Wikipedia article titled ``Annexation of Crimea by the Russian Federation'' than in its German counterpart (determined via Wikipedia's language links). Some of this difference can of course be attributed to the different language proficiencies of readers and editors in separate language editions; yet, although including English-language news sources seems to be no issue in the German edition, English references that are listed do not overlap highly with the ones in the article's English version. Remarkably, the German version, compared to its English counterpart, includes a notably higher imbalance in favor of Russian sources against Ukrainian ones, and also a lesser overall ratio of Ukrainian and Russian sources in relation to the native language of the Wikipedia edition  (cf. Figure \ref{fig:screenshots}) %\todo{either adapt the screenshot to the text (EN-DE) or vice versa (DE-RU). Whatever makes a better story. -FF -- I like the EN-DE story, updated the images -MK})  
 -- although many of these pages are written in English and can be easily included in the German article.
Such patterns could be an indicator of bias towards certain national contexts when referencing facts and statements in Wikipedia. However, determining for each reference which national context it can be traced back to, 
%-- resolving IPs, cross-checking the content etc.-- 
and comparing the link distributions to each other is infeasible for  casual readers or scientists with non-technical backgrounds.
%, although it would enable worthwhile data exploration. 

\begin{figure}[b!]
\makebox[\textwidth][c]{
\begin{subfigure}[b]{0.5\textwidth}
  \includegraphics[width=\textwidth]{img/heat-crimea-en}
    \caption{English version}
    %\label{fig:f1}
  \end{subfigure}
  \hfill
  \begin{subfigure}[b]{0.5\textwidth}
    \includegraphics[width=\textwidth]{img/heat-crimea-de}
    \caption{German version}
    %\label{fig:f2}
  \end{subfigure}
}
\caption{Comparison of the Wikiwhere heat maps of the (a) English and (b) German versions of the Wikipedia article on the "Annexation of Crimea by the Russian Federation" (English title). Blue represents least links from a country, red most, grey none. Open \url{https://goo.gl/YgJx6O} and \url{https://goo.gl/pV0Mqp} for comparison.}
\label{fig:screenshots}
\end{figure}


\textit{Wikiwhere} answers the question where Web references stem from by analyzing and visualizing the geographic location of external reference links that are included in a given Wikipedia article. Instead of relying solely on the IP location of a given URL
%(a common approach), 
our machine learning models consider several features.
%additionally consider the top level domain and the website language.

Closely related is the work by Sen et al. \cite{sen_barriers_2015} that investigates, among other aspects, the geo-provenance of URLs in Wikipedia articles describing geographic locations. A related visualization is also available.\footnote{\url{http://shilad.com/localness/index.html}}

\vspace{-0.1cm}

\section{Interface and usage}
Wikiwhere is available at \url{http://wikiwhere.west.uni-koblenz.de}.\footnote{\url{http://wikiwhere.west.uni-koblenz.de/about.php} provides additional up-to-date information.} Given a valid Wikipedia article URL of any language edition, the tool returns a classification of all references on the requested article page into countries of origin, determined by our machine learning model. The results are displayed topmost on a heat map (cf. Figure \ref{fig:screenshots}) and further down in a  bar chart. Additional bar charts show the distribution of links over countries when using only single features of our model (e.g. just IP address) and finally, all references and their location result are individually listed. By opening several language equivalents of an article, the user can thus easily compare different editions.


% If the option "Fresh crawl" is not selected and the article was analysed before, the previous results are displayed. Otherwise, a new analysis gets executed on the server. Currently we allow up to ten parallel analyses. 
%Since we extract the content of all linked websites in the given article, the analysis can take several minutes, depending on the number of external links in the article. The plotted results are shown on a separate  web page.


It is also possible to access the plotted results via URL parameters, and preprocessed analyses can be accessed via the "Articles" tab on the website. The source code is available under a free license from GitHub (\url{https://github.com/mkrnr/wikiwhere}) and can also be easily employed to classify references for research purposes beyond our visualization use case, such as statistical analyses.

\vspace{-0.2cm}

\section{Determining a reference's geographical provenance}
\vspace{-0.2cm}
We use the term \textit{reference} to refer to an URL that leads from a given Wikipedia article to another web page that is not associated with the Wikimedia foundation's  projects. 
Currently available online services aiming to determine where web sources hail from geographically often %(e.g. Event Registry \cite{leban2014event} and X\todo{add the two services we wanted to use initially, references for research?}) 
rely solely on IP-derived locations. But given that websites and documents are frequently hosted under arbitrary domains, in many different languages on remote servers, this might yield inaccurate results.  

%. We hence evaluated if IP addresses are sufficient to pinpoint the provenance location of web resources. 

To investigate this suspicion, we set up a machine learning model to infer geo-provenance.
To obtain a training set for the model, we retrieved geo-location information on Wikipedia-referenced websites from DBpedia SPARQL endpoints (\url{http://wiki.dbpedia.org/about/language-chapters}). DBpedia contains structured information that allows to link the owner of a web address to a location, or -- if such information is not explicitly encoded -- inference about the owning entity and possible parent entities (e.g., a URL of a reference belongs to a newspaper, which has no location associated, but is associated with a parent company that has location information).
In order to evaluate the accuracy of this location extraction method, we manually checked 255 locations for references that we extracted from the English DBpedia, using an explicit coding scheme. The resulting accuracy was 95\%; we thus used this data 
%the DBpedia location extraction method 
as our ground truth for the subsequent steps. 
Next, we randomly extracted references from Wikipedia articles and obtained their DBpedia geo-location.
For this list, which comprised a total of 233,932 URLs, we automatically retrieved the IP-location, top level domain (plus location), and website language. % and used these features in combination with the DBpedia geo-location as the target variable to train our machine learning model.
\begin{comment}


% be found on the wikiwhere website \footnote{\url{http://wikiwhere.west.uni-koblenz.de/about.php}}.


%The following subsections provide some more details on the individual steps.

%\subsection{Data Collection}
%In the prediction model we used three features: The IP location of the URL, top level domain (TLD) location of the URL and language of its content. The target variable is the country that we retrieved from the DBpedia geo-location that was linked to the given website.
%We built separate prediction models for the following languages: English, German, French, Italian, Spanish, Ukrainian, Slovak, and Dutch. These are the languages for which there currently exist DBpedia knowledge bases. We also built a general model that combines the data from all DBpedia knowledge bases. The Python modules for the data collection can be found in the feature\_extraction and location\_extraction packages in our wikiwhere Github repository.
%\subsection{Feature Extraction}
%In the following we briefly describe the feature extraction process. More information can be found on the wikiwhere website \footnote{\url{http://wikiwhere.west.uni-koblenz.de/about.php}}.

%\textbf{IP address:}
%Our current implementation uses the geoip2\todo{link} Python library in combination with the GeoLite2 data created by MaxMind, available from \url{http://www.maxmind.com}. This allows us to locally calculate IP-locations on a country level. A previous attempt of using online APIs for retrieving IP-locations such as the Google Maps API\todo{link} failed due to their limitations in daily requests.
%One potential source of wrong IP-locations could result from websites that use content delivery networks such as Akamai since their servers are globally distributed. Thereby, the localization of the content server can be misleading. One goal of the future work could be a better handling of such cases.
%There exist several public APIs that return the geo-coordinates for a given IP. Our first implementation relied on the Google Maps API. Yet, due to the high number of requests that we run for the analysis of larger Wikipedia articles, we quickly reached the daily limits. 

%\textbf{Top Level Domain Extraction:}
%The extraction of top level domains (TLDs) was done based on two datasets. Data from IANA\todo{link} describes whether a TLD is generic or related to a country. If a TLD belongs to a country, we use the CIA world factbook\todo{link} to retrieve the correct ISO country code.
%As a first step we created 2 datasets by parsing HTML tables. The first, taken from the IANA website, contains information on whether a TLD is generic or a country-code. The second, from the CIA, gives us the top-level domains corresponding to different country codes. With help from the Python package tld we extracted the TLD from a URL. If it's a TLD with a "." inside, for example "co.uk", we only considered the part after the final "." for further analysis. We then used our first dataset to find out whether the TLD is a country-code. If that's the case we took the corresponding ISO 2 character code from the second dataset. If it's not a country-code we set the TLD as the parameter, for example "COM". Errors in this process can be a "bad URL" or no found TLD within the usage of the TLD package or cases unknown to the IANA dataset. Empty values can be a result of a TLD which is a country-code but not corresponding to any specific country, for example ".eu".

%\textbf{Website Language Extraction:}
%In order to determine the language of a given Website, we first request the website content. After extracting the textual content from the HTML document, we use the langdetect\footnote{\url{https://code.google.com/archive/p/language-detection/}} package for detecting the language. On their website they report an precision of over 99\% for the 53 supported languages. 

%\subsection{DBpedia Location Extraction}
%In order to gather a large amount of geo-locations for websites, we used the DBpedia SPARQL endpoints. The SPARQL requests were made with the SPARQLWrapper package. The first part of getting a location is to associate a given URL to a DBpedia entity. For this entity we then query for it's location, location city, or the location of it's parent company. The SPARQL query that we used can be found here. It is possible to copy this query into the field in the English DBpedia SPARQL endpoint. In cases where we retrieve more than one location for a URL we perform a majority voting. For example, the SPARQL query returns four locations for the URL http://www.treasury.gov.au/. The geo-coordinates for all four locations only differ in the second digit after the comma. With our current threshold, geo-coordinates are considered the same if they differ by less than 0.1. Another example is http://www.bangladesh.gov.bd/maps/images/pabna/Chatmohar.gif for which the geo-coordinates two of the four locations differ by 0.2. In this case a majority voting takes place. Since there are two different locations which both appear two times, one of them is selected at random. For the English DBpedia, a majority voting was necessary for 5179 out of the 162827 URLs that we extracted from the SPARQL endpoint. In addition, we do not consider URLs that contain "web.archive.org" and "webcitation.org" since they usually reference to another website and the DBpedia location also refers to the referenced website.
%Since the quality of the geo-location that we extract from DBpedia is crucial for the performance of the machine learning and thereby the predictions of our service, we performed a manual evaluation. In the following we use the term entity to refer to the subject the website belongs to, for example companies, schools, or the government. The location result (LR) is the geo-coordinate acquired from DBpedia queries. For the manual evaluation we used the following rules:

%If the entity is surely based and active (for companies, etc.) within only one country, we evaluated every LR within the same country as correct, anything else as wrong.
%If the entity is active internationally and has publicly available information about the location of its headquarter we evaluated every LR within the country of the headquarter as correct, anything else as wron
%If the entity is active internationally and has no publicly available information about the location of its headquarter we evaluated every LR that can be related to an office as true.
%If website was not reachable (offline, etc.) we set not found as our result which was handled during evaluating by removing these cases from our statistics.
%The evaluation showed 95\% accuracy of the ground truth. The web site was not reachable in 8 out of 255 cases.

%\subsection{Learning model}
\end{comment} 
 On this data, we applied a variety of statistical learning models% including logistic regression, random forests, and support vector machines (SVMs)
. An SVM model with a one vs. one multiclass classifier consistently provided the most accurate location prediction and was selected as the eventual approach.
We trained separate prediction models for the following languages: English, German, French, Italian, Spanish, Ukrainian, Slovak, and Dutch, as for those language editions  DBpedia knowledge bases do currently exist. We also built a general model that combines the data from all DBpedia knowledge bases.
 %We  the models separately for each of our Wikipedia language editions. We also trained a general prediction model based on merged data from all DBpedia knowledge bases. We use this model as a model for all the languages.
The performance of our model
 was evaluated via 10-cross fold validation. 
% Due to the poor data quality for some of the DBpedia language editions we decided to include one exception in our final classification. If the classification from our machine learning model predicts a country that appears in none of the three features we instead use the IP-location as the classification.

%Table 1 shows the accuracy of the models over two data sets.  First we checked the accuracy over all the data we have, including cases where all the parameters of the model agree. It is represented in the entry ``All data: Model''. Then we checked how well the models can handle difficult cases, i.e., such references, where TLD, IP and language parameters would all indicate a different location. This is represented in the entry ``Difficult cases: Model''. As the baseline we used the IP location.\todo{rewrite}

Table \ref{tab:accuracy} compares the accuracy of our learned model with a baseline that relies exclusively  on  IP address location. The comparison was performed on two data sets. The first includes % all types of references, including cases where all three features of the model agree. It is denoted with  
``All data'', i.e., all references and their location indicated by one of the features. The second only includes references for which all three features indicate different locations and thus represents ``Difficult cases''.
As becomes apparent from Table 1, (i) using only IP location decreases location determination accuracy by 20\% in the general model (10\% to 45\% in the language-specific models) Table 2 moreover shows how much the different features contribute to the models; together, these are strong indicators that research and services should not rely on IP addresses as a sole gauge for location. (ii) This holds even more true when features differ, which is often the case nowadays when websites are hosted abroad or address an international audience in, e.g., English.


\begin{comment}

\begin{table}[b!]
\caption{Accuracy of the learned models in comparison to a classification based on only the IP address.}
\makebox[\textwidth][c]{

  \includegraphics[width=0.8\textwidth]{img/1.svg}
 
}

\label{tab:accuracy}
\end{table}

\vspace{-0.2cm}

\end{comment}

\section{Conclusion}

The main contributions of this work are: 1. An interactive tool for visual analysis of the geographical provenance of references in a Wikipedia article with tested accuracy, including source code for free reuse, and 2. the insight that IP-location-based tracking is insufficient for determining (geographical) provenance of reference documents (in Wikipedia). Further, the approach of using semantic knowledge bases as a ground truth seems to be promising for tracking other kinds of provenance of references, e.g., multi-national corporations and media networks by following links and ownership-relations between businesses. 

%\vspace{1.5cm}

%\input{table1}
\setlength{\tabcolsep}{2.75pt}
\begin{longtable}[hb!]{@{}lcccccccccc@{}}
\caption{Accuracy of the learned models in comparison to a classification based on only the IP address.}\tabularnewline
\toprule
Method & General & EN & FR & DE & ES & UK & IT & NL & SV &
CS\tabularnewline
\midrule
\endfirsthead
\toprule
Method & General & EN & FR & DE & ES & UK & IT & NL & SV &
CS\tabularnewline
\midrule
\endhead
All data: Model & 0.81 & 0.81 & 0.91 & 0.90 & 0.75 & 0.96 & 0.91 & 0.96 & 0.92 & 0.98\tabularnewline
All data: IP only & 0.61 & 0.30 & 0.62 & 0.77 & 0.29 & 0.86 & 0.73 & 0.86 & 0.81 & 0.80\tabularnewline
Difficult cases: Model & 0.77 & 0.78 & 0.86 & 0.80 & 0.71 & 0.89 & 0.85 & 0.91 & 0.85 & 0.93\tabularnewline
Difficult cases: IP only  & 0.30 & 0.57 & 0.64 & 0.25 & 0.81 & 0.66 & 0.80 & 0.74 & 0.79 & 0.53\tabularnewline

\bottomrule
\label{tab:accuracy}
\end{longtable}
\setlength{\tabcolsep}{6pt}



\setlength{\tabcolsep}{4.75pt}
\begin{longtable}[b]{@{}lccc@{}}
\caption{Feature contribution over all data}\tabularnewline
\toprule
Model   & IP location & TLD location & Website Language

\tabularnewline
\midrule
\endhead
General & 61\%        & 58\%         & 25\%             \tabularnewline
EN      & 30\%        & 13\%         & 2\%              \tabularnewline
FR      & 62\%        & 73\%         & 23\%             \tabularnewline
DE      & 77\%        & 68\%         & 42\%             \tabularnewline
ES      & 29\%        & 30\%         & 7\%              \tabularnewline
UK      & 86\%        & 89\%         & 29\%             \tabularnewline
IT      & 73\%        & 70\%         & 27\%             \tabularnewline
NL      & 86\%        & 76\%         & 47\%             \tabularnewline
SV      & 81\%        & 82\%         & 29\%             \tabularnewline
CS      & 80\%        & 78\%         & 34\%  \tabularnewline
          
\bottomrule
\end{longtable}

table1.tex
\setlength{\tabcolsep}{2.75pt}
\begin{longtable}[b]{@{}lcccccccccc@{}}
\caption{Accuracy of the learned models in comparison to a classification based on only the IP address.}\tabularnewline
\toprule
Method & General & EN & FR & DE & ES & UK & IT & NL & SV &
CS\tabularnewline
\midrule
\endfirsthead
\toprule
Method & General & EN & FR & DE & ES & UK & IT & NL & SV &
CS\tabularnewline
\midrule
\endhead
All data: Model & 0.81 & 0.81 & 0.91 & 0.90 & 0.75 & 0.96 & 0.91 & 0.96 & 0.92 & 0.98\tabularnewline
All data: IP only & 0.61 & 0.30 & 0.62 & 0.77 & 0.29 & 0.86 & 0.73 & 0.86 & 0.81 & 0.80\tabularnewline
Difficult cases: Model & 0.77 & 0.78 & 0.86 & 0.80 & 0.71 & 0.89 & 0.85 & 0.91 & 0.85 & 0.93\tabularnewline
Difficult cases: IP only  & 0.30 & 0.57 & 0.64 & 0.25 & 0.81 & 0.66 & 0.80 & 0.74 & 0.79 & 0.53\tabularnewline

\bottomrule
\label{tab:accuracy}
\end{longtable}
\setlength{\tabcolsep}{6pt}

abstract.bbl
\begin{thebibliography}{1}

\bibitem{sen_barriers_2015}
S.~W. Sen, H.~Ford, D.~R. Musicant, M.~Graham, O.~S. Keyes, and B.~Hecht.
\newblock Barriers to the localness of volunteered geographic information.
\newblock In {\em Proceedings of the 33rd Annual ACM Conference on Human
  Factors in Computing Systems}, CHI '15, pages 197--206, New York, NY, USA,
  2015. ACM.

\end{thebibliography}
