Introduction to Information Retrieval (Draft)
Christopher D. Manning
Prabhakar Raghavan
Hinrich Schütze
An Introduction to Information Retrieval Draft of April 1, 2009 Online edition (c) 2009 Cambridge UP Online edition
Pages 581 Page size 595 x 842 pts (A4) Year 2009
Introduction to Information Retrieval
P1: KRU/IRP irbook CUUS232/Manning 978 0 521 86571 5 May 27, 2008 is the first textbook with a coherent treatment o
This page intentionally left blank P1: KRU/IRP irbook CUUS232/Manning 978 0 521 86571 5 June 26, 2008 is the firs
An Introduction To Information Retrieval
Draft of February 17, 2007 Preliminary draft (c) 2007 Cambridge UP Preliminary draft (c) 2007 Cambridge UP Chri
Information Retrieval Technology
An Introduction To Measure Theory (January 2011 Draft)
An introduction to measure theory Terence Tao Department of Mathematics, UCLA, Los Angeles, CA 90095 E-mail address: tao
Introduction to Information Systems, 15th Edition
This page intentionally left blank obr76779_fm_i-xxxiii.indd Page i 9/25/09 8:45:02 PM s-206 INTRODUCTION TO INFORMAT
Rough Draft
- 1 By Chris Owen and Jodi Payne Chapter 1 Sent in a plain envelope and written on lined paper. Black ink and quickly
Calculus in 3D (Draft)
Calculus in 3D Geometry, Vectors, and Multivariate Calculus Zbigniew H. Nitecki Tufts University September 1, 2010 ii
Information Architecture: The Design and Integration of Information Spaces (Synthesis Lectures on Information Concepts, Retrieval, and Services)
Information: A Very Short Introduction
VERY SHORT INTRODUCTIONS are for anyone wanting a stimulating and accessible way in to a new subject. They are written b
Draft of April 1, 2009
Online edition (c) 2009 Cambridge UP
Christopher D. Manning Prabhakar Raghavan Hinrich Schütze
Cambridge University Press Cambridge, England
DRAFT! DO NOT DISTRIBUTE WITHOUT PRIOR PERMISSION
© 2009 Cambridge University Press By Christopher D. Manning, Prabhakar Raghavan & Hinrich Schütze Printed on April 1, 2009
Website: http://www.informationretrieval.org/ Comments, corrections, and other feedback most welcome at:
DRAFT! © April 1, 2009 Cambridge University Press. Feedback welcome.
1 Boolean retrieval The term vocabulary and postings lists 19 Dictionaries and tolerant retrieval 49 Index construction 67 Index compression 85 Scoring, term weighting and the vector space model 109 Computing scores in a complete search system 135 Evaluation in information retrieval 151 Relevance feedback and query expansion 177 XML retrieval 195 Probabilistic information retrieval 219 Language models for information retrieval 237 Text classification and Naive Bayes 253 Vector space classification 289 Support vector machines and machine learning on documents Flat clustering 349 Hierarchical clustering 377 Matrix decompositions and latent semantic indexing 403 Web search basics 421 Web crawling and indexes 443 Link analysis 461
List of Tables List of Figures
xv xix
Table of Notation Preface
1 Boolean retrieval 1.1 1.2 1.3 1.4 1.5
An example information retrieval problem 3 A first take at building an inverted index 6 Processing Boolean queries 10 The extended Boolean model versus ranked retrieval References and further reading 17
2 The term vocabulary and postings lists 2.1 2.2
Document delineation and character sequence decoding 2.1.1 Obtaining the character sequence in a document 2.1.2 Choosing a document unit 20 Determining the vocabulary of terms 22 2.2.1 Tokenization 22 2.2.2 Dropping common terms: stop words 27 2.2.3 Normalization (equivalence classing of terms) 2.2.4 Stemming and lemmatization 32 Faster postings list intersection via skip pointers 36 Positional postings and phrase queries 39 2.4.1 Biword indexes 39 2.4.2 Positional indexes 41 2.4.3 Combination schemes 43 References and further reading 45
49 3 Dictionaries and tolerant retrieval 3.1 Search structures for dictionaries 49 3.2 Wildcard queries 51 3.2.1 General wildcard queries 53 3.2.2 k-gram indexes for wildcard queries 54 3.3 Spelling correction 56 3.3.1 Implementing spelling correction 57 3.3.2 Forms of spelling correction 57 3.3.3 Edit distance 58 3.3.4 k-gram indexes for spelling correction 60 3.3.5 Context sensitive spelling correction 62 3.4 Phonetic correction 63 3.5 References and further reading 65 4 Index construction 67 4.1 Hardware basics 68 4.2 Blocked sort-based indexing 69 4.3 Single-pass in-memory indexing 73 4.4 Distributed indexing 74 4.5 Dynamic indexing 78 4.6 Other types of indexes 80 4.7 References and further reading 83 5 Index compression 85 5.1 Statistical properties of terms in information retrieval 5.1.1 Heaps' law: Estimating the number of terms 5.1.2 Zipf's law: Modeling the distribution of terms 5.2 Dictionary compression 90 5.2.1 Dictionary as a string 91 5.2.2 Blocked storage 92 5.3 Postings file compression 95 5.3.1 Variable byte codes 96 5.3.2 γ codes 98 5.4 References and further reading 105 6 Scoring, term weighting and the vector space model 6.1 Parametric and zone indexes 110 6.1.1 Weighted zone scoring 112 6.1.2 Learning weights 113 6.1.3 The optimal weight g 115 6.2 Term frequency and weighting 117 6.2.1 Inverse document frequency 117 6.2.2 Tf-idf weighting 118
120 The vector space model for scoring 6.3.1 Dot products 120 6.3.2 Queries as vectors 123 6.3.3 Computing vector scores 124 Variant tf-idf functions 126 6.4.1 Sublinear tf scaling 126 6.4.2 Maximum tf normalization 127 6.4.3 Document and query weighting schemes 128 6.4.4 Pivoted normalized document length 129 References and further reading 133
7 Computing scores in a complete search system 7.1
Efficient scoring and ranking 135 7.1.1 Inexact top K document retrieval 137 7.1.2 Index elimination 137 7.1.3 Champion lists 138 7.1.4 Static quality scores and ordering 138 7.1.5 Impact ordering 140 7.1.6 Cluster pruning 141 Components of an information retrieval system 143 7.2.1 Tiered indexes 143 7.2.2 Query-term proximity 144 7.2.3 Designing parsing and scoring functions 145 7.2.4 Putting it all together 146 Vector space scoring and query operator interaction 147 References and further reading 149
8 Evaluation in information retrieval 8.1 8.2 8.3 8.4 8.5 8.6
Information retrieval system evaluation 152 Standard test collections 153 Evaluation of unranked retrieval sets 154 Evaluation of ranked retrieval results 158 Assessing relevance 164 8.5.1 Critiques and justifications of the concept of relevance 166 A broader perspective: System quality and user utility 8.6.1 System issues 168 8.6.2 User utility 169 8.6.3 Refining a deployed system 170 Results snippets 170 References and further reading 173
9 Relevance feedback and query expansion
178 Relevance feedback and pseudo relevance feedback 9.1.1 The Rocchio algorithm for relevance feedback 178 9.1.2 Probabilistic relevance feedback 183 9.1.3 When does relevance feedback work? 183 9.1.4 Relevance feedback on the web 185 9.1.5 Evaluation of relevance feedback strategies 186 9.1.6 Pseudo relevance feedback 187 9.1.7 Indirect relevance feedback 187 9.1.8 Summary 188 Global methods for query reformulation 189 9.2.1 Vocabulary tools for query reformulation 189 9.2.2 Query expansion 189 9.2.3 Automatic thesaurus generation 192 References and further reading 193
10 XML retrieval 195 10.1 Basic XML concepts 197 10.2 Challenges in XML retrieval 201 10.3 A vector space model for XML retrieval 206 10.4 Evaluation of XML retrieval 210 10.5 Text-centric vs. data-centric XML retrieval 214 10.6 References and further reading 216 10.7 Exercises 217 11 Probabilistic information retrieval 219 11.1 Review of basic probability theory 220 11.2 The Probability Ranking Principle 221 11.2.1 The 1/0 loss case 221 11.2.2 The PRP with retrieval costs 222 11.3 The Binary Independence Model 222 11.3.1 Deriving a ranking function for query terms 224 11.3.2 Probability estimates in theory 226 11.3.3 Probability estimates in practice 227 11.3.4 Probabilistic approaches to relevance feedback 228 11.4 An appraisal and some extensions 230 11.4.1 An appraisal of probabilistic models 230 11.4.2 Tree-structured dependencies between terms 231 11.4.3 Okapi BM25: a non-binary model 232 11.4.4 Bayesian network approaches to IR 234 11.5 References and further reading 235 12 Language models for information retrieval 12.1 Language models 237
237 12.1.1 Finite automata and language models 12.1.2 Types of language models 240 12.1.3 Multinomial distributions over words 241 The query likelihood model 242 12.2.1 Using query likelihood language models in IR 242 12.2.2 Estimating the query generation probability 243 12.2.3 Ponte and Croft's Experiments 246 Language modeling versus other approaches in IR 248 Extended language modeling approaches 250 References and further reading 252
13 Text classification and Naive Bayes 253 13.1 The text classification problem 256 13.2 Naive Bayes text classification 258 13.2.1 Relation to multinomial unigram language model 13.3 The Bernoulli model 263 13.4 Properties of Naive Bayes 265 13.4.1 A variant of the multinomial model 270 13.5 Feature selection 271 13.5.1 Mutual information 272 13.5.2 χ2 Feature selection 275 13.5.3 Frequency-based feature selection 277 13.5.4 Feature selection for multiple classifiers 278 13.5.5 Comparison of feature selection methods 278 13.6 Evaluation of text classification 279 13.7 References and further reading 286
14 Vector space classification 289 14.1 Document representations and measures of relatedness in vector spaces 291 14.2 Rocchio classification 292 14.3 k nearest neighbor 297 14.3.1 Time complexity and optimality of kNN 299 14.4 Linear versus nonlinear classifiers 301 14.5 Classification with more than two classes 306 14.6 The bias-variance tradeoff 308 14.7 References and further reading 314 14.8 Exercises 315 15 Support vector machines and machine learning on documents 319 15.1 Support vector machines: The linearly separable case 320 15.2 Extensions to the SVM model 327 15.2.1 Soft margin classification 327
330 15.2.2 Multiclass SVMs 15.2.3 Nonlinear SVMs 330 15.2.4 Experimental results 333 15.3 Issues in the classification of text documents 334 15.3.1 Choosing what kind of classifier to use 335 15.3.2 Improving classifier performance 337 15.4 Machine learning methods in ad hoc information retrieval 341 15.4.1 A simple example of machine-learned scoring 341 15.4.2 Result ranking by machine learning 344 15.5 References and further reading 346 16 Flat clustering 349 16.1 Clustering in information retrieval 350 16.2 Problem statement 354 16.2.1 Cardinality – the number of clusters 355 16.3 Evaluation of clustering 356 16.4 K-means 360 16.4.1 Cluster cardinality in K-means 365 16.5 Model-based clustering 368 16.6 References and further reading 372 16.7 Exercises 374 17 Hierarchical clustering 377 17.1 Hierarchical agglomerative clustering 378 17.2 Single-link and complete-link clustering 382 17.2.1 Time complexity of HAC 385 17.3 Group-average agglomerative clustering 388 17.4 Centroid clustering 391 17.5 Optimality of HAC 393 17.6 Divisive clustering 395 17.7 Cluster labeling 396 17.8 Implementation notes 398 17.9 References and further reading 399 17.10 Exercises 401 18 Matrix decompositions and latent semantic indexing 18.1 Linear algebra review 403 18.1.1 Matrix decompositions 406 18.2 Term-document matrices and singular value decompositions 407 18.3 Low-rank approximations 410 18.4 Latent semantic indexing 412 18.5 References and further reading 417
421 19 Web search basics 19.1 Background and history 421 19.2 Web characteristics 423 19.2.1 The web graph 425 19.2.2 Spam 427 19.3 Advertising as the economic model 429 19.4 The search user experience 432 19.4.1 User query needs 432 19.5 Index size and estimation 433 19.6 Near-duplicates and shingling 437 19.7 References and further reading 441 20 Web crawling and indexes 443 20.1 Overview 443 20.1.1 Features a crawler must provide 20.1.2 Features a crawler should provide 20.2 Crawling 444 20.2.1 Crawler architecture 445 20.2.2 DNS resolution 449 20.2.3 The URL frontier 451 20.3 Distributing indexes 454 20.4 Connectivity servers 455 20.5 References and further reading 458
21 Link analysis 461 21.1 The Web as a graph 462 21.1.1 Anchor text and the web graph 462 21.2 PageRank 464 21.2.1 Markov chains 465 21.2.2 The PageRank computation 468 21.2.3 Topic-specific PageRank 471 21.3 Hubs and Authorities 474 21.3.1 Choosing the subset of the Web 477 21.4 References and further reading 480 Bibliography
Typical system parameters in 2007. The seek time is the time needed to position the disk head in a new position. The transfer time per byte is the rate of transfer from disk to memory when the head is in the right position. Collection statistics for Reuters-RCV1. Values are rounded for the computations in this book. The unrounded values are: 806,791 documents, 222 tokens per document, 391,523 (distinct) terms, 6.04 bytes per token with spaces and punctuation, 4.5 bytes per token without spaces and punctuation, 7.5 bytes per term, and 96,969,056 tokens. The numbers in this table correspond to the third line ("case folding") in Table 5.1 (page 87). The five steps in constructing an index for Reuters-RCV1 in blocked sort-based indexing. Line numbers refer to Figure 4.2. Collection statistics for a large collection. The effect of preprocessing on the number of terms, nonpositional postings, and tokens for Reuters-RCV1. "∆%" indicates the reduction in size from the previous line, except that "30 stop words" and "150 stop words" both use "case folding" as their reference line. "T%" is the cumulative ("total") reduction from unfiltered. We performed stemming with the Porter stemmer (Chapter 2, page 33). Dictionary compression for Reuters-RCV1. Encoding gaps instead of document IDs. For example, we store gaps 107, 5, 43, . . . , instead of docIDs 283154, 283159, 283202, . . . for computer. The first docID is left unchanged (only shown for arachnocentric). VB encoding.
Some examples of unary and γ codes. Unary codes are only shown for the smaller numbers. Commas in γ codes are for readability only and are not part of the actual codes. Index and dictionary compression for Reuters-RCV1. The compression ratio depends on the proportion of actual text in the collection. Reuters-RCV1 contains a large amount of XML markup. Using the two best compression schemes, γ encoding and blocking with front coding, the ratio compressed index to collection size is therefore especially small for Reuters-RCV1: (101 + 5.9)/3600 ≈ 0.03. Two gap sequences to be merged in blocked sort-based indexing
Cosine computation for Exercise 6.19.
Calculation of 11-point Interpolated Average Precision. Calculating the kappa statistic.
RDB (relational database) search, unstructured information retrieval and structured information retrieval. INEX 2002 collection statistics. INEX 2002 results of the vector space model in Section 10.3 for content-and-structure (CAS) queries and the quantization function Q. A comparison of content-only and full-structure search in INEX 2003/2004.
Data for parameter estimation examples. Training and test times for NB. Multinomial versus Bernoulli model. Correct estimation implies accurate prediction, but accurate prediction does not imply correct estimation. A set of documents for which the NB independence assumptions are problematic. Critical values of the χ2 distribution with one degree of freedom. For example, if the two events are independent, then P( X 2 > 6.63) < 0.01. So for X 2 > 6.63 the assumption of independence can be rejected with 99% confidence. The ten largest classes in the Reuters-21578 collection with number of documents in training and test sets.
Macro- and microaveraging. "Truth" is the true class and "call" the decision of the classifier. In this example, macroaveraged precision is [10/(10 + 10) + 90/(10 + 90)]/2 = (0.5 + 0.9)/2 = 0.7. Microaveraged precision is 100/(100 + 20) ≈ 0.83. 13.9 Text classification effectiveness numbers on Reuters-21578 for F1 (in percent). Results from Li and Yang (2003) (a), Joachims (1998) (b: kNN) and Dumais et al. (1998) (b: NB, Rocchio, trees, SVM). 13.10 Data for parameter estimation exercise.
Vectors and class centroids for the data in Table 13.1. Training and test times for Rocchio classification. Training and test times for kNN classification. A linear classifier. A confusion matrix for Reuters-21578.
Training and testing complexity of various classifiers including SVMs. SVM classifier break-even F1 from (Joachims 2002a, p. 114). Training examples for machine-learned scoring.
Some applications of clustering in information retrieval. The four external evaluation measures applied to the clustering in Figure 16.4. The EM clustering algorithm.
Comparison of HAC algorithms. Automatically computed cluster labels.
1.1 1.2 1.3 1.4 1.5 1.6 1.7 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12
A term-document incidence matrix. Results from Shakespeare for the query Brutus AND Caesar AND NOT Calpurnia. The two parts of an inverted index. Building an index by sorting and grouping. Intersecting the postings lists for Brutus and Calpurnia from Figure 1.3. Algorithm for the intersection of two postings lists p1 and p2 . Algorithm for conjunctive queries that returns the set of documents containing each term in the input list of terms. An example of a vocalized Modern Standard Arabic word. The conceptual linear order of characters is not necessarily the order that you see on the page. The standard unsegmented form of Chinese text using the simplified characters of mainland China. Ambiguities in Chinese word segmentation. A stop list of 25 semantically non-selective words which are common in Reuters-RCV1. An example of how asymmetric expansion of query terms can usefully model users' expectations. Japanese makes use of multiple intermingled writing systems and, like Chinese, does not segment words. A comparison of three stemming algorithms on a sample text. Postings lists with skip pointers. Postings lists intersection with skip pointers. Positional index example. An algorithm for proximity intersection of postings lists p1 and p2 .
4 5 7 8 10 11 12 21 21 26 26 26 28 31 34 36 37 41 42
3.1 3.2 3.3 3.4 3.5 3.6 3.7 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8
5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 5.10 6.1 6.2 6.3
A binary search tree. A B-tree. A portion of a permuterm index. Example of a postings list in a 3-gram index. Dynamic programming algorithm for computing the edit distance between strings s1 and s2 . Example Levenshtein distance computation. Matching at least two of the three 2-grams in the query bord.
Document from the Reuters newswire. Blocked sort-based indexing. Merging in blocked sort-based indexing. Inversion of a block in single-pass in-memory indexing An example of distributed indexing with MapReduce. Adapted from Dean and Ghemawat (2004). Map and reduce functions in MapReduce. Logarithmic merging. Each token (termID,docID) is initially added to in-memory index Z0 by LM ERGE A DD T OKEN. L OGARITHMIC M ERGE initializes Z0 and indexes. A user-document matrix for access control lists. Element (i, j) is 1 if user i has access to document j and 0 otherwise. During query processing, a user's access postings list is intersected with the results list returned by the text part of the index.
Heaps' law. Zipf's law for Reuters-RCV1. Storing the dictionary as an array of fixed-width entries. Dictionary-as-a-string storage. Blocked storage with four terms per block. Search of the uncompressed dictionary (a) and a dictionary compressed by blocking with k = 4 (b). Front coding. VB encoding and decoding. Entropy H ( P) as a function of P( x1 ) for a sample space with two outcomes x1 and x2 . Stratification of terms for estimating the size of a γ encoded inverted index.
Parametric search. Basic zone index Zone index in which the zone is encoded in the postings rather than the dictionary.
94 94 97 100 102 111 111 111
6.4 6.5 6.6 6.7 6.8 6.9 6.10 6.11 6.12 6.13 6.14 6.15 6.16 6.17
Algorithm for computing the weighted zone score from two postings lists. An illustration of training examples. The four possible combinations of s T and s B . Collection frequency (cf) and document frequency (df) behave differently, as in this example from the Reuters collection. Example of idf values. Table of tf values for Exercise 6.10. Cosine similarity illustrated. Euclidean normalized tf values for documents in Figure 6.9. Term frequencies in three novels. Term vectors for the three novels of Figure 6.12. The basic algorithm for computing vector space scores. SMART notation for tf-idf variants. Pivoted document length normalization. Implementing pivoted document length normalization by linear scaling.
A faster algorithm for vector space scores. A static quality-ordered index. Cluster pruning. Tiered indexes. A complete search system.
Graph comparing the harmonic mean to other means. Precision/recall graph. Averaged 11-point precision/recall graph across 50 queries for a representative TREC system. The ROC curve corresponding to the precision-recall curve in Figure 8.2. An example of selecting text for a dynamic snippet.
Relevance feedback searching over images. Example of relevance feedback on a text collection. The Rocchio optimal query for separating relevant and nonrelevant documents. An application of Rocchio's algorithm. Results showing pseudo relevance feedback greatly improving performance. An example of query expansion in the interface of the Yahoo! web search engine in 2006. Examples of query expansion via the PubMed thesaurus. An example of an automatically generated thesaurus.
An XML document. The XML document in Figure 10.1 as a simplified DOM object. An XML query in NEXI format and its partial representation as a tree. 10.4 Tree representation of XML documents and queries. 10.5 Partitioning an XML document into non-overlapping indexing units. 10.6 Schema heterogeneity: intervening nodes and mismatched names. 10.7 A structural mismatch between two queries and a document. 10.8 A mapping of an XML document (left) to a set of lexicalized subtrees (right). 10.9 The algorithm for scoring documents with S IM N O M ERGE. 10.10 Scoring of a query with one structural term in S IM N O M ERGE. 10.11 Simplified schema of the documents in the INEX collection.
A tree of dependencies between terms.
A simple finite automaton and some of the strings in the language it generates. A one-state finite automaton that acts as a unigram language model. Partial specification of two unigram language models. Results of a comparison of tf-idf with language modeling (LM) term weighting by Ponte and Croft (1998). Three ways of developing the language modeling approach: (a) query likelihood, (b) document likelihood, and (c) model comparison.
Classes, training set, and test set in text classification . Naive Bayes algorithm (multinomial model): Training and testing. NB algorithm (Bernoulli model): Training and testing. The multinomial NB model. The Bernoulli NB model. Basic feature selection algorithm for selecting the k best features. Features with high mutual information scores for six Reuters-RCV1 classes. Effect of feature set size on accuracy for multinomial and Bernoulli models. A sample document from the Reuters-21578 collection.
Vector space classification into three classes.
14.2 14.3 14.4 14.5 14.6 14.7 14.8 14.9 14.10 14.11 14.12 14.13 14.14 14.15 15.1 15.2 15.3 15.4 15.5 15.6 15.7 16.1 16.2 16.3 16.4 16.5 16.6 16.7 16.8 17.1 17.2
Projections of small areas of the unit sphere preserve distances. Rocchio classification. Rocchio classification: Training and testing. The multimodal class "a" consists of two different clusters (small upper circles centered on X's). Voronoi tessellation and decision boundaries (double lines) in 1NN classification. kNN training (with preprocessing) and testing. There are an infinite number of hyperplanes that separate two linearly separable classes. Linear classification algorithm. A linear problem with noise. A nonlinear problem. J hyperplanes do not divide space into J disjoint regions. Arithmetic transformations for the bias-variance decomposition. Example for differences between Euclidean distance, dot product similarity and cosine similarity. A simple non-separable set of points. The support vectors are the 5 points right up against the margin of the classifier. An intuition for large-margin classification. The geometric margin of a point (r) and a decision boundary (ρ). A tiny 3 data point training set for an SVM. Large margin classification with slack variables. Projecting data that is not linearly separable into a higher dimensional space can make it linearly separable. A collection of training examples. An example of a data set with a clear cluster structure. Clustering of search results to improve recall. An example of a user session in Scatter-Gather. Purity as an external evaluation criterion for cluster quality. The K-means algorithm. A K-means example for K = 2 in R2 . The outcome of clustering in K-means depends on the initial seeds. Estimated minimal residual sum of squares as a function of the number of clusters in K-means. A dendrogram of a single-link clustering of 30 documents from Reuters-RCV1. A simple, but inefficient HAC algorithm.
xxiii 291 293 295 295 297 298 301 302 304 305 307 310 316 317 320 321 323 325 327 331 343 349 352 353 357 361 362 364 366 379 381
The different notions of cluster similarity used by the four HAC algorithms. 17.4 A single-link (left) and complete-link (right) clustering of eight documents. 17.5 A dendrogram of a complete-link clustering. 17.6 Chaining in single-link clustering. 17.7 Outliers in complete-link clustering. 17.8 The priority-queue algorithm for HAC. 17.9 Single-link clustering algorithm using an NBM array. 17.10 Complete-link clustering is not best-merge persistent. 17.11 Three iterations of centroid clustering. 17.12 Centroid clustering is not monotonic. 18.1 18.2
Illustration of the singular-value decomposition. Illustration of low rank approximation using the singular-value decomposition. The documents of Example 18.4 reduced to two dimensions in (V ′ ) T . Documents for Exercise 18.11. Glossary for Exercise 18.11.
19.1 19.2 19.3 19.4 19.5 19.6 19.7 19.8 19.9
A dynamically generated web page. Two nodes of the web graph joined by a link. A sample small web graph. The bowtie structure of the Web. Cloaking as used by spammers. Search advertising triggered by query keywords. The various components of a web search engine. Illustration of shingle sketches. Two sets S j1 and S j2 ; their Jaccard coefficient is 2/5.
The basic crawler architecture. Distributing the basic crawl architecture. The URL frontier. Example of an auxiliary hosts-to-back queues table. A lexicographically ordered set of URLs. A four-row segment of the table of links.
The random surfer at node A proceeds with probability 1/3 to each of B, C and D. A simple Markov chain with three states; the numbers on the links indicate the transition probabilities. The sequence of probability vectors.
A small web graph. Topic-specific PageRank. A sample run of HITS on the query japan elementary schools. Web graph for Exercise 21.22.
Table of Notation
γ code
p. 256 Classification or clustering function: γ(d) is d's class or cluster
p. 256 Supervised learning method in Chapters 13 and 14: Γ(D ) is the classification function γ learned from training set D
p. 404 Eigenvalue
~µ (.)
p. 292 Centroid of a class (in Rocchio classification) or a cluster (in K-means and centroid clustering)
p. 114 Training example
p. 408 Singular value
Θ(·)
ω, ωk
p. 357 Cluster in clustering
p. 357 Clustering or set of clusters {ω1 , . . . , ωK }
A tight bound on the complexity of an algorithm
arg maxx f ( x ) p. 181 The value of x for which f reaches its maximum arg minx f ( x ) p. 181 The value of x for which f reaches its minimum c, c j
p. 256 Class or category in classification
p. 256 Set {c1 , . . . , c J } of all classes
The collection frequency of term t (the total number of times the term appears in the document collection)
p. 268 A random variable that takes as values members of C
p. 403 Term-document matrix
Index of the dth document in the collection D
A document
~ ~q d,
p. 181 Document vector, query vector
p. 354 Set {d1 , . . . , d N } of all documents
p. 292 Set of documents that is in class c
p. 256 Set {hd1 , c1 i, . . . , hd N , c N i} of all labeled documents in Chapters 13–15
p. 118 The document frequency of term t (the total number of documents in the collection the term appears in)
Mth harmonic number
I ( X; Y )
p. 272 Mutual information of random variables X and Y
p. 118 Inverse document frequency of term t
p. 256 Number of classes
p. 290 Top k items from a set, e.g., k nearest neighbors in kNN, top k retrieved documents, top k selected features from the vocabulary V
p. 354 Number of clusters
p. 233 Length of document d (in tokens)
p. 262 Length of the test document (or application document) in tokens
Average length of a document (in tokens)
Size of the vocabulary (|V |)
p. 262 Size of the vocabulary of the test document (or application document)
p. 237 Language model for document d
p. 259 Number of documents in class c
N (ω )
p. 298 Number of times the event ω occurred
Sequence of k characters
Average size of the vocabulary in a document in the collection Number of documents in the retrieval or training collection
O(·)
p. 221 The odds of an event
p. 155 Precision
P(·)
p. 220 Probability
p. 465 Transition probability matrix
p. 155 Recall
p. 112 Boolean values for zone scoring
sim(d1 , d2 )
p. 121 Similarity score for documents d1 , d2
p. 259 Number of occurrences of word t in documents of class c
Index of the tth term in the vocabulary V
A term in the vocabulary
tft,d
p. 117 The term frequency of term t in document d (the total number of occurrences of t in d)
p. 266 Random variable taking values 0 (term t is present) and 1 (t is not present)
p. 208 Vocabulary of terms {t1 , . . . , t M } in a collection (a.k.a. the lexicon)
~v(d) ~ (d) V
p. 122 Length-normalized document vector
wft,d
p. 125 Weight of term t in document d
p. 112 A weight, for example for zones or terms T
A bound on the complexity of an algorithm
A query A string
Total number of tokens in the document collection
p. 120 Vector of document d, not length-normalized
w ~ ~x = b
~ is the normal vector of the hyperp. 293 Hyperplane; w plane and wi component i of w ~
~x
p. 222 Term incidence vector ~x = ( x1 , . . . , x M ); more generally: document feature representation
p. 266 Random variable taking values in V, the vocabulary (e.g., at a given position k in a document)
p. 256 Document space in text classification
| A|
|S|
Set cardinality: the number of members of set A
p. 404 Determinant of the square matrix S
|si |
|~x |
|~x − ~y|
Length in characters of string si
p. 139 Length of vector ~x p. 131 Euclidean distance of ~x and ~y (which is the length of (~x − ~y))
As recently as the 1990s, studies showed that most people preferred getting information from other people rather than from information retrieval systems. Of course, in that time period, most people also used human travel agents to book their travel. However, during the last decade, relentless optimization of information retrieval effectiveness has driven web search engines to new quality levels where most people are satisfied most of the time, and web search has become a standard and often preferred source of information finding. For example, the 2004 Pew Internet Survey (Fallows 2004) found that "92% of Internet users say the Internet is a good place to go for getting everyday information." To the surprise of many, the field of information retrieval has moved from being a primarily academic discipline to being the basis underlying most people's preferred means of information access. This book presents the scientific underpinnings of this field, at a level accessible to graduate students as well as advanced undergraduates. Information retrieval did not begin with the Web. In response to various challenges of providing information access, the field of information retrieval evolved to give principled approaches to searching various forms of content. The field began with scientific publications and library records, but soon spread to other forms of content, particularly those of information professionals, such as journalists, lawyers, and doctors. Much of the scientific research on information retrieval has occurred in these contexts, and much of the continued practice of information retrieval deals with providing access to unstructured information in various corporate and governmental domains, and this work forms much of the foundation of our book. Nevertheless, in recent years, a principal driver of innovation has been the World Wide Web, unleashing publication at the scale of tens of millions of content creators. This explosion of published information would be moot if the information could not be found, annotated and analyzed so that each user can quickly find information that is both relevant and comprehensive for their needs. By the late 1990s, many people felt that continuing to index
the whole Web would rapidly become impossible, due to the Web's exponential growth in size. But major scientific innovations, superb engineering, the rapidly declining price of computer hardware, and the rise of a commercial underpinning for web search have all conspired to power today's major search engines, which are able to provide high-quality results within subsecond response times for hundreds of millions of searches a day over billions of web pages.
Book organization and course development This book is the result of a series of courses we have taught at Stanford University and at the University of Stuttgart, in a range of durations including a single quarter, one semester and two quarters. These courses were aimed at early-stage graduate students in computer science, but we have also had enrollment from upper-class computer science undergraduates, as well as students from law, medical informatics, statistics, linguistics and various engineering disciplines. The key design principle for this book, therefore, was to cover what we believe to be important in a one-term graduate course on information retrieval. An additional principle is to build each chapter around material that we believe can be covered in a single lecture of 75 to 90 minutes. The first eight chapters of the book are devoted to the basics of information retrieval, and in particular the heart of search engines; we consider this material to be core to any course on information retrieval. Chapter 1 introduces inverted indexes, and shows how simple Boolean queries can be processed using such indexes. Chapter 2 builds on this introduction by detailing the manner in which documents are preprocessed before indexing and by discussing how inverted indexes are augmented in various ways for functionality and speed. Chapter 3 discusses search structures for dictionaries and how to process queries that have spelling errors and other imprecise matches to the vocabulary in the document collection being searched. Chapter 4 describes a number of algorithms for constructing the inverted index from a text collection with particular attention to highly scalable and distributed algorithms that can be applied to very large collections. Chapter 5 covers techniques for compressing dictionaries and inverted indexes. These techniques are critical for achieving subsecond response times to user queries in large search engines. The indexes and queries considered in Chapters 1–5 only deal with Boolean retrieval, in which a document either matches a query, or does not. A desire to measure the extent to which a document matches a query, or the score of a document for a query, motivates the development of term weighting and the computation of scores in Chapters 6 and 7, leading to the idea of a list of documents that are rank-ordered for a query. Chapter 8 focuses on the evaluation of an information retrieval system based on the
relevance of the documents it retrieves, allowing us to compare the relative performances of different systems on benchmark document collections and queries. Chapters 9–21 build on the foundation of the first eight chapters to cover a variety of more advanced topics. Chapter 9 discusses methods by which retrieval can be enhanced through the use of techniques like relevance feedback and query expansion, which aim at increasing the likelihood of retrieving relevant documents. Chapter 10 considers information retrieval from documents that are structured with markup languages like XML and HTML. We treat structured retrieval by reducing it to the vector space scoring methods developed in Chapter 6. Chapters 11 and 12 invoke probability theory to compute scores for documents on queries. Chapter 11 develops traditional probabilistic information retrieval, which provides a framework for computing the probability of relevance of a document, given a set of query terms. This probability may then be used as a score in ranking. Chapter 12 illustrates an alternative, wherein for each document in a collection, we build a language model from which one can estimate a probability that the language model generates a given query. This probability is another quantity with which we can rank-order documents. Chapters 13–17 give a treatment of various forms of machine learning and numerical methods in information retrieval. Chapters 13–15 treat the problem of classifying documents into a set of known categories, given a set of documents along with the classes they belong to. Chapter 13 motivates statistical classification as one of the key technologies needed for a successful search engine, introduces Naive Bayes, a conceptually simple and efficient text classification method, and outlines the standard methodology for evaluating text classifiers. Chapter 14 employs the vector space model from Chapter 6 and introduces two classification methods, Rocchio and kNN, that operate on document vectors. It also presents the bias-variance tradeoff as an important characterization of learning problems that provides criteria for selecting an appropriate method for a text classification problem. Chapter 15 introduces support vector machines, which many researchers currently view as the most effective text classification method. We also develop connections in this chapter between the problem of classification and seemingly disparate topics such as the induction of scoring functions from a set of training examples. Chapters 16–18 consider the problem of inducing clusters of related documents from a collection. In Chapter 16, we first give an overview of a number of important applications of clustering in information retrieval. We then describe two flat clustering algorithms: the K-means algorithm, an efficient and widely used document clustering method; and the ExpectationMaximization algorithm, which is computationally more expensive, but also more flexible. Chapter 17 motivates the need for hierarchically structured
clusterings (instead of flat clusterings) in many applications in information retrieval and introduces a number of clustering algorithms that produce a hierarchy of clusters. The chapter also addresses the difficult problem of automatically computing labels for clusters. Chapter 18 develops methods from linear algebra that constitute an extension of clustering, and also offer intriguing prospects for algebraic methods in information retrieval, which have been pursued in the approach of latent semantic indexing. Chapters 19–21 treat the problem of web search. We give in Chapter 19 a summary of the basic challenges in web search, together with a set of techniques that are pervasive in web information retrieval. Next, Chapter 20 describes the architecture and requirements of a basic web crawler. Finally, Chapter 21 considers the power of link analysis in web search, using in the process several methods from linear algebra and advanced probability theory. This book is not comprehensive in covering all topics related to information retrieval. We have put aside a number of topics, which we deemed outside the scope of what we wished to cover in an introduction to information retrieval class. Nevertheless, for people interested in these topics, we provide a few pointers to mainly textbook coverage here. Cross-language IR (Grossman and Frieder 2004, ch. 4) and (Oard and Dorr 1996). Image and Multimedia IR (Grossman and Frieder 2004, ch. 4), (Baeza-Yates and Ribeiro-Neto 1999, ch. 6), (Baeza-Yates and Ribeiro-Neto 1999, ch. 11), (Baeza-Yates and Ribeiro-Neto 1999, ch. 12), (del Bimbo 1999), (Lew 2001), and (Smeulders et al. 2000). Speech retrieval (Coden et al. 2002). Music Retrieval (Downie 2006) and http://www.ismir.net/. User interfaces for IR (Baeza-Yates and Ribeiro-Neto 1999, ch. 10). Parallel and Peer-to-Peer IR (Grossman and Frieder 2004, ch. 7), (Baeza-Yates and Ribeiro-Neto 1999, ch. 9), and (Aberer 2001). Digital libraries (Baeza-Yates and Ribeiro-Neto 1999, ch. 15) and (Lesk 2004). Information science perspective (Korfhage 1997), (Meadow et al. 1999), and (Ingwersen and Järvelin 2005). Logic-based approaches to IR (van Rijsbergen 1989). Natural Language Processing techniques (Manning and Schütze 1999), (Jurafsky and Martin 2008), and (Lewis and Jones 1996).
Prerequisites Introductory courses in data structures and algorithms, in linear algebra and in probability theory suffice as prerequisites for all 21 chapters. We now give more detail for the benefit of readers and instructors who wish to tailor their reading to some of the chapters. Chapters 1–5 assume as prerequisite a basic course in algorithms and data structures. Chapters 6 and 7 require, in addition, a knowledge of basic linear algebra including vectors and dot products. No additional prerequisites are assumed until Chapter 11, where a basic course in probability theory is required; Section 11.1 gives a quick review of the concepts necessary in Chapters 11–13. Chapter 15 assumes that the reader is familiar with the notion of nonlinear optimization, although the chapter may be read without detailed knowledge of algorithms for nonlinear optimization. Chapter 18 demands a first course in linear algebra including familiarity with the notions of matrix rank and eigenvectors; a brief review is given in Section 18.1. The knowledge of eigenvalues and eigenvectors is also necessary in Chapter 21.
Book layout
✎ ✄ ?
Worked examples in the text appear with a pencil sign next to them in the left margin. Advanced or difficult material appears in sections or subsections indicated with scissors in the margin. Exercises are marked in the margin with a question mark. The level of difficulty of exercises is indicated as easy (⋆), medium (⋆⋆), or difficult (⋆ ⋆ ⋆).
Acknowledgments We would like to thank Cambridge University Press for allowing us to make the draft book available online, which facilitated much of the feedback we have received while writing the book. We also thank Lauren Cowles, who has been an outstanding editor, providing several rounds of comments on each chapter, on matters of style, organization, and coverage, as well as detailed comments on the subject matter of the book. To the extent that we have achieved our goals in writing this book, she deserves an important part of the credit. We are very grateful to the many people who have given us comments, suggestions, and corrections based on draft versions of this book. We thank for providing various corrections and comments: Cheryl Aasheim, Josh Attenberg, Daniel Beck, Luc Bélanger, Georg Buscher, Tom Breuel, Daniel Burckhardt, Fazli Can, Dinquan Chen, Stephen Clark, Ernest Davis, Pedro Domingos, Rodrigo Panchiniak Fernandes, Paolo Ferragina, Alex Fraser, Norbert
xxxvi
Fuhr, Vignesh Ganapathy, Elmer Garduno, Xiubo Geng, David Gondek, Sergio Govoni, Corinna Habets, Ben Handy, Donna Harman, Benjamin Haskell, Thomas Hühn, Deepak Jain, Ralf Jankowitsch, Dinakar Jayarajan, Vinay Kakade, Mei Kobayashi, Wessel Kraaij, Rick Lafleur, Florian Laws, Hang Li, David Losada, David Mann, Ennio Masi, Sven Meyer zu Eissen, Alexander Murzaku, Gonzalo Navarro, Frank McCown, Paul McNamee, Christoph Müller, Scott Olsson, Tao Qin, Megha Raghavan, Michal Rosen-Zvi, Klaus Rothenhäusler, Kenyu L. Runner, Alexander Salamanca, Grigory Sapunov, Evgeny Shadchnev, Tobias Scheffer, Nico Schlaefer, Ian Soboroff, Benno Stein, Marcin Sydow, Andrew Turner, Jason Utt, Huey Vo, Travis Wade, Mike Walsh, Changliang Wang, Renjing Wang, and Thomas Zeume. Many people gave us detailed feedback on individual chapters, either at our request or through their own initiative. For this, we're particularly grateful to: James Allan, Omar Alonso, Ismail Sengor Altingovde, Vo Ngoc Anh, Roi Blanco, Eric Breck, Eric Brown, Mark Carman, Carlos Castillo, Junghoo Cho, Aron Culotta, Doug Cutting, Meghana Deodhar, Susan Dumais, Johannes Fürnkranz, Andreas Heß, Djoerd Hiemstra, David Hull, Thorsten Joachims, Siddharth Jonathan J. B., Jaap Kamps, Mounia Lalmas, Amy Langville, Nicholas Lester, Dave Lewis, Daniel Lowd, Yosi Mass, Jeff Michels, Alessandro Moschitti, Amir Najmi, Marc Najork, Giorgio Maria Di Nunzio, Paul Ogilvie, Priyank Patel, Jan Pedersen, Kathryn Pedings, Vassilis Plachouras, Daniel Ramage, Ghulam Raza, Stefan Riezler, Michael Schiehlen, Helmut Schmid, Falk Nicolas Scholer, Sabine Schulte im Walde, Fabrizio Sebastiani, Sarabjeet Singh, Valentin Spitkovsky, Alexander Strehl, John Tait, Shivakumar Vaithyanathan, Ellen Voorhees, Gerhard Weikum, Dawid Weiss, Yiming Yang, Yisong Yue, Jian Zhang, and Justin Zobel. And finally there were a few reviewers who absolutely stood out in terms of the quality and quantity of comments that they provided. We thank them for their significant impact on the content and structure of the book. We express our gratitude to Pavel Berkhin, Stefan Büttcher, Jamie Callan, Byron Dom, Torsten Suel, and Andrew Trotman. Parts of the initial drafts of Chapters 13–15 were based on slides that were generously provided by Ray Mooney. While the material has gone through extensive revisions, we gratefully acknowledge Ray's contribution to the three chapters in general and to the description of the time complexities of text classification algorithms in particular. The above is unfortunately an incomplete list: we are still in the process of incorporating feedback we have received. And, like all opinionated authors, we did not always heed the advice that was so freely given. The published versions of the chapters remain solely the responsibility of the authors. The authors thank Stanford University and the University of Stuttgart for providing a stimulating academic environment for discussing ideas and the opportunity to teach courses from which this book arose and in which its
contents were refined. CM thanks his family for the many hours they've let him spend working on this book, and hopes he'll have a bit more free time on weekends next year. PR thanks his family for their patient support through the writing of this book and is also grateful to Yahoo! Inc. for providing a fertile environment in which to work on this book. HS would like to thank his parents, family, and friends for their support while writing this book.
Web and contact information This book has a companion website at http://informationretrieval.org. As well as links to some more general resources, it is our intent to maintain on this website a set of slides for each chapter which may be used for the corresponding lecture. We gladly welcome further feedback, corrections, and suggestions on the book, which may be sent to all the authors at informationretrieval (at) yahoogroups (dot) com.
1 INFORMATION RETRIEVAL
Boolean retrieval
The meaning of the term information retrieval can be very broad. Just getting a credit card out of your wallet so that you can type in the card number is a form of information retrieval. However, as an academic field of study, information retrieval might be defined thus: Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers). As defined in this way, information retrieval used to be an activity that only a few people engaged in: reference librarians, paralegals, and similar professional searchers. Now the world has changed, and hundreds of millions of people engage in information retrieval every day when they use a web search engine or search their email.1 Information retrieval is fast becoming the dominant form of information access, overtaking traditional databasestyle searching (the sort that is going on when a clerk says to you: "I'm sorry, I can only look up your order if you can give me your Order ID"). IR can also cover other kinds of data and information problems beyond that specified in the core definition above. The term "unstructured data" refers to data which does not have clear, semantically overt, easy-for-a-computer structure. It is the opposite of structured data, the canonical example of which is a relational database, of the sort companies usually use to maintain product inventories and personnel records. In reality, almost no data are truly "unstructured". This is definitely true of all text data if you count the latent linguistic structure of human languages. But even accepting that the intended notion of structure is overt structure, most text has structure, such as headings and paragraphs and footnotes, which is commonly represented in documents by explicit markup (such as the coding underlying web 1. In modern parlance, the word "search" has tended to replace "(information) retrieval"; the term "search" is quite ambiguous, but in context we use the two synonymously.
1 Boolean retrieval
pages). IR is also used to facilitate "semistructured" search such as finding a document where the title contains Java and the body contains threading. The field of information retrieval also covers supporting users in browsing or filtering document collections or further processing a set of retrieved documents. Given a set of documents, clustering is the task of coming up with a good grouping of the documents based on their contents. It is similar to arranging books on a bookshelf according to their topic. Given a set of topics, standing information needs, or other categories (such as suitability of texts for different age groups), classification is the task of deciding which class(es), if any, each of a set of documents belongs to. It is often approached by first manually classifying some documents and then hoping to be able to classify new documents automatically. Information retrieval systems can also be distinguished by the scale at which they operate, and it is useful to distinguish three prominent scales. In web search, the system has to provide search over billions of documents stored on millions of computers. Distinctive issues are needing to gather documents for indexing, being able to build systems that work efficiently at this enormous scale, and handling particular aspects of the web, such as the exploitation of hypertext and not being fooled by site providers manipulating page content in an attempt to boost their search engine rankings, given the commercial importance of the web. We focus on all these issues in Chapters 19–21. At the other extreme is personal information retrieval. In the last few years, consumer operating systems have integrated information retrieval (such as Apple's Mac OS X Spotlight or Windows Vista's Instant Search). Email programs usually not only provide search but also text classification: they at least provide a spam (junk mail) filter, and commonly also provide either manual or automatic means for classifying mail so that it can be placed directly into particular folders. Distinctive issues here include handling the broad range of document types on a typical personal computer, and making the search system maintenance free and sufficiently lightweight in terms of startup, processing, and disk space usage that it can run on one machine without annoying its owner. In between is the space of enterprise, institutional, and domain-specific search, where retrieval might be provided for collections such as a corporation's internal documents, a database of patents, or research articles on biochemistry. In this case, the documents will typically be stored on centralized file systems and one or a handful of dedicated machines will provide search over the collection. This book contains techniques of value over this whole spectrum, but our coverage of some aspects of parallel and distributed search in web-scale search systems is comparatively light owing to the relatively small published literature on the details of such systems. However, outside of a handful of web search companies, a software developer is most likely to encounter the personal search and enterprise scenarios.
1.1 An example information retrieval problem
In this chapter we begin with a very simple example of an information retrieval problem, and introduce the idea of a term-document matrix (Section 1.1) and the central inverted index data structure (Section 1.2). We will then examine the Boolean retrieval model and how Boolean queries are processed (Sections 1.3 and 1.4).
An example information retrieval problem A fat book which many people own is Shakespeare's Collected Works. Suppose you wanted to determine which plays of Shakespeare contain the words Brutus AND Caesar AND NOT Calpurnia. One way to do that is to start at the beginning and to read through all the text, noting for each play whether it contains Brutus and Caesar and excluding it from consideration if it contains Calpurnia. The simplest form of document retrieval is for a computer to do this sort of linear scan through documents. This process is commonly referred to as grepping through text, after the Unix command grep, which performs this process. Grepping through text can be a very effective process, especially given the speed of modern computers, and often allows useful possibilities for wildcard pattern matching through the use of regular expressions. With modern computers, for simple querying of modest collections (the size of Shakespeare's Collected Works is a bit under one million words of text in total), you really need nothing more. But for many purposes, you do need more: 1. To process large document collections quickly. The amount of online data has grown at least as quickly as the speed of computers, and we would now like to be able to search collections that total in the order of billions to trillions of words. 2. To allow more flexible matching operations. For example, it is impractical to perform the query Romans NEAR countrymen with grep, where NEAR might be defined as "within 5 words" or "within the same sentence". 3. To allow ranked retrieval: in many cases you want the best answer to an information need among many documents that contain certain words.
INCIDENCE MATRIX TERM
The way to avoid linearly scanning the texts for each query is to index the documents in advance. Let us stick with Shakespeare's Collected Works, and use it to introduce the basics of the Boolean retrieval model. Suppose we record for each document – here a play of Shakespeare's – whether it contains each word out of all the words Shakespeare used (Shakespeare used about 32,000 different words). The result is a binary term-document incidence matrix, as in Figure 1.1. Terms are the indexed units (further discussed in Section 2.2); they are usually words, and for the moment you can think of
Antony Brutus Caesar Calpurnia Cleopatra mercy worser ...
Antony and Cleopatra 1 1 1 0 1 1 1
◮ Figure 1.1 A term-document incidence matrix. Matrix element (t, d) is 1 if the play in column d contains the word in row t, and is 0 otherwise.
them as words, but the information retrieval literature normally speaks of terms because some of them, such as perhaps I-9 or Hong Kong are not usually thought of as words. Now, depending on whether we look at the matrix rows or columns, we can have a vector for each term, which shows the documents it appears in, or a vector for each document, showing the terms that occur in it.2 To answer the query Brutus AND Caesar AND NOT Calpurnia, we take the vectors for Brutus, Caesar and Calpurnia, complement the last, and then do a bitwise AND: 110100 AND 110111 AND 101111 = 100100
B OOLEAN RETRIEVAL MODEL
COLLECTION CORPUS
The answers for this query are thus Antony and Cleopatra and Hamlet (Figure 1.2). The Boolean retrieval model is a model for information retrieval in which we can pose any query which is in the form of a Boolean expression of terms, that is, in which terms are combined with the operators AND, OR, and NOT. The model views each document as just a set of words. Let us now consider a more realistic scenario, simultaneously using the opportunity to introduce some terminology and notation. Suppose we have N = 1 million documents. By documents we mean whatever units we have decided to build a retrieval system over. They might be individual memos or chapters of a book (see Section 2.1.2 (page 20) for further discussion). We will refer to the group of documents over which we perform retrieval as the (document) collection. It is sometimes also referred to as a corpus (a body of texts). Suppose each document is about 1000 words long (2–3 book pages). If 2. Formally, we take the transpose of the matrix to be able to get the terms as column vectors.
Antony and Cleopatra, Act III, Scene ii Agrippa [Aside to Domitius Enobarbus]: Why, Enobarbus, When Antony found Julius Caesar dead, He cried almost to roaring; and he wept When at Philippi he found Brutus slain. Hamlet, Act III, Scene ii Lord Polonius:
I did enact Julius Caesar: I was killed i' the Capitol; Brutus killed me.
◮ Figure 1.2 Results from Shakespeare for the query Brutus
AND Caesar AND NOT
Calpurnia.
AD HOC RETRIEVAL
INFORMATION NEED QUERY
we assume an average of 6 bytes per word including spaces and punctuation, then this is a document collection about 6 GB in size. Typically, there might be about M = 500,000 distinct terms in these documents. There is nothing special about the numbers we have chosen, and they might vary by an order of magnitude or more, but they give us some idea of the dimensions of the kinds of problems we need to handle. We will discuss and model these size assumptions in Section 5.1 (page 86). Our goal is to develop a system to address the ad hoc retrieval task. This is the most standard IR task. In it, a system aims to provide documents from within the collection that are relevant to an arbitrary user information need, communicated to the system by means of a one-off, user-initiated query. An information need is the topic about which the user desires to know more, and is differentiated from a query, which is what the user conveys to the computer in an attempt to communicate the information need. A document is relevant if it is one that the user perceives as containing information of value with respect to their personal information need. Our example above was rather artificial in that the information need was defined in terms of particular words, whereas usually a user is interested in a topic like "pipeline leaks" and would like to find relevant documents regardless of whether they precisely use those words or express the concept with other words such as pipeline rupture. To assess the effectiveness of an IR system (i.e., the quality of its search results), a user will usually want to know two key statistics about the system's returned results for a query: Precision: What fraction of the returned results are relevant to the information need? Recall: What fraction of the relevant documents in the collection were returned by the system?
INVERTED INDEX
DICTIONARY VOCABULARY LEXICON
POSTING POSTINGS LIST POSTINGS
Detailed discussion of relevance and evaluation measures including precision and recall is found in Chapter 8. We now cannot build a term-document matrix in a naive way. A 500K × 1M matrix has half-a-trillion 0's and 1's – too many to fit in a computer's memory. But the crucial observation is that the matrix is extremely sparse, that is, it has few non-zero entries. Because each document is 1000 words long, the matrix has no more than one billion 1's, so a minimum of 99.8% of the cells are zero. A much better representation is to record only the things that do occur, that is, the 1 positions. This idea is central to the first major concept in information retrieval, the inverted index. The name is actually redundant: an index always maps back from terms to the parts of a document where they occur. Nevertheless, inverted index, or sometimes inverted file, has become the standard term in information retrieval.3 The basic idea of an inverted index is shown in Figure 1.3. We keep a dictionary of terms (sometimes also referred to as a vocabulary or lexicon; in this book, we use dictionary for the data structure and vocabulary for the set of terms). Then for each term, we have a list that records which documents the term occurs in. Each item in the list – which records that a term appeared in a document (and, later, often, the positions in the document) – is conventionally called a posting.4 The list is then called a postings list (or inverted list), and all the postings lists taken together are referred to as the postings. The dictionary in Figure 1.3 has been sorted alphabetically and each postings list is sorted by document ID. We will see why this is useful in Section 1.3, below, but later we will also consider alternatives to doing this (Section 7.1.5).
A first take at building an inverted index To gain the speed benefits of indexing at retrieval time, we have to build the index in advance. The major steps in this are: 1. Collect the documents to be indexed: Friends, Romans, countrymen. So let it be with Caesar . . . 2. Tokenize the text, turning each document into a list of tokens: Friends Romans countrymen So . . . 3. Some information retrieval researchers prefer the term inverted file, but expressions like index construction and index compression are much more common than inverted file construction and inverted file compression. For consistency, we use (inverted) index throughout this book.
4. In a (non-positional) inverted index, a posting is just a document ID, but it is inherently associated with a term, via the postings list it is placed on; sometimes we will also talk of a (term, docID) pair as a posting.
1.2 A first take at building an inverted index
−→
.. . | {z } Dictionary
{z Postings
◮ Figure 1.3 The two parts of an inverted index. The dictionary is commonly kept in memory, with pointers to each postings list, which is stored on disk.
3. Do linguistic preprocessing, producing a list of normalized tokens, which are the indexing terms: friend roman countryman so . . . 4. Index the documents that each term occurs in by creating an inverted index, consisting of a dictionary and postings.
DOC ID
DOCUMENT FREQUENCY
We will define and discuss the earlier stages of processing, that is, steps 1–3, in Section 2.2 (page 22). Until then you can think of tokens and normalized tokens as also loosely equivalent to words. Here, we assume that the first 3 steps have already been done, and we examine building a basic inverted index by sort-based indexing. Within a document collection, we assume that each document has a unique serial number, known as the document identifier (docID). During index construction, we can simply assign successive integers to each new document when it is first encountered. The input to indexing is a list of normalized tokens for each document, which we can equally think of as a list of pairs of term and docID, as in Figure 1.4. The core indexing step is sorting this list so that the terms are alphabetical, giving us the representation in the middle column of Figure 1.4. Multiple occurrences of the same term from the same document are then merged.5 Instances of the same term are then grouped, and the result is split into a dictionary and postings, as shown in the right column of Figure 1.4. Since a term generally occurs in a number of documents, this data organization already reduces the storage requirements of the index. The dictionary also records some statistics, such as the number of documents which contain each term (the document frequency, which is here also the length of each postings list). This information is not vital for a basic Boolean search engine, but it allows us to improve the efficiency of the 5. Unix users can note that these steps are similar to use of the sort and then uniq commands.
Doc 1 I did enact Julius Caesar: I was killed i' the Capitol; Brutus killed me.
Doc 2 So let it be with Caesar. The noble Brutus hath told you Caesar was ambitious:
term docID term docID I 1 ambitious 2 term doc. freq. did 1 be 2 ambitious 1 enact 1 brutus 1 be 1 julius 1 brutus 2 brutus 2 caesar 1 capitol 1 I 1 caesar 1 capitol 1 was 1 caesar 2 caesar 2 killed 1 caesar 2 did 1 i' 1 did 1 enact 1 the 1 enact 1 hath 1 capitol 1 hath 1 brutus 1 I 1 I 1 killed 1 I 1 i' 1 me 1 i' 1 =⇒ =⇒ it 1 so 2 it 2 julius 1 let 2 julius 1 killed 1 it 2 killed 1 be 2 killed 1 let 1 with 2 let 2 me 1 caesar 2 me 1 noble 1 the 2 noble 2 so 1 noble 2 so 2 the 2 brutus 2 the 1 told 1 hath 2 the 2 told 2 told 2 you 1 you 2 you 2 was 2 caesar 2 was 1 with 1 was 2 was 2 ambitious 2 with 2
→ → → → → → → → → → → → → → → → → → → → → → →
postings lists 2 2 1 → 2 1 1 → 2 1 1 2 1 1 2 1 1 2 1 2 2 1 → 2 2 2 1 → 2 2
◮ Figure 1.4 Building an index by sorting and grouping. The sequence of terms in each document, tagged by their documentID (left) is sorted alphabetically (middle). Instances of the same term are then grouped by word and then by documentID. The terms and documentIDs are then separated out (right). The dictionary stores the terms, and has a pointer to the postings list for each term. It commonly also stores other summary information such as, here, the document frequency of each term. We use this information for improving query time efficiency and, later, for weighting in ranked retrieval models. Each postings list stores the list of documents in which a term occurs, and may store other information such as the term frequency (the frequency of each term in each document) or the position(s) of the term in each document.
search engine at query time, and it is a statistic later used in many ranked retrieval models. The postings are secondarily sorted by docID. This provides the basis for efficient query processing. This inverted index structure is essentially without rivals as the most efficient structure for supporting ad hoc text search. In the resulting index, we pay for storage of both the dictionary and the postings lists. The latter are much larger, but the dictionary is commonly kept in memory, while postings lists are normally kept on disk, so the size of each is important, and in Chapter 5 we will examine how each can be optimized for storage and access efficiency. What data structure should be used for a postings list? A fixed length array would be wasteful as some words occur in many documents, and others in very few. For an in-memory postings list, two good alternatives are singly linked lists or variable length arrays. Singly linked lists allow cheap insertion of documents into postings lists (following updates, such as when recrawling the web for updated documents), and naturally extend to more advanced indexing strategies such as skip lists (Section 2.3), which require additional pointers. Variable length arrays win in space requirements by avoiding the overhead for pointers and in time requirements because their use of contiguous memory increases speed on modern processors with memory caches. Extra pointers can in practice be encoded into the lists as offsets. If updates are relatively infrequent, variable length arrays will be more compact and faster to traverse. We can also use a hybrid scheme with a linked list of fixed length arrays for each term. When postings lists are stored on disk, they are stored (perhaps compressed) as a contiguous run of postings without explicit pointers (as in Figure 1.3), so as to minimize the size of the postings list and the number of disk seeks to read a postings list into memory.
Exercise 1.1 [ ⋆] Draw the inverted index that would be built for the following document collection. (See Figure 1.3 for an example.) Doc 1 Doc 2 Doc 3 Doc 4
new home sales top forecasts home sales rise in july increase in home sales in july july new home sales rise
Exercise 1.2 Consider these documents: Doc 1 Doc 2 Doc 3 Doc 4
breakthrough drug for schizophrenia new schizophrenia drug new approach for treatment of schizophrenia new hopes for schizophrenia patients
a. Draw the term-document incidence matrix for this document collection.
[ ⋆]
Brutus Calpurnia Intersection
1 → 2 → 4 → 11 → 31 → 45 → 173 → 174
2 → 31 → 54 → 101
=⇒
2 → 31
◮ Figure 1.5 Intersecting the postings lists for Brutus and Calpurnia from Figure 1.3.
b. Draw the inverted index representation for this collection, as in Figure 1.3 (page 7). Exercise 1.3
[⋆]
For the document collection shown in Exercise 1.2, what are the returned results for these queries: a. schizophrenia AND drug b. for AND NOT (drug OR approach)
1.3 SIMPLE CONJUNCTIVE QUERIES
Processing Boolean queries How do we process a query using an inverted index and the basic Boolean retrieval model? Consider processing the simple conjunctive query: Brutus AND Calpurnia
over the inverted index partially shown in Figure 1.3 (page 7). We: 1. Locate Brutus in the Dictionary 2. Retrieve its postings 3. Locate Calpurnia in the Dictionary 4. Retrieve its postings 5. Intersect the two postings lists, as shown in Figure 1.5. POSTINGS LIST INTERSECTION POSTINGS MERGE
The intersection operation is the crucial one: we need to efficiently intersect postings lists so as to be able to quickly find documents that contain both terms. (This operation is sometimes referred to as merging postings lists: this slightly counterintuitive name reflects using the term merge algorithm for a general family of algorithms that combine multiple sorted lists by interleaved advancing of pointers through each; here we are merging the lists with a logical AND operation.) There is a simple and effective method of intersecting postings lists using the merge algorithm (see Figure 1.6): we maintain pointers into both lists
1.3 Processing Boolean queries
I NTERSECT ( p1, p2 ) 1 answer ← h i 2 while p1 6= NIL and p2 6= NIL 3 do if docID ( p1) = docID ( p2) 4 then A DD ( answer, docID ( p1 )) 5 p1 ← next( p1 ) 6 p2 ← next( p2 ) 7 else if docID ( p1) < docID ( p2) 8 then p1 ← next( p1 ) 9 else p2 ← next( p2 ) 10 return answer ◮ Figure 1.6 Algorithm for the intersection of two postings lists p1 and p2 .
and walk through the two postings lists simultaneously, in time linear in the total number of postings entries. At each step, we compare the docID pointed to by both pointers. If they are the same, we put that docID in the results list, and advance both pointers. Otherwise we advance the pointer pointing to the smaller docID. If the lengths of the postings lists are x and y, the intersection takes O( x + y) operations. Formally, the complexity of querying is Θ( N ), where N is the number of documents in the collection.6 Our indexing methods gain us just a constant, not a difference in Θ time complexity compared to a linear scan, but in practice the constant is huge. To use this algorithm, it is crucial that postings be sorted by a single global ordering. Using a numeric sort by docID is one simple way to achieve this. We can extend the intersection operation to process more complicated queries like: (1.2) QUERY OPTIMIZATION
(Brutus OR Caesar) AND NOT Calpurnia Query optimization is the process of selecting how to organize the work of answering a query so that the least total amount of work needs to be done by the system. A major element of this for Boolean queries is the order in which postings lists are accessed. What is the best order for query processing? Consider a query that is an AND of t terms, for instance: Brutus AND Caesar AND Calpurnia
For each of the t terms, we need to get its postings, then AND them together. The standard heuristic is to process terms in order of increasing document 6. The notation Θ(·) is used to express an asymptotically tight bound on the complexity of an algorithm. Informally, this is often written as O (·), but this notation really expresses an asymptotic upper bound, which need not be tight (Cormen et al. 1990).
I NTERSECT (ht1, . . . , tn i) 1 terms ← S ORT B Y I NCREASING F REQUENCY (ht1 , . . . , tn i) 2 result ← postings( f irst(terms)) 3 terms ← rest(terms) 4 while terms 6= NIL and result 6= NIL 5 do result ← I NTERSECT (result, postings( f irst(terms))) 6 terms ← rest(terms) 7 return result ◮ Figure 1.7 Algorithm for conjunctive queries that returns the set of documents containing each term in the input list of terms.
frequency: if we start by intersecting the two smallest postings lists, then all intermediate results must be no bigger than the smallest postings list, and we are therefore likely to do the least amount of total work. So, for the postings lists in Figure 1.3 (page 7), we execute the above query as: (1.4)
(Calpurnia AND Brutus) AND Caesar This is a first justification for keeping the frequency of terms in the dictionary: it allows us to make this ordering decision based on in-memory data before accessing any postings list. Consider now the optimization of more general queries, such as:
(madding OR crowd) AND (ignoble OR strife) AND (killed OR slain) As before, we will get the frequencies for all terms, and we can then (conservatively) estimate the size of each OR by the sum of the frequencies of its disjuncts. We can then process the query in increasing order of the size of each disjunctive term. For arbitrary Boolean queries, we have to evaluate and temporarily store the answers for intermediate expressions in a complex expression. However, in many circumstances, either because of the nature of the query language, or just because this is the most common type of query that users submit, a query is purely conjunctive. In this case, rather than viewing merging postings lists as a function with two inputs and a distinct output, it is more efficient to intersect each retrieved postings list with the current intermediate result in memory, where we initialize the intermediate result by loading the postings list of the least frequent term. This algorithm is shown in Figure 1.7. The intersection operation is then asymmetric: the intermediate results list is in memory while the list it is being intersected with is being read from disk. Moreover the intermediate results list is always at least as short as the other list, and in many cases it is orders of magnitude shorter. The postings
intersection can still be done by the algorithm in Figure 1.6, but when the difference between the list lengths is very large, opportunities to use alternative techniques open up. The intersection can be calculated in place by destructively modifying or marking invalid items in the intermediate results list. Or the intersection can be done as a sequence of binary searches in the long postings lists for each posting in the intermediate results list. Another possibility is to store the long postings list as a hashtable, so that membership of an intermediate result item can be calculated in constant rather than linear or log time. However, such alternative techniques are difficult to combine with postings list compression of the sort discussed in Chapter 5. Moreover, standard postings list intersection operations remain necessary when both terms of a query are very common.
Exercise 1.4
For the queries below, can we still run through the intersection in time O( x + y), where x and y are the lengths of the postings lists for Brutus and Caesar? If not, what can we achieve? a. Brutus AND NOT Caesar b. Brutus OR NOT Caesar [ ⋆]
Extend the postings merge algorithm to arbitrary Boolean query formulas. What is its time complexity? For instance, consider: c. (Brutus OR Caesar) AND NOT (Antony OR Cleopatra) Can we always merge in linear time? Linear in what? Can we do better than this? Exercise 1.6
[⋆⋆]
We can use distributive laws for AND and OR to rewrite queries. a. Show how to rewrite the query in Exercise 1.5 into disjunctive normal form using the distributive laws. b. Would the resulting query be more or less efficiently evaluated than the original form of this query? c. Is this result true in general or does it depend on the words and the contents of the document collection? Exercise 1.7 Recommend a query processing order for d. (tangerine OR trees) AND (marmalade OR skies) AND (kaleidoscope OR eyes) given the following postings list sizes:
1 Boolean retrieval Term eyes kaleidoscope marmalade skies tangerine trees
Postings size 213312 87009 107913 271658 46653 316812
If the query is: e. friends AND romans AND (NOT countrymen) how could we use the frequency of countrymen in evaluating the best query evaluation order? In particular, propose a way of handling negation in determining the order of query processing. Exercise 1.9
For a conjunctive query, is processing postings lists in order of size guaranteed to be optimal? Explain why it is, or give an example where it isn't. Exercise 1.10 Write out a postings merge algorithm, in the style of Figure 1.6 (page 11), for an x query.
[⋆⋆] OR
Exercise 1.11 [⋆⋆] How should the Boolean query x AND NOT y be handled? Why is naive evaluation of this query normally very expensive? Write out a postings merge algorithm that evaluates this query efficiently.
1.4 RANKED RETRIEVAL MODEL FREE TEXT QUERIES
PROXIMITY OPERATOR
The extended Boolean model versus ranked retrieval The Boolean retrieval model contrasts with ranked retrieval models such as the vector space model (Section 6.3), in which users largely use free text queries, that is, just typing one or more words rather than using a precise language with operators for building up query expressions, and the system decides which documents best satisfy the query. Despite decades of academic research on the advantages of ranked retrieval, systems implementing the Boolean retrieval model were the main or only search option provided by large commercial information providers for three decades until the early 1990s (approximately the date of arrival of the World Wide Web). However, these systems did not have just the basic Boolean operations (AND, OR, and NOT) which we have presented so far. A strict Boolean expression over terms with an unordered results set is too limited for many of the information needs that people have, and these systems implemented extended Boolean retrieval models by incorporating additional operators such as term proximity operators. A proximity operator is a way of specifying that two terms in a query
1.4 The extended Boolean model versus ranked retrieval
must occur close to each other in a document, where closeness may be measured by limiting the allowed number of intervening words or by reference to a structural unit such as a sentence or paragraph.
Example 1.1: Commercial Boolean searching: Westlaw. Westlaw (http://www.westlaw.com/) is the largest commercial legal search service (in terms of the number of paying subscribers), with over half a million subscribers performing millions of searches a day over tens of terabytes of text data. The service was started in 1975. In 2005, Boolean search (called "Terms and Connectors" by Westlaw) was still the default, and used by a large percentage of users, although ranked free text querying (called "Natural Language" by Westlaw) was added in 1992. Here are some example Boolean queries on Westlaw: Information need: Information on the legal theories involved in preventing the disclosure of trade secrets by employees formerly employed by a competing company. Query: "trade secret" /s disclos! /s prevent /s employe! Information need: Requirements for disabled people to be able to access a workplace. Query: disab! /p access! /s work-site work-place (employment /3 place) Information need: Cases about a host's responsibility for drunk guests. Query: host! /p (responsib! liab!) /p (intoxicat! drunk!) /p guest Note the long, precise queries and the use of proximity operators, both uncommon in web search. Submitted queries average about ten words in length. Unlike web search conventions, a space between words represents disjunction (the tightest binding operator), & is AND and /s, /p, and /k ask for matches in the same sentence, same paragraph or within k words respectively. Double quotes give a phrase search (consecutive words); see Section 2.4 (page 39). The exclamation mark (!) gives a trailing wildcard query (see Section 3.2, page 51); thus liab! matches all words starting with liab. Additionally work-site matches any of worksite, work-site or work site; see Section 2.2.1 (page 22). Typical expert queries are usually carefully defined and incrementally developed until they obtain what look to be good results to the user. Many users, particularly professionals, prefer Boolean query models. Boolean queries are precise: a document either matches the query or it does not. This offers the user greater control and transparency over what is retrieved. And some domains, such as legal materials, allow an effective means of document ranking within a Boolean model: Westlaw returns documents in reverse chronological order, which is in practice quite effective. In 2007, the majority of law librarians still seem to recommend terms and connectors for high recall searches, and the majority of legal users think they are getting greater control by using them. However, this does not mean that Boolean queries are more effective for professional searchers. Indeed, experimenting on a Westlaw subcollection, Turtle (1994) found that free text queries produced better results than Boolean queries prepared by Westlaw's own reference librarians for the majority of the information needs in his experiments. A general problem with Boolean search is that using AND operators tends to produce high precision but low recall searches, while using OR operators gives low precision but high recall searches, and it is difficult or impossible to find a satisfactory middle ground.
In this chapter, we have looked at the structure and construction of a basic
inverted index, comprising a dictionary and postings lists. We introduced the Boolean retrieval model, and examined how to do efficient retrieval via linear time merges and simple query optimization. In Chapters 2–7 we will consider in detail richer query models and the sort of augmented index structures that are needed to handle them efficiently. Here we just mention a few of the main additional things we would like to be able to do: 1. We would like to better determine the set of terms in the dictionary and to provide retrieval that is tolerant to spelling mistakes and inconsistent choice of words. 2. It is often useful to search for compounds or phrases that denote a concept such as "operating system". As the Westlaw examples show, we might also wish to do proximity queries such as Gates NEAR Microsoft. To answer such queries, the index has to be augmented to capture the proximities of terms in documents.
TERM FREQUENCY
3. A Boolean model only records term presence or absence, but often we would like to accumulate evidence, giving more weight to documents that have a term several times as opposed to ones that contain it only once. To be able to do this we need term frequency information (the number of times a term occurs in a document) in postings lists. 4. Boolean queries just retrieve a set of matching documents, but commonly we wish to have an effective method to order (or "rank") the returned results. This requires having a mechanism for determining a document score which encapsulates how good a match a document is for a query. With these additional ideas, we will have seen most of the basic technology that supports ad hoc searching over unstructured information. Ad hoc searching over documents has recently conquered the world, powering not only web search engines but the kind of unstructured search that lies behind the large eCommerce websites. Although the main web search engines differ by emphasizing free text querying, most of the basic issues and technologies of indexing and querying remain the same, as we will see in later chapters. Moreover, over time, web search engines have added at least partial implementations of some of the most popular operators from extended Boolean models: phrase search is especially popular and most have a very partial implementation of Boolean operators. Nevertheless, while these options are liked by expert searchers, they are little used by most people and are not the main focus in work on trying to improve web search engine performance.
Exercise 1.12 [⋆] Write a query using Westlaw syntax which would find any of the words professor, teacher, or lecturer in the same sentence as a form of the verb explain.
1.5 References and further reading
Exercise 1.13
Try using the Boolean search features on a couple of major web search engines. For instance, choose a word, such as burglar, and submit the queries (i) burglar, (ii) burglar AND burglar, and (iii) burglar OR burglar. Look at the estimated number of results and top hits. Do they make sense in terms of Boolean logic? Often they haven't for major search engines. Can you make sense of what is going on? What about if you try different words? For example, query for (i) knight, (ii) conquer, and then (iii) knight OR conquer. What bound should the number of results from the first two queries place on the third query? Is this bound observed?
References and further reading The practical pursuit of computerized information retrieval began in the late 1940s (Cleverdon 1991, Liddy 2005). A great increase in the production of scientific literature, much in the form of less formal technical reports rather than traditional journal articles, coupled with the availability of computers, led to interest in automatic document retrieval. However, in those days, document retrieval was always based on author, title, and keywords; full-text search came much later. The article of Bush (1945) provided lasting inspiration for the new field: "Consider a future device for individual use, which is a sort of mechanized private file and library. It needs a name, and, to coin one at random, 'memex' will do. A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility. It is an enlarged intimate supplement to his memory." The term Information Retrieval was coined by Calvin Mooers in 1948/1950 (Mooers 1950). In 1958, much newspaper attention was paid to demonstrations at a conference (see Taube and Wooster 1958) of IBM "auto-indexing" machines, based primarily on the work of H. P. Luhn. Commercial interest quickly gravitated towards Boolean retrieval systems, but the early years saw a heady debate over various disparate technologies for retrieval systems. For example Mooers (1961) dissented: "It is a common fallacy, underwritten at this date by the investment of several million dollars in a variety of retrieval hardware, that the algebra of George Boole (1847) is the appropriate formalism for retrieval system design. This view is as widely and uncritically accepted as it is wrong." The observation of AND vs. OR giving you opposite extremes in a precision/ recall tradeoff, but not the middle ground comes from (Lee and Fox 1988).
The book (Witten et al. 1999) is the standard reference for an in-depth comparison of the space and time efficiency of the inverted index versus other possible data structures; a more succinct and up-to-date presentation appears in Zobel and Moffat (2006). We further discuss several approaches in Chapter 5. Friedl (2006) covers the practical usage of regular expressions for searching. The underlying computer science appears in (Hopcroft et al. 2000).
The term vocabulary and postings lists
Recall the major steps in inverted index construction: 1. Collect the documents to be indexed. 2. Tokenize the text. 3. Do linguistic preprocessing of tokens. 4. Index the documents that each term occurs in. In this chapter we first briefly mention how the basic unit of a document can be defined and how the character sequence that it comprises is determined (Section 2.1). We then examine in detail some of the substantive linguistic issues of tokenization and linguistic preprocessing, which determine the vocabulary of terms which a system uses (Section 2.2). Tokenization is the process of chopping character streams into tokens, while linguistic preprocessing then deals with building equivalence classes of tokens which are the set of terms that are indexed. Indexing itself is covered in Chapters 1 and 4. Then we return to the implementation of postings lists. In Section 2.3, we examine an extended postings list data structure that supports faster querying, while Section 2.4 covers building postings data structures suitable for handling phrase and proximity queries, of the sort that commonly appear in both extended Boolean models and on the web.
2.1 2.1.1
Document delineation and character sequence decoding Obtaining the character sequence in a document Digital documents that are the input to an indexing process are typically bytes in a file or on a web server. The first step of processing is to convert this byte sequence into a linear sequence of characters. For the case of plain English text in ASCII encoding, this is trivial. But often things get much more
2 The term vocabulary and postings lists
complex. The sequence of characters may be encoded by one of various single byte or multibyte encoding schemes, such as Unicode UTF-8, or various national or vendor-specific standards. We need to determine the correct encoding. This can be regarded as a machine learning classification problem, as discussed in Chapter 13,1 but is often handled by heuristic methods, user selection, or by using provided document metadata. Once the encoding is determined, we decode the byte sequence to a character sequence. We might save the choice of encoding because it gives some evidence about what language the document is written in. The characters may have to be decoded out of some binary representation like Microsoft Word DOC files and/or a compressed format such as zip files. Again, we must determine the document format, and then an appropriate decoder has to be used. Even for plain text documents, additional decoding may need to be done. In XML documents (Section 10.1, page 197), character entities, such as &, need to be decoded to give the correct character, namely & for &. Finally, the textual part of the document may need to be extracted out of other material that will not be processed. This might be the desired handling for XML files, if the markup is going to be ignored; we would almost certainly want to do this with postscript or PDF files. We will not deal further with these issues in this book, and will assume henceforth that our documents are a list of characters. Commercial products usually need to support a broad range of document types and encodings, since users want things to just work with their data as is. Often, they just think of documents as text inside applications and are not even aware of how it is encoded on disk. This problem is usually solved by licensing a software library that handles decoding document formats and character encodings. The idea that text is a linear sequence of characters is also called into question by some writing systems, such as Arabic, where text takes on some two dimensional and mixed order characteristics, as shown in Figures 2.1 and 2.2. But, despite some complicated writing system conventions, there is an underlying sequence of sounds being represented and hence an essentially linear structure remains, and this is what is represented in the digital representation of Arabic, as shown in Figure 2.1.
2.1.2 DOCUMENT UNIT
Choosing a document unit The next phase is to determine what the document unit for indexing is. Thus far we have assumed that documents are fixed units for the purposes of indexing. For example, we take each file in a folder as a document. But there 1. A classifier is a function that takes objects of some sort and assigns them to one of a number of distinct classes (see Chapter 13). Usually classification is done by machine learning methods such as probabilistic models, but it can also be done by hand-written rules.
2.1 Document delineation and character sequence decoding
‫ب‬ ٌ َ ‫⇐ ِآ‬
ٌ ‫ك ِ ت ا ب‬ un b ā t i k /kitābun/ 'a book'
◮ Figure 2.1 An example of a vocalized Modern Standard Arabic word. The writing is from right to left and letters undergo complex mutations as they are combined. The representation of short vowels (here, /i/ and /u/) and the final /n/ (nunation) departs from strict linearity by being represented as diacritics above and below letters. Nevertheless, the represented text is still clearly a linear ordering of characters representing sounds. Full vocalization, as here, normally appears only in the Koran and children's books. Day-to-day text is unvocalized (short vowels are not represented but the letter for a¯ would still appear) or partially vocalized, with short vowels inserted in places where the writer perceives ambiguities. These choices add further complexities to indexing.
. ‫"!" ! ا ل ا‬# 132 1962 ‫ا ا ا‬ ←→ ←→ ← START
'Algeria achieved its independence in 1962 after 132 years of French occupation.' ◮ Figure 2.2 The conceptual linear order of characters is not necessarily the order that you see on the page. In languages that are written right-to-left, such as Hebrew and Arabic, it is quite common to also have left-to-right text interspersed, such as numbers and dollar amounts. With modern Unicode representation concepts, the order of characters in files matches the conceptual order, and the reversal of displayed characters is handled by the rendering system, but this may not be true for documents in older encodings.
INDEXING GRANULARITY
are many cases in which you might want to do something different. A traditional Unix (mbox-format) email file stores a sequence of email messages (an email folder) in one file, but you might wish to regard each email message as a separate document. Many email messages now contain attached documents, and you might then want to regard the email message and each contained attachment as separate documents. If an email message has an attached zip file, you might want to decode the zip file and regard each file it contains as a separate document. Going in the opposite direction, various pieces of web software (such as latex2html) take things that you might regard as a single document (e.g., a Powerpoint file or a LATEX document) and split them into separate HTML pages for each slide or subsection, stored as separate files. In these cases, you might want to combine multiple files into a single document. More generally, for very long documents, the issue of indexing granularity arises. For a collection of books, it would usually be a bad idea to index an
entire book as a document. A search for Chinese toys might bring up a book that mentions China in the first chapter and toys in the last chapter, but this does not make it relevant to the query. Instead, we may well wish to index each chapter or paragraph as a mini-document. Matches are then more likely to be relevant, and since the documents are smaller it will be much easier for the user to find the relevant passages in the document. But why stop there? We could treat individual sentences as mini-documents. It becomes clear that there is a precision/recall tradeoff here. If the units get too small, we are likely to miss important passages because terms were distributed over several mini-documents, while if units are too large we tend to get spurious matches and the relevant information is hard for the user to find. The problems with large document units can be alleviated by use of explicit or implicit proximity search (Sections 2.4.2 and 7.2.2), and the tradeoffs in resulting system performance that we are hinting at are discussed in Chapter 8. The issue of index granularity, and in particular a need to simultaneously index documents at multiple levels of granularity, appears prominently in XML retrieval, and is taken up again in Chapter 10. An IR system should be designed to offer choices of granularity. For this choice to be made well, the person who is deploying the system must have a good understanding of the document collection, the users, and their likely information needs and usage patterns. For now, we will henceforth assume that a suitable size document unit has been chosen, together with an appropriate way of dividing or aggregating files, if needed.
Determining the vocabulary of terms Tokenization Given a character sequence and a defined document unit, tokenization is the task of chopping it up into pieces, called tokens, perhaps at the same time throwing away certain characters, such as punctuation. Here is an example of tokenization: Input: Friends, Romans, Countrymen, lend me your ears; Output: Friends Romans Countrymen lend me your ears
TYPE TERM
These tokens are often loosely referred to as terms or words, but it is sometimes important to make a type/token distinction. A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. A type is the class of all tokens containing the same character sequence. A term is a (perhaps normalized) type that is included in the IR system's dictionary. The set of index terms could be entirely distinct from the tokens, for instance, they could be
2.2 Determining the vocabulary of terms
semantic identifiers in a taxonomy, but in practice in modern IR systems they are strongly related to the tokens in the document. However, rather than being exactly the tokens that appear in the document, they are usually derived from them by various normalization processes which are discussed in Section 2.2.3.2 For example, if the document to be indexed is to sleep perchance to dream, then there are 5 tokens, but only 4 types (since there are 2 instances of to). However, if to is omitted from the index (as a stop word, see Section 2.2.2 (page 27)), then there will be only 3 terms: sleep, perchance, and dream. The major question of the tokenization phase is what are the correct tokens to use? In this example, it looks fairly trivial: you chop on whitespace and throw away punctuation characters. This is a starting point, but even for English there are a number of tricky cases. For example, what do you do about the various uses of the apostrophe for possession and contractions? Mr. O'Neill thinks that the boys' stories about Chile's capital aren't amusing. For O'Neill, which of the following is the desired tokenization? neill oneill o'neill o' neill o neill ? And for aren't, is it: aren't arent are n't aren t ? A simple strategy is to just split on all non-alphanumeric characters, but while o neill looks okay, aren t looks intuitively bad. For all of them, the choices determine which Boolean queries will match. A query of neill AND capital will match in three cases but not the other two. In how many cases would a query of o'neill AND capital match? If no preprocessing of a query is done, then it would match in only one of the five cases. For either 2. That is, as defined here, tokens that are not indexed (stop words) are not terms, and if multiple tokens are collapsed together via normalization, they are indexed as one term, under the normalized form. However, we later relax this definition when discussing classification and clustering in Chapters 13–18, where there is no index. In these chapters, we drop the requirement of inclusion in the dictionary. A term means a normalized word.
LANGUAGE IDENTIFICATION
Boolean or free text queries, you always want to do the exact same tokenization of document and query words, generally by processing queries with the same tokenizer. This guarantees that a sequence of characters in a text will always match the same sequence typed in a query.3 These issues of tokenization are language-specific. It thus requires the language of the document to be known. Language identification based on classifiers that use short character subsequences as features is highly effective; most languages have distinctive signature patterns (see page 46 for references). For most languages and particular domains within them there are unusual specific tokens that we wish to recognize as terms, such as the programming languages C++ and C#, aircraft names like B-52, or a T.V. show name such as M*A*S*H – which is sufficiently integrated into popular culture that you find usages such as M*A*S*H-style hospitals. Computer technology has introduced new types of character sequences that a tokenizer should probably tokenize as a single token, including email addresses ([email protected]), web URLs (http://stuff.big.com/new/specials.html), numeric IP addresses (142.32.48.231), package tracking numbers (1Z9999W99845399981), and more. One possible solution is to omit from indexing tokens such as monetary amounts, numbers, and URLs, since their presence greatly expands the size of the vocabulary. However, this comes at a large cost in restricting what people can search for. For instance, people might want to search in a bug database for the line number where an error occurs. Items such as the date of an email, which have a clear semantic type, are often indexed separately as document metadata (see Section 6.1, page 110). In English, hyphenation is used for various purposes ranging from splitting up vowels in words (co-education) to joining nouns as names (HewlettPackard) to a copyediting device to show word grouping (the hold-him-backand-drag-him-away maneuver). It is easy to feel that the first example should be regarded as one token (and is indeed more commonly written as just coeducation), the last should be separated into words, and that the middle case is unclear. Handling hyphens automatically can thus be complex: it can either be done as a classification problem, or more commonly by some heuristic rules, such as allowing short hyphenated prefixes on words, but not longer hyphenated forms. Conceptually, splitting on white space can also split what should be regarded as a single token. This occurs most commonly with names (San Francisco, Los Angeles) but also with borrowed foreign phrases (au fait) and com3. For the free text case, this is straightforward. The Boolean case is more complex: this tokenization may produce multiple terms from one query word. This can be handled by combining the terms with an AND or as a phrase query (see Section 2.4, page 39). It is harder for a system to handle the opposite case where the user entered as two terms something that was tokenized together in the document processing.
COMPOUND - SPLITTER
WORD SEGMENTATION
pounds that are sometimes written as a single word and sometimes space separated (such as white space vs. whitespace). Other cases with internal spaces that we might wish to regard as a single token include phone numbers ((800) 2342333) and dates (Mar 11, 1983). Splitting tokens on spaces can cause bad retrieval results, for example, if a search for York University mainly returns documents containing New York University. The problems of hyphens and non-separating whitespace can even interact. Advertisements for air fares frequently contain items like San Francisco-Los Angeles, where simply doing whitespace splitting would give unfortunate results. In such cases, issues of tokenization interact with handling phrase queries (which we discuss in Section 2.4 (page 39)), particularly if we would like queries for all of lowercase, lower-case and lower case to return the same results. The last two can be handled by splitting on hyphens and using a phrase index. Getting the first case right would depend on knowing that it is sometimes written as two words and also indexing it in this way. One effective strategy in practice, which is used by some Boolean retrieval systems such as Westlaw and Lexis-Nexis (Example 1.1), is to encourage users to enter hyphens wherever they may be possible, and whenever there is a hyphenated form, the system will generalize the query to cover all three of the one word, hyphenated, and two word forms, so that a query for over-eager will search for over-eager OR "over eager" OR overeager. However, this strategy depends on user training, since if you query using either of the other two forms, you get no generalization. Each new language presents some new issues. For instance, French has a variant use of the apostrophe for a reduced definite article 'the' before a word beginning with a vowel (e.g., l'ensemble) and has some uses of the hyphen with postposed clitic pronouns in imperatives and questions (e.g., donnemoi 'give me'). Getting the first case correct will affect the correct indexing of a fair percentage of nouns and adjectives: you would want documents mentioning both l'ensemble and un ensemble to be indexed under ensemble. Other languages make the problem harder in new ways. German writes compound nouns without spaces (e.g., Computerlinguistik 'computational linguistics'; Lebensversicherungsgesellschaftsangestellter 'life insurance company employee'). Retrieval systems for German greatly benefit from the use of a compound-splitter module, which is usually implemented by seeing if a word can be subdivided into multiple words that appear in a vocabulary. This phenomenon reaches its limit case with major East Asian Languages (e.g., Chinese, Japanese, Korean, and Thai), where text is written without any spaces between words. An example is shown in Figure 2.3. One approach here is to perform word segmentation as prior linguistic processing. Methods of word segmentation vary from having a large vocabulary and taking the longest vocabulary match with some heuristics for unknown words to the use of machine learning sequence models, such as hidden Markov models or conditional random fields, trained over hand-segmented words (see the references
◮ Figure 2.3 The standard unsegmented form of Chinese text using the simplified characters of mainland China. There is no whitespace between words, not even between sentences – the apparent space after the Chinese period (◦ ) is just a typographical illusion caused by placing the character on the left side of its square box. The first sentence is just words in Chinese characters with no spaces between them. The second and third sentences include Arabic numerals and punctuation breaking up the Chinese characters.
◮ Figure 2.4 Ambiguities in Chinese word segmentation. The two characters can be treated as one word meaning 'monk' or as a sequence of two words meaning 'and' and 'still'.
a has to
an he was
and in were
are is will
as it with
at its
be of
for that
◮ Figure 2.5 A stop list of 25 semantically non-selective words which are common in Reuters-RCV1.
in Section 2.5). Since there are multiple possible segmentations of character sequences (see Figure 2.4), all such methods make mistakes sometimes, and so you are never guaranteed a consistent unique tokenization. The other approach is to abandon word-based indexing and to do all indexing via just short subsequences of characters (character k-grams), regardless of whether particular sequences cross word boundaries or not. Three reasons why this approach is appealing are that an individual Chinese character is more like a syllable than a letter and usually has some semantic content, that most words are short (the commonest length is 2 characters), and that, given the lack of standardization of word breaking in the writing system, it is not always clear where word boundaries should be placed anyway. Even in English, some cases of where to put word boundaries are just orthographic conventions – think of notwithstanding vs. not to mention or into vs. on to – but people are educated to write the words with consistent use of spaces.
STOP WORDS COLLECTION FREQUENCY
Dropping common terms: stop words
Sometimes, some extremely common words which would appear to be of little value in helping select documents matching a user need are excluded from the vocabulary entirely. These words are called stop words. The general strategy for determining a stop list is to sort the terms by collection frequency (the total number of times each term appears in the document collection), and then to take the most frequent terms, often hand-filtered for their semantic content relative to the domain of the documents being indexed, as a stop list, the members of which are then discarded during indexing. An example of a stop list is shown in Figure 2.5. Using a stop list significantly reduces the number of postings that a system has to store; we will present some statistics on this in Chapter 5 (see Table 5.1, page 87). And a lot of the time not indexing stop words does little harm: keyword searches with terms like the and by don't seem very useful. However, this is not true for phrase searches. The phrase query "President of the United States", which contains two stop words, is more precise than President AND "United States". The meaning of flights to London is likely to be lost if the word to is stopped out. A search for Vannevar Bush's article As we may think will be difficult if the first three words are stopped out, and the system searches simply for documents containing the word think. Some special query types are disproportionately affected. Some song titles and well known pieces of verse consist entirely of words that are commonly on stop lists (To be or not to be, Let It Be, I don't want to be, . . . ). The general trend in IR systems over time has been from standard use of quite large stop lists (200–300 terms) to very small stop lists (7–12 terms) to no stop list whatsoever. Web search engines generally do not use stop lists. Some of the design of modern IR systems has focused precisely on how we can exploit the statistics of language so as to be able to cope with common words in better ways. We will show in Section 5.3 (page 95) how good compression techniques greatly reduce the cost of storing the postings for common words. Section 6.2.1 (page 117) then discusses how standard term weighting leads to very common words having little impact on document rankings. Finally, Section 7.1.5 (page 140) shows how an IR system with impact-sorted indexes can terminate scanning a postings list early when weights get small, and hence common words do not cause a large additional processing cost for the average query, even though postings lists for stop words are very long. So for most modern IR systems, the additional cost of including stop words is not that big – neither in terms of index size nor in terms of query processing time.
Query term Windows windows window
Terms in documents that should be matched Windows Windows, windows, window window, windows
◮ Figure 2.6 An example of how asymmetric expansion of query terms can usefully model users' expectations.
TOKEN NORMALIZATION EQUIVALENCE CLASSES
Normalization (equivalence classing of terms) Having broken up our documents (and also our query) into tokens, the easy case is if tokens in the query just match tokens in the token list of the document. However, there are many cases when two character sequences are not quite the same but you would like a match to occur. For instance, if you search for USA, you might hope to also match documents containing U.S.A. Token normalization is the process of canonicalizing tokens so that matches occur despite superficial differences in the character sequences of the tokens.4 The most standard way to normalize is to implicitly create equivalence classes, which are normally named after one member of the set. For instance, if the tokens anti-discriminatory and antidiscriminatory are both mapped onto the term antidiscriminatory, in both the document text and queries, then searches for one term will retrieve documents that contain either. The advantage of just using mapping rules that remove characters like hyphens is that the equivalence classing to be done is implicit, rather than being fully calculated in advance: the terms that happen to become identical as the result of these rules are the equivalence classes. It is only easy to write rules of this sort that remove characters. Since the equivalence classes are implicit, it is not obvious when you might want to add characters. For instance, it would be hard to know to turn antidiscriminatory into anti-discriminatory. An alternative to creating equivalence classes is to maintain relations between unnormalized tokens. This method can be extended to hand-constructed lists of synonyms such as car and automobile, a topic we discuss further in Chapter 9. These term relationships can be achieved in two ways. The usual way is to index unnormalized tokens and to maintain a query expansion list of multiple vocabulary entries to consider for a certain query term. A query term is then effectively a disjunction of several postings lists. The alternative is to perform the expansion during index construction. When the document contains automobile, we index it under car as well (and, usually, also vice-versa). Use of either of these methods is considerably less efficient than equivalence classing, as there are more postings to store and merge. The first 4. It is also often referred to as term normalization, but we prefer to reserve the name term for the output of the normalization process.
method adds a query expansion dictionary and requires more processing at query time, while the second method requires more space for storing postings. Traditionally, expanding the space required for the postings lists was seen as more disadvantageous, but with modern storage costs, the increased flexibility that comes from distinct postings lists is appealing. These approaches are more flexible than equivalence classes because the expansion lists can overlap while not being identical. This means there can be an asymmetry in expansion. An example of how such an asymmetry can be exploited is shown in Figure 2.6: if the user enters windows, we wish to allow matches with the capitalized Windows operating system, but this is not plausible if the user enters window, even though it is plausible for this query to also match lowercase windows. The best amount of equivalence classing or query expansion to do is a fairly open question. Doing some definitely seems a good idea. But doing a lot can easily have unexpected consequences of broadening queries in unintended ways. For instance, equivalence-classing U.S.A. and USA to the latter by deleting periods from tokens might at first seem very reasonable, given the prevalent pattern of optional use of periods in acronyms. However, if I put in as my query term C.A.T., I might be rather upset if it matches every appearance of the word cat in documents.5 Below we present some of the forms of normalization that are commonly employed and how they are implemented. In many cases they seem helpful, but they can also do harm. In fact, you can worry about many details of equivalence classing, but it often turns out that providing processing is done consistently to the query and to documents, the fine details may not have much aggregate effect on performance. Accents and diacritics. Diacritics on characters in English have a fairly marginal status, and we might well want cliché and cliche to match, or naive and naïve. This can be done by normalizing tokens to remove diacritics. In many other languages, diacritics are a regular part of the writing system and distinguish different sounds. Occasionally words are distinguished only by their accents. For instance, in Spanish, peña is 'a cliff', while pena is 'sorrow'. Nevertheless, the important question is usually not prescriptive or linguistic but is a question of how users are likely to write queries for these words. In many cases, users will enter queries for words without diacritics, whether for reasons of speed, laziness, limited software, or habits born of the days when it was hard to use non-ASCII text on many computer systems. In these cases, it might be best to equate all words to a form without diacritics. 5. At the time we wrote this chapter (Aug. 2005), this was actually the case on Google: the top result for the query C.A.T. was a site about cats, the Cat Fanciers Web Site http://www.fanciers.com/.
CASE - FOLDING
TRUECASING
Capitalization/case-folding. A common strategy is to do case-folding by reducing all letters to lower case. Often this is a good idea: it will allow instances of Automobile at the beginning of a sentence to match with a query of automobile. It will also help on a web search engine when most of your users type in ferrari when they are interested in a Ferrari car. On the other hand, such case folding can equate words that might better be kept apart. Many proper nouns are derived from common nouns and so are distinguished only by case, including companies (General Motors, The Associated Press), government organizations (the Fed vs. fed) and person names (Bush, Black). We already mentioned an example of unintended query expansion with acronyms, which involved not only acronym normalization (C.A.T. → CAT) but also case-folding (CAT → cat). For English, an alternative to making every token lowercase is to just make some tokens lowercase. The simplest heuristic is to convert to lowercase words at the beginning of a sentence and all words occurring in a title that is all uppercase or in which most or all words are capitalized. These words are usually ordinary words that have been capitalized. Mid-sentence capitalized words are left as capitalized (which is usually correct). This will mostly avoid case-folding in cases where distinctions should be kept apart. The same task can be done more accurately by a machine learning sequence model which uses more features to make the decision of when to case-fold. This is known as truecasing. However, trying to get capitalization right in this way probably doesn't help if your users usually use lowercase regardless of the correct case of words. Thus, lowercasing everything often remains the most practical solution. Other issues in English. Other possible normalizations are quite idiosyncratic and particular to English. For instance, you might wish to equate ne'er and never or the British spelling colour and the American spelling color. Dates, times and similar items come in multiple formats, presenting additional challenges. You might wish to collapse together 3/12/91 and Mar. 12, 1991. However, correct processing here is complicated by the fact that in the U.S., 3/12/91 is Mar. 12, 1991, whereas in Europe it is 3 Dec 1991. Other languages. English has maintained a dominant position on the WWW; approximately 60% of web pages are in English (Gerrand 2007). But that still leaves 40% of the web, and the non-English portion might be expected to grow over time, since less than one third of Internet users and less than 10% of the world's population primarily speak English. And there are signs of change: Sifry (2007) reports that only about one third of blog posts are in English. Other languages again present distinctive issues in equivalence classing.
EMENT →
would map replacement to replac, but not cement to c. The official site for the Porter Stemmer is: http://www.tartarus.org/˜martin/PorterStemmer/
Other stemmers exist, including the older, one-pass Lovins stemmer (Lovins 1968), and newer entrants like the Paice/Husk stemmer (Paice 1990); see: http://www.cs.waikato.ac.nz/˜eibe/stemmers/ http://www.comp.lancs.ac.uk/computing/research/stemming/
Figure 2.8 presents an informal comparison of the different behaviors of these stemmers. Stemmers use language-specific rules, but they require less knowledge than a lemmatizer, which needs a complete vocabulary and morphological analysis to correctly lemmatize words. Particular domains may also require special stemming rules. However, the exact stemmed form does not matter, only the equivalence classes it forms. Rather than using a stemmer, you can use a lemmatizer, a tool from Natural Language Processing which does full morphological analysis to accurately identify the lemma for each word. Doing full morphological analysis produces at most very modest benefits for retrieval. It is hard to say more,
Sample text: Such an analysis can reveal features that are not easily visible from the variations in the individual genes and can lead to a picture of expression that is more biologically transparent and accessible to interpretation Lovins stemmer: such an analys can reve featur that ar not eas vis from th vari in th individu gen and can lead to a pictur of expres that is mor biolog transpar and acces to interpres Porter stemmer: such an analysi can reveal featur that ar not easili visibl from the variat in the individu gene and can lead to a pictur of express that is more biolog transpar and access to interpret Paice stemmer: such an analys can rev feat that are not easy vis from the vary in the individ gen and can lead to a pict of express that is mor biolog transp and access to interpret ◮ Figure 2.8 A comparison of three stemming algorithms on a sample text.
because either form of normalization tends not to improve English information retrieval performance in aggregate – at least not by very much. While it helps a lot for some queries, it equally hurts performance a lot for others. Stemming increases recall while harming precision. As an example of what can go wrong, note that the Porter stemmer stems all of the following words: operate operating operates operation operative operatives operational to oper. However, since operate in its various forms is a common verb, we would expect to lose considerable precision on queries such as the following with Porter stemming: operational AND research operating AND system operative AND dentistry
For a case like this, moving to using a lemmatizer would not completely fix the problem because particular inflectional forms are used in particular collocations: a sentence with the words operate and system is not a good match for the query operating AND system. Getting better value from term normalization depends more on pragmatic issues of word use than on formal issues of linguistic morphology. The situation is different for languages with much more morphology (such as Spanish, German, and Finnish). Results in the European CLEF evaluations have repeatedly shown quite large gains from the use of stemmers (and compound splitting for languages like German); see the references in Section 2.5.
Are the following statements true or false? a. In a Boolean retrieval system, stemming never lowers precision. b. In a Boolean retrieval system, stemming never lowers recall. c. Stemming increases the size of the vocabulary. d. Stemming should be invoked at indexing time but not while processing a query. [ ⋆]
Suggest what normalized form should be used for these words (including the word itself as a possibility): a. 'Cos b. Shi'ite c. cont'd d. Hawai'i e. O'Rourke [ ⋆]
The following pairs of words are stemmed to the same form by the Porter stemmer. Which pairs would you argue shouldn't be conflated. Give your reasoning. a. abandon/abandonment b. absorbency/absorbent c. marketing/markets d. university/universe e. volume/volumes [ ⋆]
Exercise 2.4 For the Porter stemmer rule group shown in (2.1): a. What is the purpose of including an identity rule such as SS → SS?
b. Applying just this rule group, what will the following words be stemmed to? circus canaries
c. What rule should be added to correctly stem pony? d. The stemming for ponies and pony might seem strange. Does it have a deleterious effect on retrieval? Why or why not?
◮ Figure 2.9 Postings lists with skip pointers. The postings intersection can use a skip pointer when the end point is still less than the item on the other list.
SKIP LIST
Faster postings list intersection via skip pointers In the remainder of this chapter, we will discuss extensions to postings list data structures and ways to increase the efficiency of using postings lists. Recall the basic postings list intersection operation from Section 1.3 (page 10): we walk through the two postings lists simultaneously, in time linear in the total number of postings entries. If the list lengths are m and n, the intersection takes O(m + n) operations. Can we do better than this? That is, empirically, can we usually process postings list intersection in sublinear time? We can, if the index isn't changing too fast. One way to do this is to use a skip list by augmenting postings lists with skip pointers (at indexing time), as shown in Figure 2.9. Skip pointers are effectively shortcuts that allow us to avoid processing parts of the postings list that will not figure in the search results. The two questions are then where to place skip pointers and how to do efficient merging using skip pointers. Consider first efficient merging, with Figure 2.9 as an example. Suppose we've stepped through the lists in the figure until we have matched 8 on each list and moved it to the results list. We advance both pointers, giving us 16 on the upper list and 41 on the lower list. The smallest item is then the element 16 on the top list. Rather than simply advancing the upper pointer, we first check the skip list pointer and note that 28 is also less than 41. Hence we can follow the skip list pointer, and then we advance the upper pointer to 28 . We thus avoid stepping to 19 and 23 on the upper list. A number of variant versions of postings list intersection with skip pointers is possible depending on when exactly you check the skip pointer. One version is shown
2.3 Faster postings list intersection via skip pointers
I NTERSECT W ITH S KIPS ( p1 , p2 ) 1 answer ← h i 2 while p1 6= NIL and p2 6= NIL 3 do if docID ( p1) = docID ( p2) 4 then A DD ( answer, docID ( p1 )) 5 p1 ← next( p1 ) 6 p2 ← next( p2 ) 7 else if docID ( p1) < docID ( p2) 8 then if hasSkip( p1 ) and (docID (skip( p1)) ≤ docID ( p2 )) 9 then while hasSkip( p1 ) and (docID (skip( p1)) ≤ docID ( p2)) 10 do p1 ← skip( p1 ) 11 else p1 ← next( p1 ) 12 else if hasSkip( p2 ) and (docID (skip( p2)) ≤ docID ( p1 )) 13 then while hasSkip( p2 ) and (docID (skip( p2)) ≤ docID ( p1)) 14 do p2 ← skip( p2 ) 15 else p2 ← next( p2 ) 16 return answer ◮ Figure 2.10 Postings lists intersection with skip pointers.
in Figure 2.10. Skip pointers will only be available for the original postings lists. For an intermediate result in a complex query, the call hasSkip( p) will always return false. Finally, note that the presence of skip pointers only helps for AND queries, not for OR queries. Where do we place skips? There is a tradeoff. More skips means shorter skip spans, and that we are more likely to skip. But it also means lots of comparisons to skip pointers, and lots of space storing skip pointers. Fewer skips means few pointer comparisons, but then long skip spans which means that there will be fewer opportunities to skip. A simple heuristic for placing skips, which has been √ found to work well in practice, is that for a postings list of length P, use P evenly-spaced skip pointers. This heuristic can be improved upon; it ignores any details of the distribution of query terms. Building effective skip pointers is easy if an index is relatively static; it is harder if a postings list keeps changing because of updates. A malicious deletion strategy can render skip lists ineffective. Choosing the optimal encoding for an inverted index is an ever-changing game for the system builder, because it is strongly dependent on underlying computer technologies and their relative speeds and sizes. Traditionally, CPUs were slow, and so highly compressed techniques were not optimal. Now CPUs are fast and disk is slow, so reducing disk postings list size dominates. However, if you're running a search engine with everything in mem-
ory then the equation changes again. We discuss the impact of hardware parameters on index construction time in Section 4.1 (page 68) and the impact of index size on system speed in Chapter 5.
Exercise 2.5 Why are skip pointers not useful for queries of the form x
y? [⋆]
We have a two-word query. For one term the postings list consists of the following 16 entries: [4,6,10,12,14,16,18,20,22,32,47,81,120,122,157,180] and for the other it is the one entry postings list: [47]. Work out how many comparisons would be done to intersect the two postings lists with the following two strategies. Briefly justify your answers: a. Using standard postings lists b. Using postings lists stored with skip pointers, with a skip length of gested in Section 2.3.
P, as sug[⋆]
Exercise 2.7 Consider a postings intersection between this postings list, with skip pointers:
15 24 39 60 68 75 81 84 89 92 96 97 100 115
and the following intermediate result postings list (which hence has no skip pointers): 3
Trace through the postings intersection algorithm in Figure 2.10 (page 37). a. How often is a skip pointer followed (i.e., p1 is advanced to skip( p1 ))? b. How many postings comparisons will be made by this algorithm while intersecting the two lists? c. How many postings comparisons would be made if the postings lists are intersected without the use of skip pointers?
2.4 Positional postings and phrase queries
PHRASE QUERIES
BIWORD INDEX
Positional postings and phrase queries Many complex or technical concepts and many organization and product names are multiword compounds or phrases. We would like to be able to pose a query such as Stanford University by treating it as a phrase so that a sentence in a document like The inventor Stanford Ovshinsky never went to university. is not a match. Most recent search engines support a double quotes syntax ("stanford university") for phrase queries, which has proven to be very easily understood and successfully used by users. As many as 10% of web queries are phrase queries, and many more are implicit phrase queries (such as person names), entered without use of double quotes. To be able to support such queries, it is no longer sufficient for postings lists to be simply lists of documents that contain individual terms. In this section we consider two approaches to supporting phrase queries and their combination. A search engine should not only support phrase queries, but implement them efficiently. A related but distinct concept is term proximity weighting, where a document is preferred to the extent that the query terms appear close to each other in the text. This technique is covered in Section 7.2.2 (page 144) in the context of ranked retrieval.
Biword indexes One approach to handling phrases is to consider every pair of consecutive terms in a document as a phrase. For example, the text Friends, Romans, Countrymen would generate the biwords: friends romans romans countrymen
In this model, we treat each of these biwords as a vocabulary term. Being able to process two-word phrase queries is immediate. Longer phrases can be processed by breaking them down. The query stanford university palo alto can be broken into the Boolean query on biwords: "stanford university"
"university palo"
This query could be expected to work fairly well in practice, but there can and will be occasional false positives. Without examining the documents, we cannot verify that the documents matching the above Boolean query do actually contain the original 4 word phrase. Among possible queries, nouns and noun phrases have a special status in describing the concepts people are interested in searching for. But related nouns can often be divided from each other by various function words, in phrases such as the abolition of slavery or renegotiation of the constitution. These needs can be incorporated into the biword indexing model in the following
way. First, we tokenize the text and perform part-of-speech-tagging.6 We can then group terms into nouns, including proper nouns, (N) and function words, including articles and prepositions, (X), among other classes. Now deem any string of terms of the form NX*N to be an extended biword. Each such extended biword is made a term in the vocabulary. For example: renegotiation N
of X
constitution N
To process a query using such an extended biword index, we need to also parse it into N's and X's, and then segment the query into extended biwords, which can be looked up in the index. This algorithm does not always work in an intuitively optimal manner when parsing longer queries into Boolean queries. Using the above algorithm, the query cost overruns on a power plant
is parsed into "cost overruns"
PHRASE INDEX
"overruns power"
whereas it might seem a better query to omit the middle biword. Better results can be obtained by using more precise part-of-speech patterns that define which extended biwords should be indexed. The concept of a biword index can be extended to longer sequences of words, and if the index includes variable length word sequences, it is generally referred to as a phrase index. Indeed, searches for a single term are not naturally handled in a biword index (you would need to scan the dictionary for all biwords containing the term), and so we also need to have an index of single-word terms. While there is always a chance of false positive matches, the chance of a false positive match on indexed phrases of length 3 or more becomes very small indeed. But on the other hand, storing longer phrases has the potential to greatly expand the vocabulary size. Maintaining exhaustive phrase indexes for phrases of length greater than two is a daunting prospect, and even use of an exhaustive biword dictionary greatly expands the size of the vocabulary. However, towards the end of this section we discuss the utility of the strategy of using a partial phrase index in a compound indexing scheme. 6. Part of speech taggers classify words as nouns, verbs, etc. – or, in practice, often as finergrained classes like "plural proper noun". Many fairly accurate (c. 96% per-tag accuracy) partof-speech taggers now exist, usually trained by machine learning methods on hand-tagged text. See, for instance, Manning and Schütze (1999, ch. 10).
to, 993427:
h 1, 6: 2, 5: 4, 5: 5, 2: 7, 3:
h7, 18, 33, 72, 86, 231i; h1, 17, 74, 222, 255i; h8, 16, 190, 429, 433i; h363, 367i; h13, 23, 191i; . . . i
be, 178239:
h 1, 2: h17, 25i; 4, 5: h17, 191, 291, 430, 434i; 5, 3: h14, 19, 101i; . . . i ◮ Figure 2.11 Positional index example. The word to has a document frequency 993,477, and occurs 6 times in document 1 at positions 7, 18, 33, etc.
2.4.2 POSITIONAL INDEX
Positional indexes For the reasons given, a biword index is not the standard solution. Rather, a positional index is most commonly employed. Here, for each term in the vocabulary, we store postings of the form docID: hposition1, position2, . . . i, as shown in Figure 2.11, where each position is a token index in the document. Each posting will also usually record the term frequency, for reasons discussed in Chapter 6. To process a phrase query, you still need to access the inverted index entries for each distinct term. As before, you would start with the least frequent term and then work to further restrict the list of possible candidates. In the merge operation, the same general technique is used as before, but rather than simply checking that both terms are in a document, you also need to check that their positions of appearance in the document are compatible with the phrase query being evaluated. This requires working out offsets between the words. Example 2.1: Satisfying phrase queries. Suppose the postings lists for to and be are as in Figure 2.11, and the query is "to be or not to be". The postings lists to access are: to, be, or, not. We will examine intersecting the postings lists for to and be. We
first look for documents that contain both terms. Then, we look for places in the lists where there is an occurrence of be with a token index one higher than a position of to, and then we look for another occurrence of each word with token index 4 higher than the first occurrence. In the above lists, the pattern of occurrences that is a possible match is: to: h. . . ; 4:h. . . ,429,433i; . . . i be: h. . . ; 4:h. . . ,430,434i; . . . i
P OSITIONAL I NTERSECT ( p1 , p2, k) 1 answer ← h i 2 while p1 6= NIL and p2 6= NIL 3 do if docID ( p1) = docID ( p2) 4 then l ← h i 5 pp1 ← positions( p1 ) 6 pp2 ← positions( p2 ) 7 while pp1 6= NIL 8 do while pp2 6= NIL 9 do if | pos( pp1) − pos( pp2)| ≤ k 10 then A DD (l, pos( pp2)) 11 else if pos( pp2) > pos( pp1) 12 then break 13 pp2 ← next( pp2 ) 14 while l 6= h i and |l [0] − pos( pp1)| > k 15 do D ELETE(l [0]) 16 for each ps ∈ l 17 do A DD ( answer, hdocID ( p1 ), pos( pp1), psi) 18 pp1 ← next( pp1 ) 19 p1 ← next( p1 ) 20 p2 ← next( p2 ) 21 else if docID ( p1) < docID ( p2) 22 then p1 ← next( p1 ) 23 else p2 ← next( p2 ) 24 return answer ◮ Figure 2.12 An algorithm for proximity intersection of postings lists p1 and p2 . The algorithm finds places where the two terms appear within k words of each other and returns a list of triples giving docID and the term position in p1 and p2 .
The same general method is applied for within k word proximity searches, of the sort we saw in Example 1.1 (page 15): employment /3 place
Here, /k means "within k words of (on either side)". Clearly, positional indexes can be used for such queries; biword indexes cannot. We show in Figure 2.12 an algorithm for satisfying within k word proximity searches; it is further discussed in Exercise 2.12. Positional index size. Adopting a positional index expands required postings storage significantly, even if we compress position values/offsets as we
will discuss in Section 5.3 (page 95). Indeed, moving to a positional index also changes the asymptotic complexity of a postings intersection operation, because the number of items to check is now bounded not by the number of documents but by the total number of tokens in the document collection T. That is, the complexity of a Boolean query is Θ( T ) rather than Θ( N ). However, most applications have little choice but to accept this, since most users now expect to have the functionality of phrase and proximity searches. Let's examine the space implications of having a positional index. A posting now needs an entry for each occurrence of a term. The index size thus depends on the average document size. The average web page has less than 1000 terms, but documents like SEC stock filings, books, and even some epic poems easily reach 100,000 terms. Consider a term with frequency 1 in 1000 terms on average. The result is that large documents cause an increase of two orders of magnitude in the space required to store the postings list: Document size 1000 100,000
Expected postings 1 1
Expected entries in positional posting 1 100
While the exact numbers depend on the type of documents and the language being indexed, some rough rules of thumb are to expect a positional index to be 2 to 4 times as large as a non-positional index, and to expect a compressed positional index to be about one third to one half the size of the raw text (after removal of markup, etc.) of the original uncompressed documents. Specific numbers for an example collection are given in Table 5.1 (page 87) and Table 5.6 (page 103).
Combination schemes The strategies of biword indexes and positional indexes can be fruitfully combined. If users commonly query on particular phrases, such as Michael Jackson, it is quite inefficient to keep merging positional postings lists. A combination strategy uses a phrase index, or just a biword index, for certain queries and uses a positional index for other phrase queries. Good queries to include in the phrase index are ones known to be common based on recent querying behavior. But this is not the only criterion: the most expensive phrase queries to evaluate are ones where the individual words are common but the desired phrase is comparatively rare. Adding Britney Spears as a phrase index entry may only give a speedup factor to that query of about 3, since most documents that mention either word are valid results, whereas adding The Who as a phrase index entry may speed up that query by a factor of 1000. Hence, having the latter is more desirable, even if it is a relatively less common query.
NEXT WORD INDEX
Williams et al. (2004) evaluate an even more sophisticated scheme which employs indexes of both these sorts and additionally a partial next word index as a halfway house between the first two strategies. For each term, a next word index records terms that follow it in a document. They conclude that such a strategy allows a typical mixture of web phrase queries to be completed in one quarter of the time taken by use of a positional index alone, while taking up 26% more space than use of a positional index alone. Exercise 2.8
Assume a biword index. Give an example of a document which will be returned for a query of New York University but is actually a false positive which should not be returned. Exercise 2.9
Shown below is a portion of a positional index in the format: term: doc1: hposition1, position2, . . . i; doc2: hposition1, position2, . . . i; etc. angels: 2: h36,174,252,651i; 4: h12,22,102,432i; 7: h17i; fools: 2: h1,17,74,222i; 4: h8,78,108,458i; 7: h3,13,23,193i; fear: 2: h87,704,722,901i; 4: h13,43,113,433i; 7: h18,328,528i; in: 2: h3,37,76,444,851i; 4: h10,20,110,470,500i; 7: h5,15,25,195i; rush: 2: h2,66,194,321,702i; 4: h9,69,149,429,569i; 7: h4,14,404i; to: 2: h47,86,234,999i; 4: h14,24,774,944i; 7: h199,319,599,709i; tread: 2: h57,94,333i; 4: h15,35,155i; 7: h20,320i; where: 2: h67,124,393,1001i; 4: h11,41,101,421,431i; 7: h16,36,736i;
Which document(s) if any match each of the following queries, where each expression within quotes is a phrase query? a. "fools rush in" b. "fools rush in" AND "angels fear to tread" Exercise 2.10
Consider the following fragment of a positional index with the format: word: document: hposition, position, . . . i; document: hposition, . . . i ... Gates: 1: h3i; 2: h6i; 3: h2,17i; 4: h1i; IBM: 4: h3i; 7: h14i; Microsoft: 1: h1i; 2: h1,21i; 3: h3i; 5: h16,22,51i;
The /k operator, word1 /k word2 finds occurrences of word1 within k words of word2 (on either side), where k is a positive integer argument. Thus k = 1 demands that word1 be adjacent to word2. a. Describe the set of documents that satisfy the query Gates /2 Microsoft. b. Describe each set of values for k for which the query Gates /k Microsoft returns a different set of documents as the answer.
45 [⋆⋆]
Consider the general procedure for merging two positional postings lists for a given document, to determine the document positions where a document satisfies a /k clause (in general there can be multiple positions at which each term occurs in a single document). We begin with a pointer to the position of occurrence of each term and move each pointer along the list of occurrences in the document, checking as we do so whether we have a hit for /k. Each move of either pointer counts as a step. Let L denote the total number of occurrences of the two terms in the document. What is the big-O complexity of the merge procedure, if we wish to have postings including positions in the result? Exercise 2.12
Consider the adaptation of the basic algorithm for intersection of two postings lists (Figure 1.6, page 11) to the one in Figure 2.12 (page 42), which handles proximity queries. A naive algorithm for this operation could be O( PLmax 2 ), where P is the sum of the lengths of the postings lists (i.e., the sum of document frequencies) and Lmax is the maximum length of a document (in tokens). a. Go through this algorithm carefully and explain how it works. b. What is the complexity of this algorithm? Justify your answer carefully. c. For certain queries and data distributions, would another algorithm be more efficient? What complexity does it have? Exercise 2.13
Suppose we wish to use a postings intersection procedure to determine simply the list of documents that satisfy a /k clause, rather than returning the list of positions, as in Figure 2.12 (page 42). For simplicity, assume k ≥ 2. Let L denote the total number of occurrences of the two terms in the document collection (i.e., the sum of their collection frequencies). Which of the following is true? Justify your answer. a. The merge can be accomplished in a number of steps linear in L and independent of k, and we can ensure that each pointer moves only to the right. b. The merge can be accomplished in a number of steps linear in L and independent of k, but a pointer may be forced to move non-monotonically (i.e., to sometimes back up) c. The merge can require kL steps in some cases. Exercise 2.14
How could an IR system combine use of a positional index and use of stop words? What is the potential problem, and how could it be handled?
2.5 E AST A SIAN LANGUAGES
References and further reading Exhaustive discussion of the character-level processing of East Asian languages can be found in Lunde (1998). Character bigram indexes are perhaps the most standard approach to indexing Chinese, although some systems use word segmentation. Due to differences in the language and writing system, word segmentation is most usual for Japanese (Luk and Kwok 2002, Kishida
et al. 2005). The structure of a character k-gram index over unsegmented text differs from that in Section 3.2.2 (page 54): there the k-gram dictionary points to postings lists of entries in the regular dictionary, whereas here it points directly to document postings lists. For further discussion of Chinese word segmentation, see Sproat et al. (1996), Sproat and Emerson (2003), Tseng et al. (2005), and Gao et al. (2005). Lita et al. (2003) present a method for truecasing. Natural language processing work on computational morphology is presented in (Sproat 1992, Beesley and Karttunen 2003). Language identification was perhaps first explored in cryptography; for example, Konheim (1981) presents a character-level k-gram language identification algorithm. While other methods such as looking for particular distinctive function words and letter combinations have been used, with the advent of widespread digital text, many people have explored the character n-gram technique, and found it to be highly successful (Beesley 1998, Dunning 1994, Cavnar and Trenkle 1994). Written language identification is regarded as a fairly easy problem, while spoken language identification remains more difficult; see Hughes et al. (2006) for a recent survey. Experiments on and discussion of the positive and negative impact of stemming in English can be found in the following works: Salton (1989), Harman (1991), Krovetz (1995), Hull (1996). Hollink et al. (2004) provide detailed results for the effectiveness of language-specific methods on 8 European languages. In terms of percent change in mean average precision (see page 159) over a baseline system, diacritic removal gains up to 23% (being especially helpful for Finnish, French, and Swedish). Stemming helped markedly for Finnish (30% improvement) and Spanish (10% improvement), but for most languages, including English, the gain from stemming was in the range 0– 5%, and results from a lemmatizer were poorer still. Compound splitting gained 25% for Swedish and 15% for German, but only 4% for Dutch. Rather than language-particular methods, indexing character k-grams (as we suggested for Chinese) could often give as good or better results: using withinword character 4-grams rather than words gave gains of 37% in Finnish, 27% in Swedish, and 20% in German, while even being slightly positive for other languages, such as Dutch, Spanish, and English. Tomlinson (2003) presents broadly similar results. Bar-Ilan and Gutman (2005) suggest that, at the time of their study (2003), the major commercial web search engines suffered from lacking decent language-particular processing; for example, a query on www.google.fr for l'électricité did not separate off the article l' but only matched pages with precisely this string of article+noun. The classic presentation of skip pointers for IR can be found in Moffat and Zobel (1996). Extended techniques are discussed in Boldi and Vigna (2005). The main paper in the algorithms literature is Pugh (1990), which uses multilevel skip pointers to give expected O(log P) list access (the same expected
efficiency as using a tree data structure) with less implementational complexity. In practice, the effectiveness of using skip pointers depends on various system parameters. Moffat and Zobel (1996) report conjunctive queries running about five times faster with the use of skip pointers, but Bahle et al. (2002, p. 217) report that, with modern CPUs, using skip lists instead slows down search because it expands the size of the postings list (i.e., disk I/O dominates performance). In contrast, Strohman and Croft (2007) again show good performance gains from skipping, in a system architecture designed to optimize for the large memory spaces and multiple cores of recent CPUs. Johnson et al. (2006) report that 11.7% of all queries in two 2002 web query logs contained phrase queries, though Kammenhuber et al. (2006) report only 3% phrase queries for a different data set. Silverstein et al. (1999) note that many queries without explicit phrase operators are actually implicit phrase searches.
WILDCARD QUERY
Dictionaries and tolerant retrieval
In Chapters 1 and 2 we developed the ideas underlying inverted indexes for handling Boolean and proximity queries. Here, we develop techniques that are robust to typographical errors in the query, as well as alternative spellings. In Section 3.1 we develop data structures that help the search for terms in the vocabulary in an inverted index. In Section 3.2 we study the idea of a wildcard query: a query such as *a*e*i*o*u*, which seeks documents containing any term that includes all the five vowels in sequence. The * symbol indicates any (possibly empty) string of characters. Users pose such queries to a search engine when they are uncertain about how to spell a query term, or seek documents containing variants of a query term; for instance, the query automat* would seek documents containing any of the terms automatic, automation and automated. We then turn to other forms of imprecisely posed queries, focusing on spelling errors in Section 3.3. Users make spelling errors either by accident, or because the term they are searching for (e.g., Herman) has no unambiguous spelling in the collection. We detail a number of techniques for correcting spelling errors in queries, one term at a time as well as for an entire string of query terms. Finally, in Section 3.4 we study a method for seeking vocabulary terms that are phonetically close to the query term(s). This can be especially useful in cases like the Herman example, where the user may not know how a proper name is spelled in documents in the collection. Because we will develop many variants of inverted indexes in this chapter, we will use sometimes the phrase standard inverted index to mean the inverted index developed in Chapters 1 and 2, in which each vocabulary term has a postings list with the documents in the collection.
Search structures for dictionaries Given an inverted index and a query, our first task is to determine whether each query term exists in the vocabulary and if so, identify the pointer to the
3 Dictionaries and tolerant retrieval
B- TREE
corresponding postings. This vocabulary lookup operation uses a classical data structure called the dictionary and has two broad classes of solutions: hashing, and search trees. In the literature of data structures, the entries in the vocabulary (in our case, terms) are often referred to as keys. The choice of solution (hashing, or search trees) is governed by a number of questions: (1) How many keys are we likely to have? (2) Is the number likely to remain static, or change a lot – and in the case of changes, are we likely to only have new keys inserted, or to also have some keys in the dictionary be deleted? (3) What are the relative frequencies with which various keys will be accessed? Hashing has been used for dictionary lookup in some search engines. Each vocabulary term (key) is hashed into an integer over a large enough space that hash collisions are unlikely; collisions if any are resolved by auxiliary structures that can demand care to maintain.1 At query time, we hash each query term separately and following a pointer to the corresponding postings, taking into account any logic for resolving hash collisions. There is no easy way to find minor variants of a query term (such as the accented and non-accented versions of a word like resume), since these could be hashed to very different integers. In particular, we cannot seek (for instance) all terms beginning with the prefix automat, an operation that we will require below in Section 3.2. Finally, in a setting (such as the Web) where the size of the vocabulary keeps growing, a hash function designed for current needs may not suffice in a few years' time. Search trees overcome many of these issues – for instance, they permit us to enumerate all vocabulary terms beginning with automat. The best-known search tree is the binary tree, in which each internal node has two children. The search for a term begins at the root of the tree. Each internal node (including the root) represents a binary test, based on whose outcome the search proceeds to one of the two sub-trees below that node. Figure 3.1 gives an example of a binary search tree used for a dictionary. Efficient search (with a number of comparisons that is O(log M )) hinges on the tree being balanced: the numbers of terms under the two sub-trees of any node are either equal or differ by one. The principal issue here is that of rebalancing: as terms are inserted into or deleted from the binary search tree, it needs to be rebalanced so that the balance property is maintained. To mitigate rebalancing, one approach is to allow the number of sub-trees under an internal node to vary in a fixed interval. A search tree commonly used for a dictionary is the B-tree – a search tree in which every internal node has a number of children in the interval [ a, b], where a and b are appropriate positive integers; Figure 3.2 shows an example with a = 2 and b = 4. Each branch under an internal node again represents a test for a range of char1. So-called perfect hash functions are designed to preclude collisions, but are rather more complicated both to implement and to compute.
3.2 Wildcard queries
◮ Figure 3.1 A binary search tree. In this example the branch at the root partitions vocabulary terms into two subtrees, those whose first letter is between a and m, and the rest.
acter sequences, as in the binary tree example of Figure 3.1. A B-tree may be viewed as "collapsing" multiple levels of the binary tree into one; this is especially advantageous when some of the dictionary is disk-resident, in which case this collapsing serves the function of pre-fetching imminent binary tests. In such cases, the integers a and b are determined by the sizes of disk blocks. Section 3.5 contains pointers to further background on search trees and B-trees. It should be noted that unlike hashing, search trees demand that the characters used in the document collection have a prescribed ordering; for instance, the 26 letters of the English alphabet are always listed in the specific order A through Z. Some Asian languages such as Chinese do not always have a unique ordering, although by now all languages (including Chinese and Japanese) have adopted a standard ordering system for their character sets.
Wildcard queries Wildcard queries are used in any of the following situations: (1) the user is uncertain of the spelling of a query term (e.g., Sydney vs. Sidney, which
◮ Figure 3.2 A B-tree. In this example every internal node has between 2 and 4 children.
leads to the wildcard query S*dney); (2) the user is aware of multiple variants of spelling a term and (consciously) seeks documents containing any of the variants (e.g., color vs. colour); (3) the user seeks documents containing variants of a term that would be caught by stemming, but is unsure whether the search engine performs stemming (e.g., judicial vs. judiciary, leading to the wildcard query judicia*); (4) the user is uncertain of the correct rendition of a foreign word or phrase (e.g., the query Universit* Stuttgart). A query such as mon* is known as a trailing wildcard query, because the * symbol occurs only once, at the end of the search string. A search tree on the dictionary is a convenient way of handling trailing wildcard queries: we walk down the tree following the symbols m, o and n in turn, at which point we can enumerate the set W of terms in the dictionary with the prefix mon. Finally, we use |W | lookups on the standard inverted index to retrieve all documents containing any term in W. But what about wildcard queries in which the * symbol is not constrained to be at the end of the search string? Before handling this general case, we mention a slight generalization of trailing wildcard queries. First, consider leading wildcard queries, or queries of the form *mon. Consider a reverse B-tree on the dictionary – one in which each root-to-leaf path of the B-tree corresponds to a term in the dictionary written backwards: thus, the term lemon would, in the B-tree, be represented by the path root-n-o-m-e-l. A walk down the reverse B-tree then enumerates all terms R in the vocabulary with a given prefix. In fact, using a regular B-tree together with a reverse B-tree, we can handle an even more general case: wildcard queries in which there is a single * symbol, such as se*mon. To do this, we use the regular B-tree to enumerate the set W of dictionary terms beginning with the prefix se, then the reverse B-tree to
enumerate the set R of terms ending with the suffix mon. Next, we take the intersection W ∩ R of these two sets, to arrive at the set of terms that begin with the prefix se and end with the suffix mon. Finally, we use the standard inverted index to retrieve all documents containing any terms in this intersection. We can thus handle wildcard queries that contain a single * symbol using two B-trees, the normal B-tree and a reverse B-tree.
General wildcard queries We now study two techniques for handling general wildcard queries. Both techniques share a common strategy: express the given wildcard query qw as a Boolean query Q on a specially constructed index, such that the answer to Q is a superset of the set of vocabulary terms matching qw . Then, we check each term in the answer to Q against qw , discarding those vocabulary terms that do not match qw . At this point we have the vocabulary terms matching qw and can resort to the standard inverted index. Permuterm indexes
PERMUTERM INDEX
Our first special index for general wildcard queries is the permuterm index, a form of inverted index. First, we introduce a special symbol $ into our character set, to mark the end of a term. Thus, the term hello is shown here as the augmented term hello$. Next, we construct a permuterm index, in which the various rotations of each term (augmented with $) all link to the original vocabulary term. Figure 3.3 gives an example of such a permuterm index entry for the term hello. We refer to the set of rotated terms in the permuterm index as the permuterm vocabulary. How does this index help us with wildcard queries? Consider the wildcard query m*n. The key is to rotate such a wildcard query so that the * symbol appears at the end of the string – thus the rotated wildcard query becomes n$m*. Next, we look up this string in the permuterm index, where seeking n$m* (via a search tree) leads to rotations of (among others) the terms man and moron. Now that the permuterm index enables us to identify the original vocabulary terms matching a wildcard query, we look up these terms in the standard inverted index to retrieve matching documents. We can thus handle any wildcard query with a single * symbol. But what about a query such as fi*mo*er? In this case we first enumerate the terms in the dictionary that are in the permuterm index of er$fi*. Not all such dictionary terms will have the string mo in the middle - we filter these out by exhaustive enumeration, checking each candidate to see if it contains mo. In this example, the term fishmonger would survive this filtering but filibuster would not. We then
◮ Figure 3.3 A portion of a permuterm index.
run the surviving terms through the standard inverted index for document retrieval. One disadvantage of the permuterm index is that its dictionary becomes quite large, including as it does all rotations of each term. Notice the close interplay between the B-tree and the permuterm index above. Indeed, it suggests that the structure should perhaps be viewed as a permuterm B-tree. However, we follow traditional terminology here in describing the permuterm index as distinct from the B-tree that allows us to select the rotations with a given prefix.
k- GRAM INDEX
k-gram indexes for wildcard queries Whereas the permuterm index is simple, it can lead to a considerable blowup from the number of rotations per term; for a dictionary of English terms, this can represent an almost ten-fold space increase. We now present a second technique, known as the k-gram index, for processing wildcard queries. We will also use k-gram indexes in Section 3.3.4. A k-gram is a sequence of k characters. Thus cas, ast and stl are all 3-grams occurring in the term castle. We use a special character $ to denote the beginning or end of a term, so the full set of 3-grams generated for castle is: $ca, cas, ast, stl, tle, le$. In a k-gram index, the dictionary contains all k-grams that occur in any term in the vocabulary. Each postings list points from a k-gram to all vocabulary
- beetroot
petrify
- retrieval
◮ Figure 3.4 Example of a postings list in a 3-gram index. Here the 3-gram etr is illustrated. Matching vocabulary terms are lexicographically ordered in the postings.
terms containing that k-gram. For instance, the 3-gram etr would point to vocabulary terms such as metric and retrieval. An example is given in Figure 3.4. How does such an index help us with wildcard queries? Consider the wildcard query re*ve. We are seeking documents containing any term that begins with re and ends with ve. Accordingly, we run the Boolean query $re AND ve$. This is looked up in the 3-gram index and yields a list of matching terms such as relive, remove and retrieve. Each of these matching terms is then looked up in the standard inverted index to yield documents matching the query. There is however a difficulty with the use of k-gram indexes, that demands one further step of processing. Consider using the 3-gram index described above for the query red*. Following the process described above, we first issue the Boolean query $re AND red to the 3-gram index. This leads to a match on terms such as retired, which contain the conjunction of the two 3grams $re and red, yet do not match the original wildcard query red*. To cope with this, we introduce a post-filtering step, in which the terms enumerated by the Boolean query on the 3-gram index are checked individually against the original query red*. This is a simple string-matching operation and weeds out terms such as retired that do not match the original query. Terms that survive are then searched in the standard inverted index as usual. We have seen that a wildcard query can result in multiple terms being enumerated, each of which becomes a single-term query on the standard inverted index. Search engines do allow the combination of wildcard queries using Boolean operators, for example, re*d AND fe*ri. What is the appropriate semantics for such a query? Since each wildcard query turns into a disjunction of single-term queries, the appropriate interpretation of this example is that we have a conjunction of disjunctions: we seek all documents that contain any term matching re*d and any term matching fe*ri. Even without Boolean combinations of wildcard queries, the processing of a wildcard query can be quite expensive, because of the added lookup in the special index, filtering and finally the standard inverted index. A search engine may support such rich functionality, but most commonly, the capability is hidden behind an interface (say an "Advanced Query" interface) that most
users never use. Exposing such functionality in the search interface often encourages users to invoke it even when they do not require it (say, by typing a prefix of their query followed by a *), increasing the processing load on the search engine.
Exercise 3.1 In the permuterm index, each permuterm vocabulary term points to the original vocabulary term(s) from which it was derived. How many original vocabulary terms can there be in the postings list of a permuterm vocabulary term? Exercise 3.2 Write down the entries in the permuterm index dictionary that are generated by the term mama. Exercise 3.3 If you wanted to search for s*ng in a permuterm wildcard index, what key(s) would one do the lookup on? Exercise 3.4 Refer to Figure 3.4; it is pointed out in the caption that the vocabulary terms in the postings are lexicographically ordered. Why is this ordering useful? Exercise 3.5 Consider again the query fi*mo*er from Section 3.2.1. What Boolean query on a bigram index would be generated for this query? Can you think of a term that matches the permuterm query in Section 3.2.1, but does not satisfy this Boolean query? Exercise 3.6 Give an example of a sentence that falsely matches the wildcard query mon*h if the search were to simply use a conjunction of bigrams.
Spelling correction We next look at the problem of correcting spelling errors in queries. For instance, we may wish to retrieve documents containing the term carrot when the user types the query carot. Google reports (http://www.google.com/jobs/britney.html) that the following are all treated as misspellings of the query britney spears: britian spears, britney's spears, brandy spears and prittany spears. We look at two steps to solving this problem: the first based on edit distance and the second based on k-gram overlap. Before getting into the algorithmic details of these methods, we first review how search engines provide spell-correction as part of a user experience.
3.3 Spelling correction
Implementing spelling correction There are two basic principles underlying most spelling correction algorithms. 1. Of various alternative correct spellings for a mis-spelled query, choose the "nearest" one. This demands that we have a notion of nearness or proximity between a pair of queries. We will develop these proximity measures in Section 3.3.3. 2. When two correctly spelled queries are tied (or nearly tied), select the one that is more common. For instance, grunt and grant both seem equally plausible as corrections for grnt. Then, the algorithm should choose the more common of grunt and grant as the correction. The simplest notion of more common is to consider the number of occurrences of the term in the collection; thus if grunt occurs more often than grant, it would be the chosen correction. A different notion of more common is employed in many search engines, especially on the web. The idea is to use the correction that is most common among queries typed in by other users. The idea here is that if grunt is typed as a query more often than grant, then it is more likely that the user who typed grnt intended to type the query grunt. Beginning in Section 3.3.3 we describe notions of proximity between queries, as well as their efficient computation. Spelling correction algorithms build on these computations of proximity; their functionality is then exposed to users in one of several ways: 1. On the query carot always retrieve documents containing carot as well as any "spell-corrected" version of carot, including carrot and tarot. 2. As in (1) above, but only when the query term carot is not in the dictionary. 3. As in (1) above, but only when the original query returned fewer than a preset number of documents (say fewer than five documents). 4. When the original query returns fewer than a preset number of documents, the search interface presents a spelling suggestion to the end user: this suggestion consists of the spell-corrected query term(s). Thus, the search engine might respond to the user: "Did you mean carrot?"
Forms of spelling correction We focus on two specific forms of spelling correction that we refer to as isolated-term correction and context-sensitive correction. In isolated-term correction, we attempt to correct a single query term at a time – even when we
have a multiple-term query. The carot example demonstrates this type of correction. Such isolated-term correction would fail to detect, for instance, that the query flew form Heathrow contains a mis-spelling of the term from – because each term in the query is correctly spelled in isolation. We begin by examining two techniques for addressing isolated-term correction: edit distance, and k-gram overlap. We then proceed to contextsensitive correction.
3.3.3 EDIT DISTANCE
L EVENSHTEIN DISTANCE
Edit distance Given two character strings s1 and s2 , the edit distance between them is the minimum number of edit operations required to transform s1 into s2 . Most commonly, the edit operations allowed for this purpose are: (i) insert a character into a string; (ii) delete a character from a string and (iii) replace a character of a string by another character; for these operations, edit distance is sometimes known as Levenshtein distance. For example, the edit distance between cat and dog is 3. In fact, the notion of edit distance can be generalized to allowing different weights for different kinds of edit operations, for instance a higher weight may be placed on replacing the character s by the character p, than on replacing it by the character a (the latter being closer to s on the keyboard). Setting weights in this way depending on the likelihood of letters substituting for each other is very effective in practice (see Section 3.4 for the separate issue of phonetic similarity). However, the remainder of our treatment here will focus on the case in which all edit operations have the same weight. It is well-known how to compute the (weighted) edit distance between two strings in time O(|s1 | × |s2 |), where |si | denotes the length of a string si . The idea is to use the dynamic programming algorithm in Figure 3.5, where the characters in s1 and s2 are given in array form. The algorithm fills the (integer) entries in a matrix m whose two dimensions equal the lengths of the two strings whose edit distances is being computed; the (i, j) entry of the matrix will hold (after the algorithm is executed) the edit distance between the strings consisting of the first i characters of s1 and the first j characters of s2 . The central dynamic programming step is depicted in Lines 8-10 of Figure 3.5, where the three quantities whose minimum is taken correspond to substituting a character in s1 , inserting a character in s1 and inserting a character in s2 . Figure 3.6 shows an example Levenshtein distance computation of Figure 3.5. The typical cell [i, j] has four entries formatted as a 2 × 2 cell. The lower right entry in each cell is the min of the other three, corresponding to the main dynamic programming step in Figure 3.5. The other three entries are the three entries m[i − 1, j − 1] + 0 or 1 depending on whether s1 [i ] =
E DIT D ISTANCE (s1 , s2 ) 1 int m[i, j] = 0 2 for i ← 1 to |s1 | 3 do m[i, 0] = i 4 for j ← 1 to |s2 | 5 do m[0, j] = j 6 for i ← 1 to |s1 | 7 do for j ← 1 to |s2 | 8 do m[i, j] = min{m[i − 1, j − 1] + if (s1 [i ] = s2 [ j]) then 0 else 1fi, 9 m[i − 1, j] + 1, 10 m[i, j − 1] + 1} 11 return m[|s1 |, |s2 |] ◮ Figure 3.5 Dynamic programming algorithm for computing the edit distance between strings s1 and s2 .
c a t s
a 1 2 1 2 2 3 3 4 4
s 2 3 2 3 1 2 2 3 3
t 3 4 3 4 2 3 2 3 2
◮ Figure 3.6 Example Levenshtein distance computation. The 2 × 2 cell in the [i, j] entry of the table shows the three numbers whose minimum yields the fourth. The cells in italics determine the edit distance in this example.
s2 [ j], m[i − 1, j] + 1 and m[i, j − 1] + 1. The cells with numbers in italics depict the path by which we determine the Levenshtein distance. The spelling correction problem however demands more than computing edit distance: given a set S of strings (corresponding to terms in the vocabulary) and a query string q, we seek the string(s) in V of least edit distance from q. We may view this as a decoding problem, in which the codewords (the strings in V) are prescribed in advance. The obvious way of doing this is to compute the edit distance from q to each string in V, before selecting the
string(s) of minimum edit distance. This exhaustive search is inordinately expensive. Accordingly, a number of heuristics are used in practice to efficiently retrieve vocabulary terms likely to have low edit distance to the query term(s). The simplest such heuristic is to restrict the search to dictionary terms beginning with the same letter as the query string; the hope would be that spelling errors do not occur in the first character of the query. A more sophisticated variant of this heuristic is to use a version of the permuterm index, in which we omit the end-of-word symbol $. Consider the set of all rotations of the query string q. For each rotation r from this set, we traverse the B-tree into the permuterm index, thereby retrieving all dictionary terms that have a rotation beginning with r. For instance, if q is mase and we consider the rotation r = sema, we would retrieve dictionary terms such as semantic and semaphore that do not have a small edit distance to q. Unfortunately, we would miss more pertinent dictionary terms such as mare and mane. To address this, we refine this rotation scheme: for each rotation, we omit a suffix of ℓ characters before performing the B-tree traversal. This ensures that each term in the set R of terms retrieved from the dictionary includes a "long" substring in common with q. The value of ℓ could depend on the length of q. Alternatively, we may set it to a fixed constant such as 2.
k-gram indexes for spelling correction To further limit the set of vocabulary terms for which we compute edit distances to the query term, we now show how to invoke the k-gram index of Section 3.2.2 (page 54) to assist with retrieving vocabulary terms with low edit distance to the query q. Once we retrieve such terms, we can then find the ones of least edit distance from q. In fact, we will use the k-gram index to retrieve vocabulary terms that have many k-grams in common with the query. We will argue that for reasonable definitions of "many k-grams in common," the retrieval process is essentially that of a single scan through the postings for the k-grams in the query string q. The 2-gram (or bigram) index in Figure 3.7 shows (a portion of) the postings for the three bigrams in the query bord. Suppose we wanted to retrieve vocabulary terms that contained at least two of these three bigrams. A single scan of the postings (much as in Chapter 1) would let us enumerate all such terms; in the example of Figure 3.7 we would enumerate aboard, boardroom and border. This straightforward application of the linear scan intersection of postings immediately reveals the shortcoming of simply requiring matched vocabulary terms to contain a fixed number of k-grams from the query q: terms like boardroom, an implausible "correction" of bord, get enumerated. Conse-
- aboard
- boardroom -
- morbid
sordid
◮ Figure 3.7 Matching at least two of the three 2-grams in the query bord.
J ACCARD COEFFICIENT
quently, we require more nuanced measures of the overlap in k-grams between a vocabulary term and q. The linear scan intersection can be adapted when the measure of overlap is the Jaccard coefficient for measuring the overlap between two sets A and B, defined to be | A ∩ B|/| A ∪ B|. The two sets we consider are the set of k-grams in the query q, and the set of k-grams in a vocabulary term. As the scan proceeds, we proceed from one vocabulary term t to the next, computing on the fly the Jaccard coefficient between q and t. If the coefficient exceeds a preset threshold, we add t to the output; if not, we move on to the next term in the postings. To compute the Jaccard coefficient, we need the set of k-grams in q and t. Since we are scanning the postings for all k-grams in q, we immediately have these k-grams on hand. What about the k-grams of t? In principle, we could enumerate these on the fly from t; in practice this is not only slow but potentially infeasible since, in all likelihood, the postings entries themselves do not contain the complete string t but rather some encoding of t. The crucial observation is that to compute the Jaccard coefficient, we only need the length of the string t. To see this, recall the example of Figure 3.7 and consider the point when the postings scan for query q = bord reaches term t = boardroom. We know that two bigrams match. If the postings stored the (pre-computed) number of bigrams in boardroom (namely, 8), we have all the information we require to compute the Jaccard coefficient to be 2/(8 + 3 − 2); the numerator is obtained from the number of postings hits (2, from bo and rd) while the denominator is the sum of the number of bigrams in bord and boardroom, less the number of postings hits. We could replace the Jaccard coefficient by other measures that allow efficient on the fly computation during postings scans. How do we use these
for spelling correction? One method that has some empirical support is to first use the k-gram index to enumerate a set of candidate vocabulary terms that are potential corrections of q. We then compute the edit distance from q to each term in this set, selecting terms from the set with small edit distance to q.
Context sensitive spelling correction Isolated-term correction would fail to correct typographical errors such as flew form Heathrow, where all three query terms are correctly spelled. When a phrase such as this retrieves few documents, a search engine may like to offer the corrected query flew from Heathrow. The simplest way to do this is to enumerate corrections of each of the three query terms (using the methods leading up to Section 3.3.4) even though each query term is correctly spelled, then try substitutions of each correction in the phrase. For the example flew form Heathrow, we enumerate such phrases as fled form Heathrow and flew fore Heathrow. For each such substitute phrase, the search engine runs the query and determines the number of matching results. This enumeration can be expensive if we find many corrections of the individual terms, since we could encounter a large number of combinations of alternatives. Several heuristics are used to trim this space. In the example above, as we expand the alternatives for flew and form, we retain only the most frequent combinations in the collection or in the query logs, which contain previous queries by users. For instance, we would retain flew from as an alternative to try and extend to a three-term corrected query, but perhaps not fled fore or flea form. In this example, the biword fled fore is likely to be rare compared to the biword flew from. Then, we only attempt to extend the list of top biwords (such as flew from), to corrections of Heathrow. As an alternative to using the biword statistics in the collection, we may use the logs of queries issued by users; these could of course include queries with spelling errors.
Exercise 3.7 If | si | denotes the length of string si , show that the edit distance between s1 and s2 is never more than max{| s1 |, | s2 |}. Exercise 3.8 Compute the edit distance between paris and alice. Write down the 5 × 5 array of distances between all prefixes as computed by the algorithm in Figure 3.5. Exercise 3.9 Write pseudocode showing the details of computing on the fly the Jaccard coefficient while scanning the postings of the k-gram index, as mentioned on page 61. Exercise 3.10 Compute the Jaccard coefficients between the query bord and each of the terms in Figure 3.7 that contain the bigram or.
3.4 Phonetic correction
Exercise 3.11 Consider the four-term query catched in the rye and suppose that each of the query terms has five alternative terms suggested by isolated-term correction. How many possible corrected phrases must we consider if we do not trim the space of corrected phrases, but instead try all six variants for each of the terms? Exercise 3.12 For each of the prefixes of the query — catched, catched in and catched in the — we have a number of substitute prefixes arising from each term and its alternatives. Suppose that we were to retain only the top 10 of these substitute prefixes, as measured by its number of occurrences in the collection. We eliminate the rest from consideration for extension to longer prefixes: thus, if batched in is not one of the 10 most common 2-term queries in the collection, we do not consider any extension of batched in as possibly leading to a correction of catched in the rye. How many of the possible substitute prefixes are we eliminating at each phase? Exercise 3.13 Are we guaranteed that retaining and extending only the 10 commonest substitute prefixes of catched in will lead to one of the 10 commonest substitute prefixes of catched in the?
Phonetic correction Our final technique for tolerant retrieval has to do with phonetic correction: misspellings that arise because the user types a query that sounds like the target term. Such algorithms are especially applicable to searches on the names of people. The main idea here is to generate, for each term, a "phonetic hash" so that similar-sounding terms hash to the same value. The idea owes its origins to work in international police departments from the early 20th century, seeking to match names for wanted criminals despite the names being spelled differently in different countries. It is mainly used to correct phonetic misspellings in proper nouns. Algorithms for such phonetic hashing are commonly collectively known as soundex algorithms. However, there is an original soundex algorithm, with various variants, built on the following scheme: 1. Turn every term to be indexed into a 4-character reduced form. Build an inverted index from these reduced forms to the original terms; call this the soundex index. 2. Do the same with query terms. 3. When the query calls for a soundex match, search this soundex index. The variations in different soundex algorithms have to do with the conversion of terms to 4-character forms. A commonly used conversion results in a 4-character code, with the first character being a letter of the alphabet and the other three being digits between 0 and 9.
1. Retain the first letter of the term. 2. Change all occurrences of the following letters to '0' (zero): 'A', E', 'I', 'O', 'U', 'H', 'W', 'Y'. 3. Change letters to digits as follows: B, F, P, V to 1. C, G, J, K, Q, S, X, Z to 2. D,T to 3. L to 4. M, N to 5. R to 6. 4. Repeatedly remove one out of each pair of consecutive identical digits. 5. Remove all zeros from the resulting string. Pad the resulting string with trailing zeros and return the first four positions, which will consist of a letter followed by three digits. For an example of a soundex map, Hermann maps to H655. Given a query (say herman), we compute its soundex code and then retrieve all vocabulary terms matching this soundex code from the soundex index, before running the resulting query on the standard inverted index. This algorithm rests on a few observations: (1) vowels are viewed as interchangeable, in transcribing names; (2) consonants with similar sounds (e.g., D and T) are put in equivalence classes. This leads to related names often having the same soundex codes. While these rules work for many cases, especially European languages, such rules tend to be writing system dependent. For example, Chinese names can be written in Wade-Giles or Pinyin transcription. While soundex works for some of the differences in the two transcriptions, for instance mapping both Wade-Giles hs and Pinyin x to 2, it fails in other cases, for example Wade-Giles j and Pinyin r are mapped differently.
Exercise 3.14 Find two differently spelled proper nouns whose soundex codes are the same. Exercise 3.15 Find two phonetically similar proper nouns whose soundex codes are different.
References and further reading Knuth (1997) is a comprehensive source for information on search trees, including B-trees and their use in searching through dictionaries. Garfield (1976) gives one of the first complete descriptions of the permuterm index. Ferragina and Venturini (2007) give an approach to addressing the space blowup in permuterm indexes. One of the earliest formal treatments of spelling correction was due to Damerau (1964). The notion of edit distance that we have used is due to Levenshtein (1965) and the algorithm in Figure 3.5 is due to Wagner and Fischer (1974). Peterson (1980) and Kukich (1992) developed variants of methods based on edit distances, culminating in a detailed empirical study of several methods by Zobel and Dart (1995), which shows that k-gram indexing is very effective for finding candidate mismatches, but should be combined with a more fine-grained technique such as edit distance to determine the most likely misspellings. Gusfield (1997) is a standard reference on string algorithms such as edit distance. Probabilistic models ("noisy channel" models) for spelling correction were pioneered by Kernighan et al. (1990) and further developed by Brill and Moore (2000) and Toutanova and Moore (2002). In these models, the misspelled query is viewed as a probabilistic corruption of a correct query. They have a similar mathematical basis to the language model methods presented in Chapter 12, and also provide ways of incorporating phonetic similarity, closeness on the keyboard, and data from the actual spelling mistakes of users. Many would regard them as the state-of-the-art approach. Cucerzan and Brill (2004) show how this work can be extended to learning spelling correction models based on query reformulations in search engine logs. The soundex algorithm is attributed to Margaret K. Odell and Robert C. Russelli (from U.S. patents granted in 1918 and 1922); the version described here draws on Bourne and Ford (1961). Zobel and Dart (1996) evaluate various phonetic matching algorithms, finding that a variant of the soundex algorithm performs poorly for general spelling correction, but that other algorithms based on the phonetic similarity of term pronunciations perform well.
4 INDEXING INDEXER
Index construction
In this chapter, we look at how to construct an inverted index. We call this process index construction or indexing; the process or machine that performs it the indexer. The design of indexing algorithms is governed by hardware constraints. We therefore begin this chapter with a review of the basics of computer hardware that are relevant for indexing. We then introduce blocked sort-based indexing (Section 4.2), an efficient single-machine algorithm designed for static collections that can be viewed as a more scalable version of the basic sort-based indexing algorithm we introduced in Chapter 1. Section 4.3 describes single-pass in-memory indexing, an algorithm that has even better scaling properties because it does not hold the vocabulary in memory. For very large collections like the web, indexing has to be distributed over computer clusters with hundreds or thousands of machines. We discuss this in Section 4.4. Collections with frequent changes require dynamic indexing introduced in Section 4.5 so that changes in the collection are immediately reflected in the index. Finally, we cover some complicating issues that can arise in indexing – such as security and indexes for ranked retrieval – in Section 4.6. Index construction interacts with several topics covered in other chapters. The indexer needs raw text, but documents are encoded in many ways (see Chapter 2). Indexers compress and decompress intermediate files and the final index (see Chapter 5). In web search, documents are not on a local file system, but have to be spidered or crawled (see Chapter 20). In enterprise search, most documents are encapsulated in varied content management systems, email applications, and databases. We give some examples in Section 4.7. Although most of these applications can be accessed via http, native Application Programming Interfaces (APIs) are usually more efficient. The reader should be aware that building the subsystem that feeds raw text to the indexing process can in itself be a challenging problem.
4 Index construction
◮ Table 4.1 Typical system parameters in 2007. The seek time is the time needed to position the disk head in a new position. The transfer time per byte is the rate of transfer from disk to memory when the head is in the right position.
Symbol s b p
Statistic average seek time transfer time per byte processor's clock rate lowlevel operation (e.g., compare & swap a word) size of main memory size of disk space
Value 5 ms = 5 × 10−3 s 0.02 µs = 2 × 10−8 s 109 s−1 0.01 µs = 10−8 s several GB 1 TB or more
Hardware basics When building an information retrieval (IR) system, many decisions are based on the characteristics of the computer hardware on which the system runs. We therefore begin this chapter with a brief review of computer hardware. Performance characteristics typical of systems in 2007 are shown in Table 4.1. A list of hardware basics that we need in this book to motivate IR system design follows.
SEEK TIME
• Access to data in memory is much faster than access to data on disk. It takes a few clock cycles (perhaps 5 × 10−9 seconds) to access a byte in memory, but much longer to transfer it from disk (about 2 × 10−8 seconds). Consequently, we want to keep as much data as possible in memory, especially those data that we need to access frequently. We call the technique of keeping frequently used disk data in main memory caching. • When doing a disk read or write, it takes a while for the disk head to move to the part of the disk where the data are located. This time is called the seek time and it averages 5 ms for typical disks. No data are being transferred during the seek. To maximize data transfer rates, chunks of data that will be read together should therefore be stored contiguously on disk. For example, using the numbers in Table 4.1 it may take as little as 0.2 seconds to transfer 10 megabytes (MB) from disk to memory if it is stored as one chunk, but up to 0.2 + 100 × (5 × 10−3 ) = 0.7 seconds if it is stored in 100 noncontiguous chunks because we need to move the disk head up to 100 times. • Operating systems generally read and write entire blocks. Thus, reading a single byte from disk can take as much time as reading the entire block.
4.2 Blocked sort-based indexing
Block sizes of 8, 16, 32, and 64 kilobytes (KB) are common. We call the part of main memory where a block being read or written is stored a buffer. • Data transfers from disk to memory are handled by the system bus, not by the processor. This means that the processor is available to process data during disk I/O. We can exploit this fact to speed up data transfers by storing compressed data on disk. Assuming an efficient decompression algorithm, the total time of reading and then decompressing compressed data is usually less than reading uncompressed data. • Servers used in IR systems typically have several gigabytes (GB) of main memory, sometimes tens of GB. Available disk space is several orders of magnitude larger.
TERM ID
R EUTERS -RCV1
Blocked sort-based indexing The basic steps in constructing a nonpositional index are depicted in Figure 1.4 (page 8). We first make a pass through the collection assembling all term–docID pairs. We then sort the pairs with the term as the dominant key and docID as the secondary key. Finally, we organize the docIDs for each term into a postings list and compute statistics like term and document frequency. For small collections, all this can be done in memory. In this chapter, we describe methods for large collections that require the use of secondary storage. To make index construction more efficient, we represent terms as termIDs (instead of strings as we did in Figure 1.4), where each termID is a unique serial number. We can build the mapping from terms to termIDs on the fly while we are processing the collection; or, in a two-pass approach, we compile the vocabulary in the first pass and construct the inverted index in the second pass. The index construction algorithms described in this chapter all do a single pass through the data. Section 4.7 gives references to multipass algorithms that are preferable in certain applications, for example, when disk space is scarce. We work with the Reuters-RCV1 collection as our model collection in this chapter, a collection with roughly 1 GB of text. It consists of about 800,000 documents that were sent over the Reuters newswire during a 1-year period between August 20, 1996, and August 19, 1997. A typical document is shown in Figure 4.1, but note that we ignore multimedia information like images in this book and are only concerned with text. Reuters-RCV1 covers a wide range of international topics, including politics, business, sports, and (as in this example) science. Some key statistics of the collection are shown in Table 4.2.
◮ Table 4.2 Collection statistics for Reuters-RCV1. Values are rounded for the computations in this book. The unrounded values are: 806,791 documents, 222 tokens per document, 391,523 (distinct) terms, 6.04 bytes per token with spaces and punctuation, 4.5 bytes per token without spaces and punctuation, 7.5 bytes per term, and 96,969,056 tokens. The numbers in this table correspond to the third line ("case folding") in Table 5.1 (page 87).
Symbol N Lave M
Statistic documents avg. # tokens per document terms avg. # bytes per token (incl. spaces/punct.) avg. # bytes per token (without spaces/punct.) avg. # bytes per term tokens
Value 800,000 200 400,000 6 4.5 7.5 100,000,000
REUTERS You are here: Home > News > Science > Article Go to a Section:
Extreme conditions create rare Antarctic clouds Tue Aug 1, 2006 3:20am ET Email This Article | Print This Article | Reprints
SYDNEY (Reuters) - Rare, mother-of-pearl colored clouds caused by extreme weather conditions above Antarctica are a possible indication of global warming, Australian scientists said on Tuesday.
[-] Text [+]
Known as nacreous clouds, the spectacular formations showing delicate wisps of colors were photographed in the sky over an Australian meteorological base at Mawson Station on July 25.
◮ Figure 4.1 Document from the Reuters newswire.
EXTERNAL SORTING ALGORITHM
Reuters-RCV1 has 100 million tokens. Collecting all termID–docID pairs of the collection using 4 bytes each for termID and docID therefore requires 0.8 GB of storage. Typical collections today are often one or two orders of magnitude larger than Reuters-RCV1. You can easily see how such collections overwhelm even large computers if we try to sort their termID–docID pairs in memory. If the size of the intermediate files during index construction is within a small factor of available memory, then the compression techniques introduced in Chapter 5 can help; however, the postings file of many large collections cannot fit into memory even after compression. With main memory insufficient, we need to use an external sorting algorithm, that is, one that uses disk. For acceptable speed, the central require-
BSBI NDEX C ONSTRUCTION () 1 n←0 2 while (all documents have not been processed) 3 do n ← n + 1 4 block ← PARSE N EXT B LOCK () 5 BSBI-I NVERT(block) 6 W RITE B LOCK T O D ISK (block, f n ) 7 M ERGE B LOCKS ( f 1 , . . . , f n ; f merged ) ◮ Figure 4.2 Blocked sort-based indexing. The algorithm stores inverted blocks in files f 1 , . . . , f n and the merged index in f merged .
BLOCKED SORT- BASED INDEXING ALGORITHM
ment of such an algorithm is that it minimize the number of random disk seeks during sorting – sequential disk reads are far faster than seeks as we explained in Section 4.1. One solution is the blocked sort-based indexing algorithm or BSBI in Figure 4.2. BSBI (i) segments the collection into parts of equal size, (ii) sorts the termID–docID pairs of each part in memory, (iii) stores intermediate sorted results on disk, and (iv) merges all intermediate results into the final index. The algorithm parses documents into termID–docID pairs and accumulates the pairs in memory until a block of a fixed size is full (PARSE N EXT B LOCK in Figure 4.2). We choose the block size to fit comfortably into memory to permit a fast in-memory sort. The block is then inverted and written to disk. Inversion involves two steps. First, we sort the termID–docID pairs. Next, we collect all termID–docID pairs with the same termID into a postings list, where a posting is simply a docID. The result, an inverted index for the block we have just read, is then written to disk. Applying this to Reuters-RCV1 and assuming we can fit 10 million termID–docID pairs into memory, we end up with ten blocks, each an inverted index of one part of the collection. In the final step, the algorithm simultaneously merges the ten blocks into one large merged index. An example with two blocks is shown in Figure 4.3, where we use di to denote the ith document of the collection. To do the merging, we open all block files simultaneously, and maintain small read buffers for the ten blocks we are reading and a write buffer for the final merged index we are writing. In each iteration, we select the lowest termID that has not been processed yet using a priority queue or a similar data structure. All postings lists for this termID are read and merged, and the merged list is written back to disk. Each read buffer is refilled from its file when necessary. How expensive is BSBI? Its time complexity is Θ( T log T ) because the step with the highest time complexity is sorting and T is an upper bound for the number of items we must sort (i.e., the number of termID–docID pairs). But
postings lists to be merged brutus caesar noble with
d1,d3 d1,d2,d4 d5 d1,d2,d3,d5
brutus caesar julius killed
d6,d7 d8,d9 d10 d8
brutus caesar julius killed noble with
d1,d3,d6,d7 d1,d2,d4,d8,d9 d10 d8 d5 d1,d2,d3,d5
merged postings lists
disk ◮ Figure 4.3 Merging in blocked sort-based indexing. Two blocks ("postings lists to be merged") are loaded from disk into memory, merged in memory ("merged postings lists") and written back to disk. We show terms instead of termIDs for better readability.
the actual indexing time is usually dominated by the time it takes to parse the documents (PARSE N EXT B LOCK) and to do the final merge (M ERGE B LOCKS). Exercise 4.6 asks you to compute the total index construction time for RCV1 that includes these steps as well as inverting the blocks and writing them to disk. Notice that Reuters-RCV1 is not particularly large in an age when one or more GB of memory are standard on personal computers. With appropriate compression (Chapter 5), we could have created an inverted index for RCV1 in memory on a not overly beefy server. The techniques we have described are needed, however, for collections that are several orders of magnitude larger.
Exercise 4.1 If we need T log2 T comparisons (where T is the number of termID–docID pairs) and two disk seeks for each comparison, how much time would index construction for Reuters-RCV1 take if we used disk instead of memory for storage and an unoptimized sorting algorithm (i.e., not an external sorting algorithm)? Use the system parameters in Table 4.1. Exercise 4.2
How would you create the dictionary in blocked sort-based indexing on the fly to avoid an extra pass through the data?
4.3 Single-pass in-memory indexing
SPIMI-I NVERT (token_stream) 1 output_ f ile = N EW F ILE () 2 dictionary = N EW H ASH () 3 while (free memory available) 4 do token ← next(token_stream) 5 if term(token) ∈ / dictionary 6 then postings_list = A DD T O D ICTIONARY (dictionary, term(token)) 7 else postings_list = G ET P OSTINGS L IST (dictionary, term(token)) 8 if f ull ( postings_list) 9 then postings_list = D OUBLE P OSTINGS L IST (dictionary, term(token)) 10 A DD T O P OSTINGS L IST ( postings_list, docID (token)) 11 sorted_terms ← S ORT T ERMS (dictionary) 12 W RITE B LOCK T O D ISK (sorted_terms, dictionary, output_ f ile) 13 return output_ f ile ◮ Figure 4.4 Inversion of a block in single-pass in-memory indexing
SINGLE - PASS IN - MEMORY INDEXING
Single-pass in-memory indexing Blocked sort-based indexing has excellent scaling properties, but it needs a data structure for mapping terms to termIDs. For very large collections, this data structure does not fit into memory. A more scalable alternative is single-pass in-memory indexing or SPIMI. SPIMI uses terms instead of termIDs, writes each block's dictionary to disk, and then starts a new dictionary for the next block. SPIMI can index collections of any size as long as there is enough disk space available. The SPIMI algorithm is shown in Figure 4.4. The part of the algorithm that parses documents and turns them into a stream of term–docID pairs, which we call tokens here, has been omitted. SPIMI-I NVERT is called repeatedly on the token stream until the entire collection has been processed. Tokens are processed one by one (line 4) during each successive call of SPIMI-I NVERT. When a term occurs for the first time, it is added to the dictionary (best implemented as a hash), and a new postings list is created (line 6). The call in line 7 returns this postings list for subsequent occurrences of the term. A difference between BSBI and SPIMI is that SPIMI adds a posting directly to its postings list (line 10). Instead of first collecting all termID–docID pairs and then sorting them (as we did in BSBI), each postings list is dynamic (i.e., its size is adjusted as it grows) and it is immediately available to collect postings. This has two advantages: It is faster because there is no sorting required, and it saves memory because we keep track of the term a postings
list belongs to, so the termIDs of postings need not be stored. As a result, the blocks that individual calls of SPIMI-I NVERT can process are much larger and the index construction process as a whole is more efficient. Because we do not know how large the postings list of a term will be when we first encounter it, we allocate space for a short postings list initially and double the space each time it is full (lines 8–9). This means that some memory is wasted, which counteracts the memory savings from the omission of termIDs in intermediate data structures. However, the overall memory requirements for the dynamically constructed index of a block in SPIMI are still lower than in BSBI. When memory has been exhausted, we write the index of the block (which consists of the dictionary and the postings lists) to disk (line 12). We have to sort the terms (line 11) before doing this because we want to write postings lists in lexicographic order to facilitate the final merging step. If each block's postings lists were written in unsorted order, merging blocks could not be accomplished by a simple linear scan through each block. Each call of SPIMI-I NVERT writes a block to disk, just as in BSBI. The last step of SPIMI (corresponding to line 7 in Figure 4.2; not shown in Figure 4.4) is then to merge the blocks into the final inverted index. In addition to constructing a new dictionary structure for each block and eliminating the expensive sorting step, SPIMI has a third important component: compression. Both the postings and the dictionary terms can be stored compactly on disk if we employ compression. Compression increases the efficiency of the algorithm further because we can process even larger blocks, and because the individual blocks require less space on disk. We refer readers to the literature for this aspect of the algorithm (Section 4.7). The time complexity of SPIMI is Θ( T ) because no sorting of tokens is required and all operations are at most linear in the size of the collection.
Distributed indexing Collections are often so large that we cannot perform index construction efficiently on a single machine. This is particularly true of the World Wide Web for which we need large computer clusters1 to construct any reasonably sized web index. Web search engines, therefore, use distributed indexing algorithms for index construction. The result of the construction process is a distributed index that is partitioned across several machines – either according to term or according to document. In this section, we describe distributed indexing for a term-partitioned index. Most large search engines prefer a document1. A cluster in this chapter is a group of tightly coupled computers that work together closely. This sense of the word is different from the use of cluster as a group of documents that are semantically similar in Chapters 16–18.
4.4 Distributed indexing
M AP R EDUCE
MASTER NODE
KEY- VALUE PAIRS
MAP PHASE
PARSER SEGMENT FILE REDUCE PHASE
partitioned index (which can be easily generated from a term-partitioned index). We discuss this topic further in Section 20.3 (page 454). The distributed index construction method we describe in this section is an application of MapReduce, a general architecture for distributed computing. MapReduce is designed for large computer clusters. The point of a cluster is to solve large computing problems on cheap commodity machines or nodes that are built from standard parts (processor, memory, disk) as opposed to on a supercomputer with specialized hardware. Although hundreds or thousands of machines are available in such clusters, individual machines can fail at any time. One requirement for robust distributed indexing is, therefore, that we divide the work up into chunks that we can easily assign and – in case of failure – reassign. A master node directs the process of assigning and reassigning tasks to individual worker nodes. The map and reduce phases of MapReduce split up the computing job into chunks that standard machines can process in a short time. The various steps of MapReduce are shown in Figure 4.5 and an example on a collection consisting of two documents is shown in Figure 4.6. First, the input data, in our case a collection of web pages, are split into n splits where the size of the split is chosen to ensure that the work can be distributed evenly (chunks should not be too large) and efficiently (the total number of chunks we need to manage should not be too large); 16 or 64 MB are good sizes in distributed indexing. Splits are not preassigned to machines, but are instead assigned by the master node on an ongoing basis: As a machine finishes processing one split, it is assigned the next one. If a machine dies or becomes a laggard due to hardware problems, the split it is working on is simply reassigned to another machine. In general, MapReduce breaks a large computing problem into smaller parts by recasting it in terms of manipulation of key-value pairs. For indexing, a key-value pair has the form (termID,docID). In distributed indexing, the mapping from terms to termIDs is also distributed and therefore more complex than in single-machine indexing. A simple solution is to maintain a (perhaps precomputed) mapping for frequent terms that is copied to all nodes and to use terms directly (instead of termIDs) for infrequent terms. We do not address this problem here and assume that all nodes share a consistent term → termID mapping. The map phase of MapReduce consists of mapping splits of the input data to key-value pairs. This is the same parsing task we also encountered in BSBI and SPIMI, and we therefore call the machines that execute the map phase parsers. Each parser writes its output to local intermediate files, the segment files (shown as a-f g-p q-z in Figure 4.5). For the reduce phase, we want all values for a given key to be stored close together, so that they can be read and processed quickly. This is achieved by
a-f g-p q-z
segment files
inve rter
reduce phase
◮ Figure 4.5 An example of distributed indexing with MapReduce. Adapted from Dean and Ghemawat (2004).
partitioning the keys into j term partitions and having the parsers write keyvalue pairs for each term partition into a separate segment file. In Figure 4.5, the term partitions are according to first letter: a–f, g–p, q–z, and j = 3. (We chose these key ranges for ease of exposition. In general, key ranges need not correspond to contiguous terms or termIDs.) The term partitions are defined by the person who operates the indexing system (Exercise 4.10). The parsers then write corresponding segment files, one for each term partition. Each term partition thus corresponds to r segments files, where r is the number of parsers. For instance, Figure 4.5 shows three a–f segment files of the a–f partition, corresponding to the three parsers shown in the figure. Collecting all values (here: docIDs) for a given key (here: termID) into one list is the task of the inverters in the reduce phase. The master assigns each term partition to a different inverter – and, as in the case of parsers, reassigns term partitions in case of failing or slow inverters. Each term partition (corresponding to r segment files, one on each parser) is processed by one inverter. We assume here that segment files are of a size that a single machine can handle (Exercise 4.9). Finally, the list of values is sorted for each key and written to the final sorted postings list ("postings" in the figure). (Note that postings in Figure 4.6 include term frequencies, whereas each posting in the other sections of this chapter is simply a docID without term frequency information.) The data flow is shown for a–f in Figure 4.5. This completes the construction of the inverted index.
4.4 Distributed indexing Schema of map and reduce functions map: input reduce: (k,list(v))
→ list(k , v) → output
Instantiation of the schema for index construction → list(termID, docID) map: web collection reduce: (〈termID1, list(docID)〉, 〈termID2 , list(docID)〉, . . . ) → (postings list1, postings list 2 , . . . ) Example for index construction map: d2 : C died. d1 : C came, C c'ed. reduce: ( 〈C,(d2 ,d1 ,d1 )〉, 〈died,(d2 )〉, 〈came,(d1 )〉, 〈c'ed,(d1 ) 〉)
→ ( 〈C, d2 〉, 〈died,d2〉, 〈C,d1〉, 〈came,d1〉, 〈C,d1〉, 〈c'ed,d1〉) → (〈C,(d1:2,d2:1)〉, 〈died,(d2:1)〉, 〈came,(d1:1)〉, 〈c'ed,(d1:1)〉 )
◮ Figure 4.6 Map and reduce functions in MapReduce. In general, the map function produces a list of key-value pairs. All values for a key are collected into one list in the reduce phase. This list is then processed further. The instantiations of the two functions and an example are shown for index construction. Because the map phase processes documents in a distributed fashion, termID–docID pairs need not be ordered correctly initially as in this example. The example shows terms instead of termIDs for better readability. We abbreviate Caesar as C and conquered as c'ed.
Parsers and inverters are not separate sets of machines. The master identifies idle machines and assigns tasks to them. The same machine can be a parser in the map phase and an inverter in the reduce phase. And there are often other jobs that run in parallel with index construction, so in between being a parser and an inverter a machine might do some crawling or another unrelated task. To minimize write times before inverters reduce the data, each parser writes its segment files to its local disk. In the reduce phase, the master communicates to an inverter the locations of the relevant segment files (e.g., of the r segment files of the a–f partition). Each segment file only requires one sequential read because all data relevant to a particular inverter were written to a single segment file by the parser. This setup minimizes the amount of network traffic needed during indexing. Figure 4.6 shows the general schema of the MapReduce functions. Input and output are often lists of key-value pairs themselves, so that several MapReduce jobs can run in sequence. In fact, this was the design of the Google indexing system in 2004. What we describe in this section corresponds to only one of five to ten MapReduce operations in that indexing system. Another MapReduce operation transforms the term-partitioned index we just created into a document-partitioned one. MapReduce offers a robust and conceptually simple framework for implementing index construction in a distributed environment. By providing a semiautomatic method for splitting index construction into smaller tasks, it can scale to almost arbitrarily large collections, given computer clusters of
sufficient size.
Dynamic indexing
AUXILIARY INDEX
For n = 15 splits, r = 10 segments, and j = 3 term partitions, how long would distributed index creation take for Reuters-RCV1 in a MapReduce architecture? Base your assumptions about cluster machines on Table 4.1.
Thus far, we have assumed that the document collection is static. This is fine for collections that change infrequently or never (e.g., the Bible or Shakespeare). But most collections are modified frequently with documents being added, deleted, and updated. This means that new terms need to be added to the dictionary, and postings lists need to be updated for existing terms. The simplest way to achieve this is to periodically reconstruct the index from scratch. This is a good solution if the number of changes over time is small and a delay in making new documents searchable is acceptable – and if enough resources are available to construct a new index while the old one is still available for querying. If there is a requirement that new documents be included quickly, one solution is to maintain two indexes: a large main index and a small auxiliary index that stores new documents. The auxiliary index is kept in memory. Searches are run across both indexes and results merged. Deletions are stored in an invalidation bit vector. We can then filter out deleted documents before returning the search result. Documents are updated by deleting and reinserting them. Each time the auxiliary index becomes too large, we merge it into the main index. The cost of this merging operation depends on how we store the index in the file system. If we store each postings list as a separate file, then the merge simply consists of extending each postings list of the main index by the corresponding postings list of the auxiliary index. In this scheme, the reason for keeping the auxiliary index is to reduce the number of disk seeks required over time. Updating each document separately requires up to Mave disk seeks, where Mave is the average size of the vocabulary of documents in the collection. With an auxiliary index, we only put additional load on the disk when we merge auxiliary and main indexes. Unfortunately, the one-file-per-postings-list scheme is infeasible because most file systems cannot efficiently handle very large numbers of files. The simplest alternative is to store the index as one large file, that is, as a concatenation of all postings lists. In reality, we often choose a compromise between the two extremes (Section 4.7). To simplify the discussion, we choose the simple option of storing the index as one large file here.
4.5 Dynamic indexing
LM ERGE A DD T OKEN (indexes, Z0 , token) 1 Z0 ← M ERGE( Z0 , {token}) 2 if | Z0 | = n 3 then for i ← 0 to ∞ 4 do if Ii ∈ indexes 5 then Zi+1 ← M ERGE( Ii , Zi ) 6 (Zi+1 is a temporary index on disk.) 7 indexes ← indexes − { Ii } 8 else Ii ← Zi (Zi becomes the permanent index Ii .) 9 indexes ← indexes ∪ { Ii } 10 B REAK 11 Z0 ← ∅ L OGARITHMIC M ERGE () 1 Z0 ← ∅ (Z0 is the in-memory index.) 2 indexes ← ∅ 3 while true 4 do LM ERGE A DD T OKEN (indexes, Z0 , GET N EXT T OKEN()) ◮ Figure 4.7 Logarithmic merging. Each token (termID,docID) is initially added to in-memory index Z0 by LM ERGE A DD T OKEN . L OGARITHMIC M ERGE initializes Z0 and indexes.
LOGARITHMIC MERGING
In this scheme, we process each posting ⌊ T/n⌋ times because we touch it during each of ⌊ T/n⌋ merges where n is the size of the auxiliary index and T the total number of postings. Thus, the overall time complexity is Θ( T 2 /n). (We neglect the representation of terms here and consider only the docIDs. For the purpose of time complexity, a postings list is simply a list of docIDs.) We can do better than Θ( T 2 /n) by introducing log2 ( T/n) indexes I0 , I1 , I2 , . . . of size 20 × n, 21 × n, 22 × n . . . . Postings percolate up this sequence of indexes and are processed only once on each level. This scheme is called logarithmic merging (Figure 4.7). As before, up to n postings are accumulated in an in-memory auxiliary index, which we call Z0 . When the limit n is reached, the 20 × n postings in Z0 are transferred to a new index I0 that is created on disk. The next time Z0 is full, it is merged with I0 to create an index Z1 of size 21 × n. Then Z1 is either stored as I1 (if there isn't already an I1 ) or merged with I1 into Z2 (if I1 exists); and so on. We service search requests by querying in-memory Z0 and all currently valid indexes Ii on disk and merging the results. Readers familiar with the binomial heap data structure2 will recog2. See, for example, (Cormen et al. 1990, Chapter 19).
nize its similarity with the structure of the inverted indexes in logarithmic merging. Overall index construction time is Θ( T log( T/n)) because each posting is processed only once on each of the log( T/n) levels. We trade this efficiency gain for a slow down of query processing; we now need to merge results from log( T/n) indexes as opposed to just two (the main and auxiliary indexes). As in the auxiliary index scheme, we still need to merge very large indexes occasionally (which slows down the search system during the merge), but this happens less frequently and the indexes involved in a merge on average are smaller. Having multiple indexes complicates the maintenance of collection-wide statistics. For example, it affects the spelling correction algorithm in Section 3.3 (page 56) that selects the corrected alternative with the most hits. With multiple indexes and an invalidation bit vector, the correct number of hits for a term is no longer a simple lookup. In fact, all aspects of an IR system – index maintenance, query processing, distribution, and so on – are more complex in logarithmic merging. Because of this complexity of dynamic indexing, some large search engines adopt a reconstruction-from-scratch strategy. They do not construct indexes dynamically. Instead, a new index is built from scratch periodically. Query processing is then switched from the new index and the old index is deleted.
Exercise 4.4 For n = 2 and 1 ≤ T ≤ 30, perform a step-by-step simulation of the algorithm in Figure 4.7. Create a table that shows, for each point in time at which T = 2 ∗ k tokens have been processed (1 ≤ k ≤ 15), which of the three indexes I0 , . . . , I3 are in use. The first three lines of the table are given below. 2 4 6
I3 0 0 0
Other types of indexes This chapter only describes construction of nonpositional indexes. Except for the much larger data volume we need to accommodate, the main difference for positional indexes is that (termID, docID, (position1, position2, . . . )) triples, instead of (termID, docID) pairs have to be processed and that tokens and postings contain positional information in addition to docIDs. With this change, the algorithms discussed here can all be applied to positional indexes. In the indexes we have considered so far, postings lists are ordered with respect to docID. As we see in Chapter 5, this is advantageous for compres-
4.6 Other types of indexes
0 if user can't read doc, 1 otherwise.
◮ Figure 4.8 A user-document matrix for access control lists. Element (i, j) is 1 if user i has access to document j and 0 otherwise. During query processing, a user's access postings list is intersected with the results list returned by the text part of the index.
RANKED RETRIEVAL SYSTEMS
sion – instead of docIDs we can compress smaller gaps between IDs, thus reducing space requirements for the index. However, this structure for the index is not optimal when we build ranked (Chapters 6 and 7) – as opposed to Boolean – retrieval systems. In ranked retrieval, postings are often ordered according to weight or impact, with the highest-weighted postings occurring first. With this organization, scanning of long postings lists during query processing can usually be terminated early when weights have become so small that any further documents can be predicted to be of low similarity to the query (see Chapter 6). In a docID-sorted index, new documents are always inserted at the end of postings lists. In an impact-sorted index (Section 7.1.5, page 140), the insertion can occur anywhere, thus complicating the update of the inverted index. Securityis an important consideration for retrieval systems in corporations. A low-level employee should not be able to find the salary roster of the corporation, but authorized managers need to be able to search for it. Users' results lists must not contain documents they are barred from opening; the very existence of a document can be sensitive information. User authorization is often mediated through access control lists or ACLs. ACLs can be dealt with in an information retrieval system by representing each document as the set of users that can access them (Figure 4.8) and then inverting the resulting user-document matrix. The inverted ACL index has, for each user, a "postings list" of documents they can access – the user's access list. Search results are then intersected with this list. However, such an index is difficult to maintain when access permissions change – we discussed these difficulties in the context of incremental indexing for regular postings lists in Section 4.5. It also requires the processing of very long postings lists for users with access to large document subsets. User membership is therefore often verified by retrieving access information directly from the file system at query time – even though this slows down retrieval.
◮ Table 4.3 The five steps in constructing an index for Reuters-RCV1 in blocked sort-based indexing. Line numbers refer to Figure 4.2.
Step reading of collection (line 4) 10 initial sorts of 107 records each (line 5) writing of 10 blocks (line 6) total disk transfer time for merging (line 7) time of actual merging (line 7) total
◮ Table 4.4 Collection statistics for a large collection.
Statistic # documents # tokens per document # distinct terms
Value 1,000,000,000 1000 44,000,000
We discussed indexes for storing and retrieving terms (as opposed to documents) in Chapter 3.
Exercise 4.5 Can spelling correction compromise document-level security? Consider the case where a spelling correction is based on documents to which the user does not have access. Exercise 4.6 Total index construction time in blocked sort-based indexing is broken down in Table 4.3. Fill out the time column of the table for Reuters-RCV1 assuming a system with the parameters given in Table 4.1. Exercise 4.7 Repeat Exercise 4.6 for the larger collection in Table 4.4. Choose a block size that is realistic for current technology (remember that a block should easily fit into main memory). How many blocks do you need? Exercise 4.8 Assume that we have a collection of modest size whose index can be constructed with the simple in-memory indexing algorithm in Figure 1.4 (page 8). For this collection, compare memory, disk and time requirements of the simple algorithm in Figure 1.4 and blocked sort-based indexing. Exercise 4.9 Assume that machines in MapReduce have 100 GB of disk space each. Assume further that the postings list of the term the has a size of 200 GB. Then the MapReduce algorithm as described cannot be run to construct the index. How would you modify MapReduce so that it can handle this case?
Exercise 4.10 For optimal load balancing, the inverters in MapReduce must get segmented postings files of similar sizes. For a new collection, the distribution of key-value pairs may not be known in advance. How would you solve this problem? Exercise 4.11 Apply MapReduce to the problem of counting how often each term occurs in a set of files. Specify map and reduce operations for this task. Write down an example along the lines of Figure 4.6. Exercise 4.12 We claimed (on page 80) that an auxiliary index can impair the quality of collection statistics. An example is the term weighting method idf, which is defined as log( N/dfi ) where N is the total number of documents and dfi is the number of documents that term i occurs in (Section 6.2.1, page 117). Show that even a small auxiliary index can cause significant error in idf when it is computed on the main index only. Consider a rare term that suddenly occurs frequently (e.g., Flossie as in Tropical Storm Flossie).
References and further reading Witten et al. (1999, Chapter 5) present an extensive treatment of the subject of index construction and additional indexing algorithms with different tradeoffs of memory, disk space, and time. In general, blocked sort-based indexing does well on all three counts. However, if conserving memory or disk space is the main criterion, then other algorithms may be a better choice. See Witten et al. (1999), Tables 5.4 and 5.5; BSBI is closest to "sort-based multiway merge," but the two algorithms differ in dictionary structure and use of compression. Moffat and Bell (1995) show how to construct an index "in situ," that is, with disk space usage close to what is needed for the final index and with a minimum of additional temporary files (cf. also Harman and Candela (1990)). They give Lesk (1988) and Somogyi (1990) credit for being among the first to employ sorting for index construction. The SPIMI method in Section 4.3 is from (Heinz and Zobel 2003). We have simplified several aspects of the algorithm, including compression and the fact that each term's data structure also contains, in addition to the postings list, its document frequency and house keeping information. We recommend Heinz and Zobel (2003) and Zobel and Moffat (2006) as up-do-date, in-depth treatments of index construction. Other algorithms with good scaling properties with respect to vocabulary size require several passes through the data, e.g., FAST-INV (Fox and Lee 1991, Harman et al. 1992). The MapReduce architecture was introduced by Dean and Ghemawat (2004). An open source implementation of MapReduce is available at http://lucene.apache.org/hadoop/. Ribeiro-Neto et al. (1999) and Melnik et al. (2001) describe other approaches
to distributed indexing. Introductory chapters on distributed IR are (BaezaYates and Ribeiro-Neto 1999, Chapter 9) and (Grossman and Frieder 2004, Chapter 8). See also Callan (2000). Lester et al. (2005) and Büttcher and Clarke (2005a) analyze the properties of logarithmic merging and compare it with other construction methods. One of the first uses of this method was in Lucene (http://lucene.apache.org). Other dynamic indexing methods are discussed by Büttcher et al. (2006) and Lester et al. (2006). The latter paper also discusses the strategy of replacing the old index by one built from scratch. Heinz et al. (2002) compare data structures for accumulating the vocabulary in memory. Büttcher and Clarke (2005b) discuss security models for a common inverted index for multiple users. A detailed characterization of the Reuters-RCV1 collection can be found in (Lewis et al. 2004). NIST distributes the collection (see http://trec.nist.gov/data/reuters/reuters.html). Garcia-Molina et al. (1999, Chapter 2) review computer hardware relevant to system design in depth. An effective indexer for enterprise search needs to be able to communicate efficiently with a number of applications that hold text data in corporations, including Microsoft Outlook, IBM's Lotus software, databases like Oracle and MySQL, content management systems like Open Text, and enterprise resource planning software like SAP.
Index compression
Chapter 1 introduced the dictionary and the inverted index as the central data structures in information retrieval (IR). In this chapter, we employ a number of compression techniques for dictionary and inverted index that are essential for efficient IR systems. One benefit of compression is immediately clear. We need less disk space. As we will see, compression ratios of 1:4 are easy to achieve, potentially cutting the cost of storing the index by 75%. There are two more subtle benefits of compression. The first is increased use of caching. Search systems use some parts of the dictionary and the index much more than others. For example, if we cache the postings list of a frequently used query term t, then the computations necessary for responding to the one-term query t can be entirely done in memory. With compression, we can fit a lot more information into main memory. Instead of having to expend a disk seek when processing a query with t, we instead access its postings list in memory and decompress it. As we will see below, there are simple and efficient decompression methods, so that the penalty of having to decompress the postings list is small. As a result, we are able to decrease the response time of the IR system substantially. Because memory is a more expensive resource than disk space, increased speed owing to caching – rather than decreased space requirements – is often the prime motivator for compression. The second more subtle advantage of compression is faster transfer of data from disk to memory. Efficient decompression algorithms run so fast on modern hardware that the total time of transferring a compressed chunk of data from disk and then decompressing it is usually less than transferring the same chunk of data in uncompressed form. For instance, we can reduce input/output (I/O) time by loading a much smaller compressed postings list, even when you add on the cost of decompression. So, in most cases, the retrieval system runs faster on compressed postings lists than on uncompressed postings lists. If the main goal of compression is to conserve disk space, then the speed
5 Index compression
RULE OF
of compression algorithms is of no concern. But for improved cache utilization and faster disk-to-memory transfer, decompression speeds must be high. The compression algorithms we discuss in this chapter are highly efficient and can therefore serve all three purposes of index compression. In this chapter, we define a posting as a docID in a postings list. For example, the postings list (6; 20, 45, 100), where 6 is the termID of the list's term, contains three postings. As discussed in Section 2.4.2 (page 41), postings in most search systems also contain frequency and position information; but we will only consider simple docID postings here. See Section 5.4 for references on compressing frequencies and positions. This chapter first gives a statistical characterization of the distribution of the entities we want to compress – terms and postings in large collections (Section 5.1). We then look at compression of the dictionary, using the dictionaryas-a-string method and blocked storage (Section 5.2). Section 5.3 describes two techniques for compressing the postings file, variable byte encoding and γ encoding.
Statistical properties of terms in information retrieval As in the last chapter, we use Reuters-RCV1 as our model collection (see Table 4.2, page 70). We give some term and postings statistics for the collection in Table 5.1. "∆%" indicates the reduction in size from the previous line. "T%" is the cumulative reduction from unfiltered. The table shows the number of terms for different levels of preprocessing (column 2). The number of terms is the main factor in determining the size of the dictionary. The number of nonpositional postings (column 3) is an indicator of the expected size of the nonpositional index of the collection. The expected size of a positional index is related to the number of positions it must encode (column 4). In general, the statistics in Table 5.1 show that preprocessing affects the size of the dictionary and the number of nonpositional postings greatly. Stemming and case folding reduce the number of (distinct) terms by 17% each and the number of nonpositional postings by 4% and 3%, respectively. The treatment of the most frequent words is also important. The rule of 30 states that the 30 most common words account for 30% of the tokens in written text (31% in the table). Eliminating the 150 most common words from indexing (as stop words; cf. Section 2.2.2, page 27) cuts 25% to 30% of the nonpositional postings. But, although a stop list of 150 words reduces the number of postings by a quarter or more, this size reduction does not carry over to the size of the compressed index. As we will see later in this chapter, the postings lists of frequent words require only a few bits per posting after compression. The deltas in the table are in a range typical of large collections. Note,
5.1 Statistical properties of terms in information retrieval
◮ Table 5.1 The effect of preprocessing on the number of terms, nonpositional postings, and tokens for Reuters-RCV1. "∆%" indicates the reduction in size from the previous line, except that "30 stop words" and "150 stop words" both use "case folding" as their reference line. "T%" is the cumulative ("total") reduction from unfiltered. We performed stemming with the Porter stemmer (Chapter 2, page 33).
(distinct) terms unfiltered no numbers case folding 30 stop words 150 stop words stemming
LOSSLESS LOSSY COMPRESSION
number 484,494 473,723 391,523 391,493 391,373 322,383
∆%
−2 −17 −0 −0 −17
−2 −19 −19 −19 −33
nonpositional postings number 109,971,179 100,680,242 96,969,056 83,390,443 67,001,847 63,812,300
−8 −3 −14 −30 −4
tokens (= number of positio entries in postings) number 197,879,290 179,158,204 179,158,204 121,857,825 94,516,599 94,516,599
however, that the percentage reductions can be very different for some text collections. For example, for a collection of web pages with a high proportion of French text, a lemmatizer for French reduces vocabulary size much more than the Porter stemmer does for an English-only collection because French is a morphologically richer language than English. The compression techniques we describe in the remainder of this chapter are lossless, that is, all information is preserved. Better compression ratios can be achieved with lossy compression, which discards some information. Case folding, stemming, and stop word elimination are forms of lossy compression. Similarly, the vector space model (Chapter 6) and dimensionality reduction techniques like latent semantic indexing (Chapter 18) create compact representations from which we cannot fully restore the original collection. Lossy compression makes sense when the "lost" information is unlikely ever to be used by the search system. For example, web search is characterized by a large number of documents, short queries, and users who only look at the first few pages of results. As a consequence, we can discard postings of documents that would only be used for hits far down the list. Thus, there are retrieval scenarios where lossy methods can be used for compression without any reduction in effectiveness. Before introducing techniques for compressing the dictionary, we want to estimate the number of distinct terms M in a collection. It is sometimes said that languages have a vocabulary of a certain size. The second edition of the Oxford English Dictionary (OED) defines more than 600,000 words. But the vocabulary of most large collections is much larger than the OED. The OED does not include most names of people, locations, products, or scientific
−9 −9 −38 −52 −52
log10 M
log10 T
◮ Figure 5.1 Heaps' law. Vocabulary size M as a function of collection size T (number of tokens) for Reuters-RCV1. For these data, the dashed line log10 M = 0.49 ∗ log10 T + 1.64 is the best least-squares fit. Thus, k = 101.64 ≈ 44 and b = 0.49.
entities like genes. These names need to be included in the inverted index, so our users can search for them.
5.1.1 H EAPS ' LAW
Heaps' law: Estimating the number of terms A better way of getting a handle on M is Heaps' law, which estimates vocabulary size as a function of collection size: M = kT b where T is the number of tokens in the collection. Typical values for the parameters k and b are: 30 ≤ k ≤ 100 and b ≈ 0.5. The motivation for Heaps' law is that the simplest possible relationship between collection size and vocabulary size is linear in log–log space and the assumption of linearity is usually born out in practice as shown in Figure 5.1 for Reuters-RCV1. In this case, the fit is excellent for T > 105 = 100,000, for the parameter values b = 0.49 and k = 44. For example, for the first 1,000,020 tokens Heaps' law
predicts 38,323 terms: 44 × 1,000,0200.49 ≈ 38,323. The actual number is 38,365 terms, very close to the prediction. The parameter k is quite variable because vocabulary growth depends a lot on the nature of the collection and how it is processed. Case-folding and stemming reduce the growth rate of the vocabulary, whereas including numbers and spelling errors increase it. Regardless of the values of the parameters for a particular collection, Heaps' law suggests that (i) the dictionary size continues to increase with more documents in the collection, rather than a maximum vocabulary size being reached, and (ii) the size of the dictionary is quite large for large collections. These two hypotheses have been empirically shown to be true of large text collections (Section 5.4). So dictionary compression is important for an effective information retrieval system.
Z IPF ' S LAW
Zipf's law: Modeling the distribution of terms We also want to understand how terms are distributed across documents. This helps us to characterize the properties of the algorithms for compressing postings lists in Section 5.3. A commonly used model of the distribution of terms in a collection is Zipf's law. It states that, if t1 is the most common term in the collection, t2 is the next most common, and so on, then the collection frequency cfi of the ith most common term is proportional to 1/i: cfi ∝
1 . i
So if the most frequent term occurs cf1 times, then the second most frequent term has half as many occurrences, the third most frequent term a third as many occurrences, and so on. The intuition is that frequency decreases very rapidly with rank. Equation (5.2) is one of the simplest ways of formalizing such a rapid decrease and it has been found to be a reasonably good model. Equivalently, we can write Zipf's law as cfi = ci k or as log cfi = log c + k log i where k = −1 and c is a constant to be defined in Section 5.3.2. It is therefore a power law with exponent k = −1. See Chapter 19, page 426, for another power law, a law characterizing the distribution of links on web pages. The log–log graph in Figure 5.2 plots the collection frequency of a term as a function of its rank for Reuters-RCV1. A line with slope –1, corresponding to the Zipf function log cfi = log c − log i, is also shown. The fit of the data to the law is not particularly good, but good enough to serve as a model for term distributions in our calculations in Section 5.3.
log10 cf
log10 rank
◮ Figure 5.2 Zipf's law for Reuters-RCV1. Frequency is plotted as a function of frequency rank for the terms in the collection. The line is the distribution predicted by Zipf's law (weighted least-squares fit; intercept is 6.95).
Dictionary compression
Assuming one machine word per posting, what is the size of the uncompressed (nonpositional) index for different tokenizations based on Table 5.1? How do these numbers compare with Table 5.6?
This section presents a series of dictionary data structures that achieve increasingly higher compression ratios. The dictionary is small compared with the postings file as suggested by Table 5.1. So why compress it if it is responsible for only a small percentage of the overall space requirements of the IR system? One of the primary factors in determining the response time of an IR system is the number of disk seeks necessary to process a query. If parts of the dictionary are on disk, then many more disk seeks are necessary in query evaluation. Thus, the main goal of compressing the dictionary is to fit it in main memory, or at least a large portion of it, to support high query through-
5.2 Dictionary compression
space needed: ◮ Figure 5.3
a aachen ... zulu 20 bytes
document frequency 656,265 65 ... 221 4 bytes
pointer to postings list −→ −→ ... −→ 4 bytes
Storing the dictionary as an array of fixed-width entries.
put. Although dictionaries of very large collections fit into the memory of a standard desktop machine, this is not true of many other application scenarios. For example, an enterprise search server for a large corporation may have to index a multiterabyte collection with a comparatively large vocabulary because of the presence of documents in many different languages. We also want to be able to design search systems for limited hardware such as mobile phones and onboard computers. Other reasons for wanting to conserve memory are fast startup time and having to share resources with other applications. The search system on your PC must get along with the memory-hogging word processing suite you are using at the same time.
Dictionary as a string The simplest data structure for the dictionary is to sort the vocabulary lexicographically and store it in an array of fixed-width entries as shown in Figure 5.3. We allocate 20 bytes for the term itself (because few terms have more than twenty characters in English), 4 bytes for its document frequency, and 4 bytes for the pointer to its postings list. Four-byte pointers resolve a 4 gigabytes (GB) address space. For large collections like the web, we need to allocate more bytes per pointer. We look up terms in the array by binary search. For Reuters-RCV1, we need M × (20 + 4 + 4) = 400,000 × 28 = 11.2megabytes (MB) for storing the dictionary in this scheme. Using fixed-width entries for terms is clearly wasteful. The average length of a term in English is about eight characters (Table 4.2, page 70), so on average we are wasting twelve characters in the fixed-width scheme. Also, we have no way of storing terms with more than twenty characters like hydrochlorofluorocarbons and supercalifragilisticexpialidocious. We can overcome these shortcomings by storing the dictionary terms as one long string of characters, as shown in Figure 5.4. The pointer to the next term is also used to demarcate the end of the current term. As before, we locate terms in the data structure by way of binary search in the (now smaller) table. This scheme saves us 60% compared to fixed-width storage – 12 bytes on average of the
5 Index compression . . . s y s t i l e s yzyge t i c s yzyg i a l s yzygy s za i be l y i t e s z e c i ns zono . . .
freq.
postings ptr.
term ptr.
◮ Figure 5.4 Dictionary-as-a-string storage. Pointers mark the end of the preceding term and the beginning of the next. For example, the first three terms in this example are systile, syzygetic, and syzygial.
20 bytes we allocated for terms before. However, we now also need to store term pointers. The term pointers resolve 400,000 × 8 = 3.2 × 106 positions, so they need to be log2 3.2 × 106 ≈ 22 bits or 3 bytes long. In this new scheme, we need 400,000 × (4 + 4 + 3 + 8) = 7.6 MB for the Reuters-RCV1 dictionary: 4 bytes each for frequency and postings pointer, 3 bytes for the term pointer, and 8 bytes on average for the term. So we have reduced the space requirements by one third from 11.2 to 7.6 MB.
Blocked storage We can further compress the dictionary by grouping terms in the string into blocks of size k and keeping a term pointer only for the first term of each block (Figure 5.5). We store the length of the term in the string as an additional byte at the beginning of the term. We thus eliminate k − 1 term pointers, but need an additional k bytes for storing the length of each term. For k = 4, we save (k − 1) × 3 = 9 bytes for term pointers, but need an additional k = 4 bytes for term lengths. So the total space requirements for the dictionary of Reuters-RCV1 are reduced by 5 bytes per four-term block, or a total of 400,000 × 1/4 × 5 = 0.5 MB, bringing us down to 7.1 MB.
. . . 7 s y s t i l e 9 s y z y g e t i c 8 s y z y g i a l 6 s y z y g y11 s z a i b e l y i t e 6 s z e c i n . . .
◮ Figure 5.5 Blocked storage with four terms per block. The first block consists of systile, syzygetic, syzygial, and syzygy with lengths of seven, nine, eight, and six charac-
ters, respectively. Each term is preceded by a byte encoding its length that indicates how many bytes to skip to reach subsequent terms.
FRONT CODING
By increasing the block size k, we get better compression. However, there is a tradeoff between compression and the speed of term lookup. For the eight-term dictionary in Figure 5.6, steps in binary search are shown as double lines and steps in list search as simple lines. We search for terms in the uncompressed dictionary by binary search (a). In the compressed dictionary, we first locate the term's block by binary search and then its position within the list by linear search through the block (b). Searching the uncompressed dictionary in (a) takes on average (0 + 1 + 2 + 3 + 2 + 1 + 2 + 2)/8 ≈ 1.6 steps, assuming each term is equally likely to come up in a query. For example, finding the two terms, aid and box, takes three and two steps, respectively. With blocks of size k = 4 in (b), we need (0 + 1 + 2 + 3 + 4 + 1 + 2 + 3)/8 = 2 steps on average, ≈ 25% more. For example, finding den takes one binary search step and two steps through the block. By increasing k, we can get the size of the compressed dictionary arbitrarily close to the minimum of 400,000 × (4 + 4 + 1 + 8) = 6.8 MB, but term lookup becomes prohibitively slow for large values of k. One source of redundancy in the dictionary we have not exploited yet is the fact that consecutive entries in an alphabetically sorted list share common prefixes. This observation leads to front coding (Figure 5.7). A common prefix
aid box den ex job ox pit win
◮ Figure 5.6 Search of the uncompressed dictionary (a) and a dictionary compressed by blocking with k = 4 (b).
One block in blocked compression (k = 4) . . . 8automata8automate9au t omatic10automation
⇓ . . . further compressed with front coding. 8automat∗a1 ⋄ e2 ⋄ i c3⋄ i on ◮ Figure 5.7 Front coding. A sequence of terms with identical prefix ("automat") is encoded by marking the end of the prefix with ∗ and replacing it with ⋄ in subsequent terms. As before, the first byte of each entry encodes the number of characters.
5.3 Postings file compression
◮ Table 5.2 Dictionary compression for Reuters-RCV1.
data structure dictionary, fixed-width dictionary, term pointers into string ∼, with blocking, k = 4 ∼, with blocking & front coding
size in MB 11.2 7.6 7.1 5.9
is identified for a subsequence of the term list and then referred to with a special character. In the case of Reuters, front coding saves another 1.2 MB, as we found in an experiment. Other schemes with even greater compression rely on minimal perfect hashing, that is, a hash function that maps M terms onto [1, . . . , M ] without collisions. However, we cannot adapt perfect hashes incrementally because each new term causes a collision and therefore requires the creation of a new perfect hash function. Therefore, they cannot be used in a dynamic environment. Even with the best compression scheme, it may not be feasible to store the entire dictionary in main memory for very large text collections and for hardware with limited memory. If we have to partition the dictionary onto pages that are stored on disk, then we can index the first term of each page using a B-tree. For processing most queries, the search system has to go to disk anyway to fetch the postings. One additional seek for retrieving the term's dictionary page from disk is a significant, but tolerable increase in the time it takes to process a query. Table 5.2 summarizes the compression achieved by the four dictionary data structures.
Exercise 5.2 Estimate the space usage of the Reuters-RCV1 dictionary with blocks of size k = 8 and k = 16 in blocked dictionary storage. Exercise 5.3 Estimate the time needed for term lookup in the compressed dictionary of ReutersRCV1 with block sizes of k = 4 (Figure 5.6, b), k = 8, and k = 16. What is the slowdown compared with k = 1 (Figure 5.6, a)?
Postings file compression Recall from Table 4.2 (page 70) that Reuters-RCV1 has 800,000 documents, 200 tokens per document, six characters per token, and 100,000,000 postings where we define a posting in this chapter as a docID in a postings list, that is, excluding frequency and position information. These numbers
◮ Table 5.3 Encoding gaps instead of document IDs. For example, we store gaps 107, 5, 43, . . . , instead of docIDs 283154, 283159, 283202, . . . for computer. The first docID is left unchanged (only shown for arachnocentric). the computer arachnocentric
encoding docIDs gaps docIDs gaps docIDs gaps
postings list ...
283044 1 283159 5
correspond to line 3 ("case folding") in Table 5.1. Document identifiers are log2 800,000 ≈ 20 bits long. Thus, the size of the collection is about 800,000 × 200 × 6 bytes = 960 MB and the size of the uncompressed postings file is 100,000,000 × 20/8 = 250 MB. To devise a more efficient representation of the postings file, one that uses fewer than 20 bits per document, we observe that the postings for frequent terms are close together. Imagine going through the documents of a collection one by one and looking for a frequent term like computer. We will find a document containing computer, then we skip a few documents that do not contain it, then there is again a document with the term and so on (see Table 5.3). The key idea is that the gaps between postings are short, requiring a lot less space than 20 bits to store. In fact, gaps for the most frequent terms such as the and for are mostly equal to 1. But the gaps for a rare term that occurs only once or twice in a collection (e.g., arachnocentric in Table 5.3) have the same order of magnitude as the docIDs and need 20 bits. For an economical representation of this distribution of gaps, we need a variable encoding method that uses fewer bits for short gaps. To encode small numbers in less space than large numbers, we look at two types of methods: bytewise compression and bitwise compression. As the names suggest, these methods attempt to encode gaps with the minimum number of bytes and bits, respectively.
5.3.1 VARIABLE BYTE ENCODING CONTINUATION BIT
Variable byte codes Variable byte (VB) encoding uses an integral number of bytes to encode a gap. The last 7 bits of a byte are "payload" and encode part of the gap. The first bit of the byte is a continuation bit.It is set to 1 for the last byte of the encoded gap and to 0 otherwise. To decode a variable byte code, we read a sequence of bytes with continuation bit 0 terminated by a byte with continuation bit 1. We then extract and concatenate the 7-bit parts. Figure 5.8 gives pseudocode
283045 1 283202 43
VBE NCODE N UMBER (n) 1 bytes ← hi 2 while true 3 do P REPEND (bytes, n mod 128) 4 if n < 128 5 then B REAK 6 n ← n div 128 7 bytes[ L ENGTH (bytes)] += 128 8 return bytes VBE NCODE (numbers) 1 bytestream ← hi 2 for each n ∈ numbers 3 do bytes ← VBE NCODE N UMBER (n) 4 bytestream ← E XTEND (bytestream, bytes) 5 return bytestream VBD ECODE (bytestream) 1 numbers ← hi 2 n←0 3 for i ← 1 to L ENGTH (bytestream) 4 do if bytestream[i ] < 128 5 then n ← 128 × n + bytestream[i ] 6 else n ← 128 × n + (bytestream[i ] − 128) 7 A PPEND (numbers, n) 8 n←0 9 return numbers ◮ Figure 5.8 VB encoding and decoding. The functions div and mod compute integer division and remainder after integer division, respectively. P REPEND adds an element to the beginning of a list, for example, P REPEND (h1, 2i, 3) = h3, 1, 2i. E XTEND extends a list, for example, E XTEND (h1, 2i, h3, 4i) = h1, 2, 3, 4i.
◮ Table 5.4 VB encoding. Gaps are encoded using an integral number of bytes. The first bit, the continuation bit, of each byte indicates whether the code ends with this byte (1) or not (0).
docIDs gaps VB code
824 00000110 10111000
215406 214577 00001101 00001100 10110001
◮ Table 5.5 Some examples of unary and γ codes. Unary codes are only shown for the smaller numbers. Commas in γ codes are for readability only and are not part of the actual codes.
number 0 1 2 3 4 9 13 24 511 1025
unary code 0 10 110 1110 11110 1111111110
0 10 10 110 1110 1110 11110 111111110 11111111110
0 1 00 001 101 1000 11111111 0000000001
0 10,0 10,1 110,00 1110,001 1110,101 11110,1000 111111110,11111111 11111111110,0000000001
for VB encoding and decoding and Table 5.4 an example of a VB-encoded postings list. 1 With VB compression, the size of the compressed index for Reuters-RCV1 is 116 MB as we verified in an experiment. This is a more than 50% reduction of the size of the uncompressed index (see Table 5.6). The idea of VB encoding can also be applied to larger or smaller units than bytes: 32-bit words, 16-bit words, and 4-bit words or nibbles. Larger words further decrease the amount of bit manipulation necessary at the cost of less effective (or no) compression. Word sizes smaller than bytes get even better compression ratios at the cost of more bit manipulation. In general, bytes offer a good compromise between compression ratio and speed of decompression. For most IR systems variable byte codes offer an excellent tradeoff between time and space. They are also simple to implement – most of the alternatives referred to in Section 5.4 are more complex. But if disk space is a scarce resource, we can achieve better compression ratios by using bit-level encodings, in particular two closely related encodings: γ codes, which we will turn to next, and δ codes (Exercise 5.9).
γ codes VB codes use an adaptive number of bytes depending on the size of the gap. Bit-level codes adapt the length of the code on the finer grained bit level. The 1. Note that the origin is 0 in the table. Because we never need to encode a docID or a gap of 0, in practice the origin is usually 1, so that 10000000 encodes 1, 10000101 encodes 6 (not 5 as in the table), and so on.
UNARY CODE
γ ENCODING
simplest bit-level code is unary code. The unary code of n is a string of n 1s followed by a 0 (see the first two columns of Table 5.5). Obviously, this is not a very efficient code, but it will come in handy in a moment. How efficient can a code be in principle? Assuming the 2n gaps G with 1 ≤ G ≤ 2n are all equally likely, the optimal encoding uses n bits for each G. So some gaps (G = 2n in this case) cannot be encoded with fewer than log2 G bits. Our goal is to get as close to this lower bound as possible. A method that is within a factor of optimal is γ encoding. γ codes implement variable-length encoding by splitting the representation of a gap G into a pair of length and offset. Offset is G in binary, but with the leading 1 removed.2 For example, for 13 (binary 1101) offset is 101. Length encodes the length of offset in unary code. For 13, the length of offset is 3 bits, which is 1110 in unary. The γ code of 13 is therefore 1110101, the concatenation of length 1110 and offset 101. The right hand column of Table 5.5 gives additional examples of γ codes. A γ code is decoded by first reading the unary code up to the 0 that terminates it, for example, the four bits 1110 when decoding 1110101. Now we know how long the offset is: 3 bits. The offset 101 can then be read correctly and the 1 that was chopped off in encoding is prepended: 101 → 1101 = 13. The length of offset is ⌊log2 G ⌋ bits and the length of length is ⌊log2 G ⌋ + 1 bits, so the length of the entire code is 2 × ⌊log2 G ⌋ + 1 bits. γ codes are always of odd length and they are within a factor of 2 of what we claimed to be the optimal encoding length log2 G. We derived this optimum from the assumption that the 2n gaps between 1 and 2n are equiprobable. But this need not be the case. In general, we do not know the probability distribution over gaps a priori. The characteristic of a discrete probability distribution3 P that determines its coding properties (including whether a code is optimal) is its entropy H ( P), which is defined as follows: H ( P) = −
∑ x∈X
P( x ) log2 P( x )
where X is the set of all possible numbers we need to be able to encode (and therefore ∑ x ∈ X P( x ) = 1.0). Entropy is a measure of uncertainty as shown in Figure 5.9 for a probability distribution P over two possible outcomes, namely, X = { x1 , x2 }. Entropy is maximized (H ( P) = 1) for P( x1 ) = P( x2 ) = 0.5 when uncertainty about which xi will appear next is largest; and 2. We assume here that G has no leading 0s. If there are any, they are removed before deleting the leading 1. 3. Readers who want to review basic concepts of probability theory may want to consult Rice (2006) or Ross (2006). Note that we are interested in probability distributions over integers (gaps, frequencies, etc.), but that the coding properties of a probability distribution are independent of whether the outcomes are integers or something else.
H(P) 0.6
0.6 P(x1)
◮ Figure 5.9 Entropy H ( P ) as a function of P ( x1 ) for a sample space with two outcomes x1 and x2 .
minimized (H ( P) = 0) for P( x1 ) = 1, P( x2 ) = 0 and for P( x1 ) = 0, P( x2) = 1 when there is absolute certainty. It can be shown that the lower bound for the expected length E( L) of a code L is H ( P) if certain conditions hold (see the references). It can further be shown that for 1 < H ( P) < ∞, γ encoding is within a factor of 3 of this optimal encoding, approaching 2 for large H ( P): E( Lγ ) 1 ≤ 2+ ≤ 3. H ( P) H ( P)
UNIVERSAL CODE
PREFIX FREE
PARAMETER FREE
What is remarkable about this result is that it holds for any probability distribution P. So without knowing anything about the properties of the distribution of gaps, we can apply γ codes and be certain that they are within a factor of ≈ 2 of the optimal code for distributions of large entropy. A code like γ code with the property of being within a factor of optimal for an arbitrary distribution P is called universal. In addition to universality, γ codes have two other properties that are useful for index compression. First, they are prefix free, namely, no γ code is the prefix of another. This means that there is always a unique decoding of a sequence of γ codes – and we do not need delimiters between them, which would decrease the efficiency of the code. The second property is that γ codes are parameter free. For many other efficient codes, we have to fit the parameters of a model (e.g., the binomial distribution) to the distribution
of gaps in the index. This complicates the implementation of compression and decompression. For instance, the parameters need to be stored and retrieved. And in dynamic indexing, the distribution of gaps can change, so that the original parameters are no longer appropriate. These problems are avoided with a parameter-free code. How much compression of the inverted index do γ codes achieve? To answer this question we use Zipf's law, the term distribution model introduced in Section 5.1.2. According to Zipf's law, the collection frequency cfi is proportional to the inverse of the rank i, that is, there is a constant c′ such that: cfi =
c′ . i
We can choose a different constant c such that the fractions c/i are relative frequencies and sum to 1 (that is, c/i = cfi /T): (5.4) (5.5)
M c 1 =c∑ i i i =1 i =1
c=
= c HM
where M is the number of distinct terms and H M is the Mth harmonic number. 4 Reuters-RCV1 has M = 400,000 distinct terms and H M ≈ ln M, so we have 1 1 1 1 ≈ = ≈ . c= HM ln M ln 400,000 13 Thus the ith term has a relative frequency of roughly 1/(13i ), and the expected average number of occurrences of term i in a document of length L is: 1 200 × 13 15 c ≈ L ≈ i i i where we interpret the relative frequency as a term occurrence probability. Recall that 200 is the average number of tokens per document in ReutersRCV1 (Table 4.2). Now we have derived term statistics that characterize the distribution of terms in the collection and, by extension, the distribution of gaps in the postings lists. From these statistics, we can calculate the space requirements for an inverted index compressed with γ encoding. We first stratify the vocabulary into blocks of size Lc = 15. On average, term i occurs 15/i times per 4. Note that, unfortunately, the conventional symbol for both entropy and harmonic number is H. Context should make clear which is meant in this chapter.
N documents Lc most frequent terms Lc next most frequent terms Lc next most frequent terms ... ◮ Figure 5.10 index.
N gaps of 1 each
N/2 gaps of 2 each
N/3 gaps of 3 each ...
Stratification of terms for estimating the size of a γ encoded inverted
document. So the average number of occurrences f per document is 1 ≤ f for terms in the first block, corresponding to a total number of N gaps per term. The average is 21 ≤ f < 1 for terms in the second block, corresponding to N/2 gaps per term, and 31 ≤ f < 21 for terms in the third block, corresponding to N/3 gaps per term, and so on. (We take the lower bound because it simplifies subsequent calculations. As we will see, the final estimate is too pessimistic, even with this assumption.) We will make the somewhat unrealistic assumption that all gaps for a given term have the same size as shown in Figure 5.10. Assuming such a uniform distribution of gaps, we then have gaps of size 1 in block 1, gaps of size 2 in block 2, and so on. Encoding the N/j gaps of size j with γ codes, the number of bits needed for the postings list of a term in the jth block (corresponding to one row in the figure) is: bits-per-row
N × (2 × ⌊log2 j⌋ + 1) j 2N log2 j . j
= ≈
To encode the entire block, we need ( Lc) · (2N log2 j)/j bits. There are M/( Lc) blocks, so the postings file as a whole will take up:
M Lc
2NLc log2 j . j j =1
◮ Table 5.6 Index and dictionary compression for Reuters-RCV1. The compression ratio depends on the proportion of actual text in the collection. Reuters-RCV1 contains a large amount of XML markup. Using the two best compression schemes, γ encoding and blocking with front coding, the ratio compressed index to collection size is therefore especially small for Reuters-RCV1: (101 + 5.9)/3600 ≈ 0.03.
data structure dictionary, fixed-width dictionary, term pointers into string ∼, with blocking, k = 4 ∼, with blocking & front coding collection (text, xml markup etc) collection (text) term incidence matrix postings, uncompressed (32-bit words) postings, uncompressed (20 bits) postings, variable byte encoded postings, γ encoded
For Reuters-RCV1, (5.7)
size in MB 11.2 7.6 7.1 5.9 3600.0 960.0 40,000.0 400.0 250.0 116.0 101.0
≈ 400,000/15 ≈ 27,000 and
∑ j =1
2 × 106 × 15 log2 j ≈ 224 MB. j
So the postings file of the compressed inverted index for our 960 MB collection has a size of 224 MB, one fourth the size of the original collection. When we run γ compression on Reuters-RCV1, the actual size of the compressed index is even lower: 101 MB, a bit more than one tenth of the size of the collection. The reason for the discrepancy between predicted and actual value is that (i) Zipf's law is not a very good approximation of the actual distribution of term frequencies for Reuters-RCV1 and (ii) gaps are not uniform. The Zipf model predicts an index size of 251 MB for the unrounded numbers from Table 4.2. If term frequencies are generated from the Zipf model and a compressed index is created for these artificial terms, then the compressed size is 254 MB. So to the extent that the assumptions about the distribution of term frequencies are accurate, the predictions of the model are correct. Table 5.6 summarizes the compression techniques covered in this chapter. The term incidence matrix (Figure 1.1, page 4) for Reuters-RCV1 has size 400,000 × 800,000 = 40 × 8 × 109 bits or 40 GB. γ codes achieve great compression ratios – about 15% better than variable byte codes for Reuters-RCV1. But they are expensive to decode. This is because many bit-level operations – shifts and masks – are necessary to decode a sequence of γ codes as the boundaries between codes will usually be
somewhere in the middle of a machine word. As a result, query processing is more expensive for γ codes than for variable byte codes. Whether we choose variable byte or γ encoding depends on the characteristics of an application, for example, on the relative weights we give to conserving disk space versus maximizing query response time. The compression ratio for the index in Table 5.6 is about 25%: 400 MB (uncompressed, each posting stored as a 32-bit word) versus 101 MB (γ) and 116 MB (VB). This shows that both γ and VB codes meet the objectives we stated in the beginning of the chapter. Index compression substantially improves time and space efficiency of indexes by reducing the amount of disk space needed, increasing the amount of information that can be kept in the cache, and speeding up data transfers from disk to memory.
Compute variable byte codes for the numbers in Tables 5.3 and 5.5. Exercise 5.5
Compute variable byte and γ codes for the postings list h 777, 17743, 294068, 31251336i. Use gaps instead of docIDs where possible. Write binary codes in 8-bit blocks. Exercise 5.6 Consider the postings list h4, 10, 11, 12, 15, 62, 63, 265, 268, 270, 400i with a corresponding list of gaps h4, 6, 1, 1, 3, 47, 1, 202, 3, 2, 130i. Assume that the length of the postings list is stored separately, so the system knows when a postings list is complete. Using variable byte encoding: (i) What is the largest gap you can encode in 1 byte? (ii) What is the largest gap you can encode in 2 bytes? (iii) How many bytes will the above postings list require under this encoding? (Count only space for encoding the sequence of numbers.) Exercise 5.7 A little trick is to notice that a gap cannot be of length 0 and that the stuff left to encode after shifting cannot be 0. Based on these observations: (i) Suggest a modification to variable byte encoding that allows you to encode slightly larger gaps in the same amount of space. (ii) What is the largest gap you can encode in 1 byte? (iii) What is the largest gap you can encode in 2 bytes? (iv) How many bytes will the postings list in Exercise 5.6 require under this encoding? (Count only space for encoding the sequence of numbers.) Exercise 5.8
From the following sequence of γ-coded gaps, reconstruct first the gap sequence and then the postings sequence: 1110001110101011111101101111011. Exercise 5.9 δ CODES
γ codes are relatively inefficient for large numbers (e.g., 1025 in Table 5.5) as they encode the length of the offset in inefficient unary code. δ codes differ from γ codes in that they encode the first part of the code (length) in γ code instead of unary code. The encoding of offset is the same. For example, the δ code of 7 is 10,0,11 (again, we add commas for readability). 10,0 is the γ code for length (2 in this case) and the encoding of offset (11) is unchanged. (i) Compute the δ codes for the other numbers
◮ Table 5.7
Two gap sequences to be merged in blocked sort-based indexing
γ encoded gap sequence of run 1 γ encoded gap sequence of run 2
1110110111111001011111111110100011111001 11111010000111111000100011111110010000011111010101
in Table 5.5. For what range of numbers is the δ code shorter than the γ code? (ii) γ code beats variable byte code in Table 5.6 because the index contains stop words and thus many small gaps. Show that variable byte code is more compact if larger gaps dominate. (iii) Compare the compression ratios of δ code and variable byte code for a distribution of gaps dominated by large gaps. Exercise 5.10 Go through the above calculation of index size and explicitly state all the approximations that were made to arrive at Equation (5.6). Exercise 5.11 For a collection of your choosing, determine the number of documents and terms and the average length of a document. (i) How large is the inverted index predicted to be by Equation (5.6)? (ii) Implement an indexer that creates a γ-compressed inverted index for the collection. How large is the actual index? (iii) Implement an indexer that uses variable byte encoding. How large is the variable byte encoded index? Exercise 5.12 To be able to hold as many postings as possible in main memory, it is a good idea to compress intermediate index files during index construction. (i) This makes merging runs in blocked sort-based indexing more complicated. As an example, work out the γ-encoded merged sequence of the gaps in Table 5.7. (ii) Index construction is more space efficient when using compression. Would you also expect it to be faster? Exercise 5.13 (i) Show that the size of the vocabulary is finite according to Zipf's law and infinite according to Heaps' law. (ii) Can we derive Heaps' law from Zipf's law?
References and further reading Heaps' law was discovered by Heaps (1978). See also Baeza-Yates and RibeiroNeto (1999). A detailed study of vocabulary growth in large collections is (Williams and Zobel 2005). Zipf's law is due to Zipf (1949). Witten and Bell (1990) investigate the quality of the fit obtained by the law. Other term distribution models, including K mixture and two-poisson model, are discussed by Manning and Schütze (1999, Chapter 15). Carmel et al. (2001), Büttcher and Clarke (2006), Blanco and Barreiro (2007), and Ntoulas and Cho (2007) show that lossy compression can achieve good compression with no or no significant decrease in retrieval effectiveness. Dictionary compression is covered in detail by Witten et al. (1999, Chapter 4), which is recommended as additional reading.
PARAMETERIZED CODE
G OLOMB CODES
Subsection 5.3.1 is based on (Scholer et al. 2002). The authors find that variable byte codes process queries two times faster than either bit-level compressed indexes or uncompressed indexes with a 30% penalty in compression ratio compared with the best bit-level compression method. They also show that compressed indexes can be superior to uncompressed indexes not only in disk usage, but also in query processing speed. Compared with VB codes, "variable nibble" codes showed 5% to 10% better compression and up to one third worse effectiveness in one experiment (Anh and Moffat 2005). Trotman (2003) also recommends using VB codes unless disk space is at a premium. In recent work, Anh and Moffat (2005; 2006a) and Zukowski et al. (2006) have constructed word-aligned binary codes that are both faster in decompression and at least as efficient as VB codes. Zhang et al. (2007) investigate the increased effectiveness of caching when a number of different compression techniques for postings lists are used on modern hardware. δ codes (Exercise 5.9) and γ codes were introduced by Elias (1975), who proved that both codes are universal. In addition, δ codes are asymptotically optimal for H ( P) → ∞. δ codes perform better than γ codes if large numbers (greater than 15) dominate. A good introduction to information theory, including the concept of entropy, is (Cover and Thomas 1991). While Elias codes are only asymptotically optimal, arithmetic codes (Witten et al. 1999, Section 2.4) can be constructed to be arbitrarily close to the optimum H ( P) for any P. Several additional index compression techniques are covered by Witten et al. (1999; Sections 3.3 and 3.4 and Chapter 5). They recommend using parameterized codes for index compression, codes that explicitly model the probability distribution of gaps for each term. For example, they show that Golomb codes achieve better compression ratios than γ codes for large collections. Moffat and Zobel (1992) compare several parameterized methods, including LLRUN (Fraenkel and Klein 1985). The distribution of gaps in a postings list depends on the assignment of docIDs to documents. A number of researchers have looked into assigning docIDs in a way that is conducive to the efficient compression of gap sequences (Moffat and Stuiver 1996; Blandford and Blelloch 2002; Silvestri et al. 2004; Blanco and Barreiro 2006; Silvestri 2007). These techniques assign docIDs in a small range to documents in a cluster where a cluster can consist of all documents in a given time period, on a particular web site, or sharing another property. As a result, when a sequence of documents from a cluster occurs in a postings list, their gaps are small and can be more effectively compressed. Different considerations apply to the compression of term frequencies and word positions than to the compression of docIDs in postings lists. See Scholer et al. (2002) and Zobel and Moffat (2006). Zobel and Moffat (2006) is recommended in general as an in-depth and up-to-date tutorial on inverted
indexes, including index compression. This chapter only looks at index compression for Boolean retrieval. For ranked retrieval (Chapter 6), it is advantageous to order postings according to term frequency instead of docID. During query processing, the scanning of many postings lists can then be terminated early because smaller weights do not change the ranking of the highest ranked k documents found so far. It is not a good idea to precompute and store weights in the index (as opposed to frequencies) because they cannot be compressed as well as integers (see Section 7.1.5, page 140). Document compression can also be important in an efficient information retrieval system. de Moura et al. (2000) and Brisaboa et al. (2007) describe compression schemes that allow direct searching of terms and phrases in the compressed text, which is infeasible with standard text compression utilities like gzip and compress.
Exercise 5.14 [ ⋆] We have defined unary codes as being "10": sequences of 1s terminated by a 0. Interchanging the roles of 0s and 1s yields an equivalent "01" unary code. When this 01 unary code is used, the construction of a γ code can be stated as follows: (1) Write G down in binary using b = ⌊log2 j⌋ + 1 bits. (2) Prepend (b − 1) 0s. (i) Encode the numbers in Table 5.5 in this alternative γ code. (ii) Show that this method produces a well-defined alternative γ code in the sense that it has the same length and can be uniquely decoded. Exercise 5.15 [ ⋆ ⋆ ⋆] Unary code is not a universal code in the sense defined above. However, there exists a distribution over gaps for which unary code is optimal. Which distribution is this? Exercise 5.16 Give some examples of terms that violate the assumption that gaps all have the same size (which we made when estimating the space requirements of a γ-encoded index). What are general characteristics of these terms? Exercise 5.17 Consider a term whose postings list has size n, say, n = 10,000. Compare the size of the γ-compressed gap-encoded postings list if the distribution of the term is uniform (i.e., all gaps have the same size) versus its size when the distribution is not uniform. Which compressed postings list is smaller? Exercise 5.18 Work out the sum in Equation (5.7) and show it adds up to about 251 MB. Use the numbers in Table 4.2, but do not round Lc, c, and the number of vocabulary blocks.
Scoring, term weighting and the vector space model
Thus far we have dealt with indexes that support Boolean queries: a document either matches or does not match a query. In the case of large document collections, the resulting number of matching documents can far exceed the number a human user could possibly sift through. Accordingly, it is essential for a search engine to rank-order the documents matching a query. To do this, the search engine computes, for each matching document, a score with respect to the query at hand. In this chapter we initiate the study of assigning a score to a (query, document) pair. This chapter consists of three main ideas. 1. We introduce parametric and zone indexes in Section 6.1, which serve two purposes. First, they allow us to index and retrieve documents by metadata such as the language in which a document is written. Second, they give us a simple means for scoring (and thereby ranking) documents in response to a query. 2. Next, in Section 6.2 we develop the idea of weighting the importance of a term in a document, based on the statistics of occurrence of the term. 3. In Section 6.3 we show that by viewing each document as a vector of such weights, we can compute a score between a query and each document. This view is known as vector space scoring. Section 6.4 develops several variants of term-weighting for the vector space model. Chapter 7 develops computational aspects of vector space scoring, and related topics. As we develop these ideas, the notion of a query will assume multiple nuances. In Section 6.1 we consider queries in which specific query terms occur in specified regions of a matching document. Beginning Section 6.2 we will in fact relax the requirement of matching specific regions of a document; instead, we will look at so-called free text queries that simply consist of query terms with no specification on their relative order, importance or where in a document they should be found. The bulk of our study of scoring will be in this latter notion of a query being such a set of terms.
6 Scoring, term weighting and the vector space model
PARAMETRIC INDEX
WEIGHTED ZONE SCORING
Parametric and zone indexes
We have thus far viewed a document as a sequence of terms. In fact, most documents have additional structure. Digital documents generally encode, in machine-recognizable form, certain metadata associated with each document. By metadata, we mean specific forms of data about a document, such as its author(s), title and date of publication. This metadata would generally include fields such as the date of creation and the format of the document, as well the author and possibly the title of the document. The possible values of a field should be thought of as finite – for instance, the set of all dates of authorship. Consider queries of the form "find documents authored by William Shakespeare in 1601, containing the phrase alas poor Yorick". Query processing then consists as usual of postings intersections, except that we may merge postings from standard inverted as well as parametric indexes. There is one parametric index for each field (say, date of creation); it allows us to select only the documents matching a date specified in the query. Figure 6.1 illustrates the user's view of such a parametric search. Some of the fields may assume ordered values, such as dates; in the example query above, the year 1601 is one such field value. The search engine may support querying ranges on such ordered values; to this end, a structure like a B-tree may be used for the field's dictionary. Zones are similar to fields, except the contents of a zone can be arbitrary free text. Whereas a field may take on a relatively small set of values, a zone can be thought of as an arbitrary, unbounded amount of text. For instance, document titles and abstracts are generally treated as zones. We may build a separate inverted index for each zone of a document, to support queries such as "find documents with merchant in the title and william in the author list and the phrase gentle rain in the body". This has the effect of building an index that looks like Figure 6.2. Whereas the dictionary for a parametric index comes from a fixed vocabulary (the set of languages, or the set of dates), the dictionary for a zone index must structure whatever vocabulary stems from the text of that zone. In fact, we can reduce the size of the dictionary by encoding the zone in which a term occurs in the postings. In Figure 6.3 for instance, we show how occurrences of william in the title and author zones of various documents are encoded. Such an encoding is useful when the size of the dictionary is a concern (because we require the dictionary to fit in main memory). But there is another important reason why the encoding of Figure 6.3 is useful: the efficient computation of scores using a technique we will call weighted zone scoring.
6.1 Parametric and zone indexes
◮ Figure 6.1 Parametric search. In this example we have a collection with fields allowing us to select publications by zones such as Author and fields such as Language.
william.abstract
william.title
william.author
◮ Figure 6.2 Basic zone index ; zones are encoded as extensions of dictionary entries.
- 2.author,2.title
- 3.author
4.title
◮ Figure 6.3 Zone index in which the zone is encoded in the postings rather than the dictionary.
Weighted zone scoring Thus far in Section 6.1 we have focused on retrieving documents based on Boolean queries on fields and zones. We now turn to a second application of zones and fields. Given a Boolean query q and a document d, weighted zone scoring assigns to the pair (q, d) a score in the interval [0, 1], by computing a linear combination of zone scores, where each zone of the document contributes a Boolean value. More specifically, consider a set of documents each of which has ℓ zones. Let g1 , . . . , gℓ ∈ [0, 1] such that ∑ℓi=1 gi = 1. For 1 ≤ i ≤ ℓ, let si be the Boolean score denoting a match (or absence thereof) between q and the ith zone. For instance, the Boolean score from a zone could be 1 if all the query term(s) occur in that zone, and zero otherwise; indeed, it could be any Boolean function that maps the presence of query terms in a zone to 0, 1. Then, the weighted zone score is defined to be ℓ
∑ gi si .
R ANKED B OOLEAN RETRIEVAL
Weighted zone scoring is sometimes referred to also as ranked Boolean retrieval. Consider the query shakespeare in a collection in which each document has three zones: author, title and body. The Boolean score function for a zone takes on the value 1 if the query term shakespeare is present in the zone, and zero otherwise. Weighted zone scoring in such a collection would require three weights g1 , g2 and g3 , respectively corresponding to the author, title and body zones. Suppose we set g1 = 0.2, g2 = 0.3 and g3 = 0.5 (so that the three weights add up to 1); this corresponds to an application in which a match in the author zone is least important to the overall score, the title zone somewhat more, and the body contributes even more. Thus if the term shakespeare were to appear in the title and body zones but not the author zone of a document, the score of this document would be 0.8.
Example 6.1:
How do we implement the computation of weighted zone scores? A simple approach would be to compute the score for each document in turn, adding in all the contributions from the various zones. However, we now show how we may compute weighted zone scores directly from inverted indexes. The algorithm of Figure 6.4 treats the case when the query q is a twoterm query consisting of query terms q1 and q2 , and the Boolean function is AND: 1 if both query terms are present in a zone and 0 otherwise. Following the description of the algorithm, we describe the extension to more complex queries and Boolean functions. The reader may have noticed the close similarity between this algorithm and that in Figure 1.6. Indeed, they represent the same postings traversal, except that instead of merely adding a document to the set of results for
Z ONE S CORE (q1 , q2 ) 1 float scores[ N ] = [0] 2 constant g[ℓ] 3 p1 ← postings(q1 ) 4 p2 ← postings(q2 ) 5 // scores[] is an array with a score entry for each document, initialized to zero. 6 //p1 and p2 are initialized to point to the beginning of their respective postings. 7 //Assume g[] is initialized to the respective zone weights. 8 while p1 6= NIL and p2 6= NIL 9 do if docID ( p1) = docID ( p2) 10 then scores[docID ( p1 )] ← W EIGHTED Z ONE ( p1, p2 , g) 11 p1 ← next( p1 ) 12 p2 ← next( p2 ) 13 else if docID ( p1) < docID ( p2) 14 then p1 ← next( p1 ) 15 else p2 ← next( p2 ) 16 return scores ◮ Figure 6.4 Algorithm for computing the weighted zone score from two postings lists. Function W EIGHTED Z ONE (not shown here) is assumed to compute the inner loop of Equation 6.1.
MACHINE - LEARNED RELEVANCE
a Boolean AND query, we now compute a score for each such document. Some literature refers to the array scores[] above as a set of accumulators. The reason for this will be clear as we consider more complex Boolean functions than the AND; thus we may assign a non-zero score to a document even if it does not contain all query terms.
Learning weights How do we determine the weights gi for weighted zone scoring? These weights could be specified by an expert (or, in principle, the user); but increasingly, these weights are "learned" using training examples that have been judged editorially. This latter methodology falls under a general class of approaches to scoring and ranking in information retrieval, known as machine-learned relevance. We provide a brief introduction to this topic here because weighted zone scoring presents a clean setting for introducing it; a complete development demands an understanding of machine learning and is deferred to Chapter 15. 1. We are provided with a set of training examples, each of which is a tuple consisting of a query q and a document d, together with a relevance
judgment for d on q. In the simplest form, each relevance judgments is either Relevant or Non-relevant. More sophisticated implementations of the methodology make use of more nuanced judgments. 2. The weights gi are then "learned" from these examples, in order that the learned scores approximate the relevance judgments in the training examples. For weighted zone scoring, the process may be viewed as learning a linear function of the Boolean match scores contributed by the various zones. The expensive component of this methodology is the labor-intensive assembly of user-generated relevance judgments from which to learn the weights, especially in a collection that changes frequently (such as the Web). We now detail a simple example that illustrates how we can reduce the problem of learning the weights gi to a simple optimization problem. We now consider a simple case of weighted zone scoring, where each document has a title zone and a body zone. Given a query q and a document d, we use the given Boolean match function to compute Boolean variables s T (d, q) and s B (d, q), depending on whether the title (respectively, body) zone of d matches query q. For instance, the algorithm in Figure 6.4 uses an AND of the query terms for this Boolean function. We will compute a score between 0 and 1 for each (document, query) pair using s T (d, q) and s B (d, q) by using a constant g ∈ [0, 1], as follows: (6.2)
score(d, q) = g · s T (d, q) + (1 − g)s B (d, q). We now describe how to determine the constant g from a set of training examples, each of which is a triple of the form Φ j = (d j , q j , r (d j , q j )). In each training example, a given training document d j and a given training query q j are assessed by a human editor who delivers a relevance judgment r (d j , q j ) that is either Relevant or Non-relevant. This is illustrated in Figure 6.5, where seven training examples are shown. For each training example Φ j we have Boolean values s T (d j , q j ) and s B (d j , q j ) that we use to compute a score from (6.2)
score(d j , q j ) = g · s T (d j , q j ) + (1 − g)s B (d j , q j ). We now compare this computed score to the human relevance judgment for the same document-query pair (d j , q j ); to this end, we will quantize each Relevant judgment as a 1 and each Non-relevant judgment as a 0. Suppose that we define the error of the scoring function with weight g as ε( g, Φ j ) = (r (d j , q j ) − score(d j , q j ))2 ,
Example Φ1 Φ2 Φ3 Φ4 Φ5 Φ6 Φ7
DocID 37 37 238 238 1741 2094 3191
Query linux penguin system penguin kernel driver driver
sT 1 0 0 0 1 0 1
sB 1 1 1 0 1 1 0
Judgment Relevant Non-relevant Relevant Non-relevant Relevant Relevant Non-relevant
◮ Figure 6.5 An illustration of training examples.
sT 0 0 1 1
sB 0 1 0 1
Score 0 1−g g 1
◮ Figure 6.6 The four possible combinations of s T and s B .
where we have quantized the editorial relevance judgment r (d j , q j ) to 0 or 1. Then, the total error of a set of training examples is given by (6.4)
∑ ε( g, Φ j ). j
The problem of learning the constant g from the given training examples then reduces to picking the value of g that minimizes the total error in (6.4). Picking the best value of g in (6.4) in the formulation of Section 6.1.3 reduces to the problem of minimizing a quadratic function of g over the interval [0, 1]. This reduction is detailed in Section 6.1.3.
The optimal weight g We begin by noting that for any training example Φ j for which s T (d j , q j ) = 0 and s B (d j , q j ) = 1, the score computed by Equation (6.2) is 1 − g. In similar fashion, we may write down the score computed by Equation (6.2) for the three other possible combinations of s T (d j , q j ) and s B (d j , q j ); this is summarized in Figure 6.6. Let n01r (respectively, n01n ) denote the number of training examples for which s T (d j , q j ) = 0 and s B (d j , q j ) = 1 and the editorial judgment is Relevant (respectively, Non-relevant). Then the contribution to the total error in Equation (6.4) from training examples for which s T (d j , q j ) = 0 and s B (d j , q j ) = 1
[1 − (1 − g)]2 n01r + [0 − (1 − g)]2 n01n . By writing in similar fashion the error contributions from training examples of the other three combinations of values for s T (d j , q j ) and s B (d j , q j ) (and extending the notation in the obvious manner), the total error corresponding to Equation (6.4) is
(n01r + n10n ) g2 + (n10r + n01n )(1 − g)2 + n00r + n11n .
By differentiating Equation (6.5) with respect to g and setting the result to zero, it follows that the optimal value of g is n10r + n01n . n10r + n10n + n01r + n01n
Exercise 6.1 When using weighted zone scoring, is it necessary for all zones to use the same Boolean match function? Exercise 6.2 In Example 6.1 above with weights g1 = 0.2, g2 = 0.31 and g3 = 0.49, what are all the distinct score values a document may get? Exercise 6.3 Rewrite the algorithm in Figure 6.4 to the case of more than two query terms. Exercise 6.4 Write pseudocode for the function WeightedZone for the case of two postings lists in Figure 6.4. Exercise 6.5 Apply Equation 6.6 to the sample training set in Figure 6.5 to estimate the best value of g for this sample. Exercise 6.6 For the value of g estimated in Exercise 6.5, compute the weighted zone score for each (query, document) example. How do these scores relate to the relevance judgments in Figure 6.5 (quantized to 0/1)? Exercise 6.7 Why does the expression for g in (6.6) not involve training examples in which s T (dt , q t ) and s B (dt , q t ) have the same value?
6.2 Term frequency and weighting
Term frequency and weighting Thus far, scoring has hinged on whether or not a query term is present in a zone within a document. We take the next logical step: a document or zone that mentions a query term more often has more to do with that query and therefore should receive a higher score. To motivate this, we recall the notion of a free text query introduced in Section 1.4: a query in which the terms of the query are typed freeform into the search interface, without any connecting search operators (such as Boolean operators). This query style, which is extremely popular on the web, views the query as simply a set of words. A plausible scoring mechanism then is to compute a score that is the sum, over the query terms, of the match scores between each query term and the document. Towards this end, we assign to each term in a document a weight for that term, that depends on the number of occurrences of the term in the document. We would like to compute a score between a query term t and a document d, based on the weight of t in d. The simplest approach is to assign the weight to be equal to the number of occurrences of term t in document d. This weighting scheme is referred to as term frequency and is denoted tft,d , with the subscripts denoting the term and the document in order. For a document d, the set of weights determined by the tf weights above (or indeed any weighting function that maps the number of occurrences of t in d to a positive real value) may be viewed as a quantitative digest of that document. In this view of a document, known in the literature as the bag of words model, the exact ordering of the terms in a document is ignored but the number of occurrences of each term is material (in contrast to Boolean retrieval). We only retain information on the number of occurrences of each term. Thus, the document "Mary is quicker than John" is, in this view, identical to the document "John is quicker than Mary". Nevertheless, it seems intuitive that two documents with similar bag of words representations are similar in content. We will develop this intuition further in Section 6.3. Before doing so we first study the question: are all words in a document equally important? Clearly not; in Section 2.2.2 (page 27) we looked at the idea of stop words – words that we decide not to index at all, and therefore do not contribute in any way to retrieval and scoring.
Inverse document frequency Raw term frequency as above suffers from a critical problem: all terms are considered equally important when it comes to assessing relevancy on a query. In fact certain terms have little or no discriminating power in determining relevance. For instance, a collection of documents on the auto industry is likely to have the term auto in almost every document. To this
Word try insurance
cf 10422 10440
df 8760 3997
◮ Figure 6.7 Collection frequency (cf) and document frequency (df) behave differently, as in this example from the Reuters collection.
INVERSE DOCUMENT FREQUENCY
end, we introduce a mechanism for attenuating the effect of terms that occur too often in the collection to be meaningful for relevance determination. An immediate idea is to scale down the term weights of terms with high collection frequency, defined to be the total number of occurrences of a term in the collection. The idea would be to reduce the tf weight of a term by a factor that grows with its collection frequency. Instead, it is more commonplace to use for this purpose the document frequency dft , defined to be the number of documents in the collection that contain a term t. This is because in trying to discriminate between documents for the purpose of scoring it is better to use a document-level statistic (such as the number of documents containing a term) than to use a collection-wide statistic for the term. The reason to prefer df to cf is illustrated in Figure 6.7, where a simple example shows that collection frequency (cf) and document frequency (df) can behave rather differently. In particular, the cf values for both try and insurance are roughly equal, but their df values differ significantly. Intuitively, we want the few documents that contain insurance to get a higher boost for a query on insurance than the many documents containing try get from a query on try. How is the document frequency df of a term used to scale its weight? Denoting as usual the total number of documents in a collection by N, we define the inverse document frequency (idf) of a term t as follows: idft = log
N . dft
Thus the idf of a rare term is high, whereas the idf of a frequent term is likely to be low. Figure 6.8 gives an example of idf's in the Reuters collection of 806,791 documents; in this example logarithms are to the base 10. In fact, as we will see in Exercise 6.12, the precise base of the logarithm is not material to ranking. We will give on page 227 a justification of the particular form in Equation (6.7).
Tf-idf weighting We now combine the definitions of term frequency and inverse document frequency, to produce a composite weight for each term in each document.
term car auto insurance best
dft 18,165 6723 19,241 25,235
idft 1.65 2.08 1.62 1.5
◮ Figure 6.8 Example of idf values. Here we give the idf's of terms with various frequencies in the Reuters collection of 806,791 documents.
TF - IDF
The tf-idf weighting scheme assigns to term t a weight in document d given by
tf-idft,d = tft,d × idft .
In other words, tf-idft,d assigns to term t a weight in document d that is 1. highest when t occurs many times within a small number of documents (thus lending high discriminating power to those documents); 2. lower when the term occurs fewer times in a document, or occurs in many documents (thus offering a less pronounced relevance signal); 3. lowest when the term occurs in virtually all documents. DOCUMENT VECTOR
At this point, we may view each document as a vector with one component corresponding to each term in the dictionary, together with a weight for each component that is given by (6.8). For dictionary terms that do not occur in a document, this weight is zero. This vector form will prove to be crucial to scoring and ranking; we will develop these ideas in Section 6.3. As a first step, we introduce the overlap score measure: the score of a document d is the sum, over all query terms, of the number of times each of the query terms occurs in d. We can refine this idea so that we add up not the number of occurrences of each query term t in d, but instead the tf-idf weight of each term in d. Score(q, d) = ∑ tf-idft,d . t∈q
In Section 6.3 we will develop a more rigorous form of Equation (6.9).
Exercise 6.8 Why is the idf of a term always finite? Exercise 6.9 What is the idf of a term that occurs in every document? Compare this with the use of stop word lists.
car auto insurance best
Doc1 27 3 0 14
Doc2 4 33 33 0
Doc3 24 0 29 17
◮ Figure 6.9 Table of tf values for Exercise 6.10.
Exercise 6.10 Consider the table of term frequencies for 3 documents denoted Doc1, Doc2, Doc3 in Figure 6.9. Compute the tf-idf weights for the terms car, auto, insurance, best, for each document, using the idf values from Figure 6.8. Exercise 6.11 Can the tf-idf weight of a term in a document exceed 1? Exercise 6.12 How does the base of the logarithm in (6.7) affect the score calculation in (6.9)? How does the base of the logarithm affect the relative scores of two documents on a given query? Exercise 6.13 If the logarithm in (6.7) is computed base 2, suggest a simple approximation to the idf of a term.
VECTOR SPACE MODEL
The vector space model for scoring In Section 6.2 (page 117) we developed the notion of a document vector that captures the relative importance of the terms in a document. The representation of a set of documents as vectors in a common vector space is known as the vector space model and is fundamental to a host of information retrieval operations ranging from scoring documents on a query, document classification and document clustering. We first develop the basic ideas underlying vector space scoring; a pivotal step in this development is the view (Section 6.3.2) of queries as vectors in the same vector space as the document collection.
Dot products ~ (d) the vector derived from document d, with one comWe denote by V ponent in the vector for each dictionary term. Unless otherwise specified, the reader may assume that the components are computed using the tf-idf weighting scheme, although the particular weighting scheme is immaterial to the discussion that follows. The set of documents in a collection then may be viewed as a set of vectors in a vector space, in which there is one axis for
6.3 The vector space model for scoring
gossip ~v(d1 ) 1
~v(q) ~v(d2 ) θ
~v(d3 ) jealous 1
◮ Figure 6.10 Cosine similarity illustrated. sim(d1 , d2 ) = cos θ.
(6.10) DOT PRODUCT
E UCLIDEAN LENGTH
each term. This representation loses the relative ordering of the terms in each document; recall our example from Section 6.2 (page 117), where we pointed out that the documents Mary is quicker than John and John is quicker than Mary are identical in such a bag of words representation. How do we quantify the similarity between two documents in this vector space? A first attempt might consider the magnitude of the vector difference between two document vectors. This measure suffers from a drawback: two documents with very similar content can have a significant vector difference simply because one is much longer than the other. Thus the relative distributions of terms may be identical in the two documents, but the absolute term frequencies of one may be far larger. To compensate for the effect of document length, the standard way of quantifying the similarity between two documents d1 and d2 is to compute ~ (d1 ) and V ~ ( d2 ) the cosine similarity of their vector representations V sim(d1 , d2 ) =
~ ( d1 ) · V ~ ( d2 ) V , ~ (d1 )||V ~ (d2 )| |V
where the numerator represents the dot product (also known as the inner prod~ (d1 ) and V ~ (d2 ), while the denominator is the product of uct) of the vectors V their Euclidean lengths. The dot product ~x · ~y of two vectors is defined as ~ ∑iM =1 x i y i . Let V ( d ) denote the document vector for d, with q M components ~ ~ ~ 2 ( d ). V1 (d) . . . VM (d). The Euclidean length of d is defined to be ∑ M V i =1
LENGTH NORMALIZATION
The effect of the denominator of Equation (6.10) is thus to length-normalize ~ (d1 ) and V ~ (d2 ) to unit vectors ~v(d1 ) = V ~ ( d 1 ) / |V ~ (d1 )| and the vectors V
Doc1 0.88 0.10 0 0.46
Doc2 0.09 0.71 0.71 0
Doc3 0.58 0 0.70 0.41
◮ Figure 6.11 Euclidean normalized tf values for documents in Figure 6.9.
term affection jealous gossip
SaS 115 10 2
PaP 58 7 0
WH 20 11 6
◮ Figure 6.12 Term frequencies in three novels. The novels are Austen's Sense and Sensibility, Pride and Prejudice and Brontë's Wuthering Heights.
~ ( d 2 ) / |V ~ (d2 )|. We can then rewrite (6.10) as ~v(d2 ) = V (6.11)
sim(d1 , d2 ) = ~v(d1 ) · ~v(d2 ). Consider the documents in Figure 6.9. We now apply Euclidean normalization to theq tf values from the table, for each of the three documents in the ~2 table. The quantity ∑iM =1 Vi (d) has the values 30.56, 46.84 and 41.30 respectively for Doc1, Doc2 and Doc3. The resulting Euclidean normalized tf values for these documents are shown in Figure 6.11.
Thus, (6.11) can be viewed as the dot product of the normalized versions of the two document vectors. This measure is the cosine of the angle θ between the two vectors, shown in Figure 6.10. What use is the similarity measure sim(d1 , d2 )? Given a document d (potentially one of the di in the collection), consider searching for the documents in the collection most similar to d. Such a search is useful in a system where a user may identify a document and seek others like it – a feature available in the results lists of search engines as a more like this feature. We reduce the problem of finding the document(s) most similar to d to that of finding the di with the highest dot products (sim values) ~v(d) ·~v(di ). We could do this by computing the dot products between ~v(d) and each of ~v(d1 ), . . . , ~v(d N ), then picking off the highest resulting sim values.
Example 6.3: Figure 6.12 shows the number of occurrences of three terms (affection, jealous and gossip) in each of the following three novels: Jane Austen's Sense and Sensi-
bility (SaS) and Pride and Prejudice (PaP) and Emily Brontë's Wuthering Heights (WH).
SaS 0.996 0.087 0.017
PaP 0.993 0.120 0
WH 0.847 0.466 0.254
◮ Figure 6.13 Term vectors for the three novels of Figure 6.12. These are based on raw term frequency only and are normalized as if these were the only terms in the collection. (Since affection and jealous occur in all three documents, their tf-idf weight would be 0 in most formulations.)
Of course, there are many other terms occurring in each of these novels. In this example we represent each of these novels as a unit vector in three dimensions, corresponding to these three terms (only); we use raw term frequencies here, with no idf multiplier. The resulting weights are as shown in Figure 6.13. Now consider the cosine similarities between pairs of the resulting three-dimensional vectors. A simple computation shows that sim(~v(SAS), ~v(PAP)) is 0.999, whereas sim(~v(SAS), ~v(WH)) is 0.888; thus, the two books authored by Austen (SaS and PaP) are considerably closer to each other than to Brontë's Wuthering Heights. In fact, the similarity between the first two is almost perfect (when restricted to the three terms we consider). Here we have considered tf weights, but we could of course use other term weight functions.
TERM - DOCUMENT MATRIX
Viewing a collection of N documents as a collection of vectors leads to a natural view of a collection as a term-document matrix: this is an M × N matrix whose rows represent the M terms (dimensions) of the N columns, each of which corresponds to a document. As always, the terms being indexed could be stemmed before indexing; for instance, jealous and jealousy would under stemming be considered as a single dimension. This matrix view will prove to be useful in Chapter 18.
Queries as vectors There is a far more compelling reason to represent documents as vectors: we can also view a query as a vector. Consider the query q = jealous gossip. This query turns into the unit vector ~v(q) = (0, 0.707, 0.707) on the three coordinates of Figures 6.12 and 6.13. The key idea now: to assign to each document d a score equal to the dot product
~v(q) · ~v(d). In the example of Figure 6.13, Wuthering Heights is the top-scoring document for this query with a score of 0.509, with Pride and Prejudice a distant second with a score of 0.085, and Sense and Sensibility last with a score of 0.074. This simple example is somewhat misleading: the number of dimen-
sions in practice will be far larger than three: it will equal the vocabulary size M. To summarize, by viewing a query as a "bag of words", we are able to treat it as a very short document. As a consequence, we can use the cosine similarity between the query vector and a document vector as a measure of the score of the document for that query. The resulting scores can then be used to select the top-scoring documents for a query. Thus we have (6.12)
score(q, d) =
~ ( q) · V ~ (d) V . ~ (q)||V ~ (d)| |V
A document may have a high cosine score for a query even if it does not contain all query terms. Note that the preceding discussion does not hinge on any specific weighting of terms in the document vector, although for the present we may think of them as either tf or tf-idf weights. In fact, a number of weighting schemes are possible for query as well as document vectors, as illustrated in Example 6.4 and developed further in Section 6.4. Computing the cosine similarities between the query vector and each document vector in the collection, sorting the resulting scores and selecting the top K documents can be expensive — a single similarity computation can entail a dot product in tens of thousands of dimensions, demanding tens of thousands of arithmetic operations. In Section 7.1 we study how to use an inverted index for this purpose, followed by a series of heuristics for improving on this.
Example 6.4: We now consider the query best car insurance on a fictitious collection with N = 1,000,000 documents where the document frequencies of auto, best, car and insurance are respectively 5000, 50000, 10000 and 1000. query document product term tf df idf wt,q tf wf wt,d auto 0 5000 2.3 0 1 1 0.41 0 1 50000 1.3 1.3 0 0 0 0 best car 1 10000 2.0 2.0 1 1 0.41 0.82 insurance 1 1000 3.0 3.0 2 2 0.82 2.46 In this example the weight of a term in the query is simply the idf (and zero for a term not in the query, such as auto); this is reflected in the column header wt,q (the entry for auto is zero because the query does not contain the termauto). For documents, we use tf weighting with no use of idf but with Euclidean normalization. The former is shown under the column headed wf, while the latter is shown under the column headed wt,d . Invoking (6.9) now gives a net score of 0 + 0 + 0.82 + 2.46 = 3.28.
Computing vector scores In a typical setting we have a collection of documents each represented by a vector, a free text query represented by a vector, and a positive integer K. We
C OSINE S CORE (q) 1 float Scores[ N ] = 0 2 Initialize Length[ N ] 3 for each query term t 4 do calculate wt,q and fetch postings list for t 5 for each pair(d, tft,d ) in postings list 6 do Scores[d] += wft,d × wt,q 7 Read the array Length[d] 8 for each d 9 do Scores[d] = Scores[d]/Length[d] 10 return Top K components of Scores[] ◮ Figure 6.14 The basic algorithm for computing vector space scores.
TERM - AT- A - TIME ACCUMULATOR
seek the K documents of the collection with the highest vector space scores on the given query. We now initiate the study of determining the K documents with the highest vector space scores for a query. Typically, we seek these K top documents in ordered by decreasing score; for instance many search engines use K = 10 to retrieve and rank-order the first page of the ten best results. Here we give the basic algorithm for this computation; we develop a fuller treatment of efficient techniques and approximations in Chapter 7. Figure 6.14 gives the basic algorithm for computing vector space scores. The array Length holds the lengths (normalization factors) for each of the N documents, whereas the array Scores holds the scores for each of the documents. When the scores are finally computed in Step 9, all that remains in Step 10 is to pick off the K documents with the highest scores. The outermost loop beginning Step 3 repeats the updating of Scores, iterating over each query term t in turn. In Step 5 we calculate the weight in the query vector for term t. Steps 6-8 update the score of each document by adding in the contribution from term t. This process of adding in contributions one query term at a time is sometimes known as term-at-a-time scoring or accumulation, and the N elements of the array Scores are therefore known as accumulators. For this purpose, it would appear necessary to store, with each postings entry, the weight wft,d of term t in document d (we have thus far used either tf or tf-idf for this weight, but leave open the possibility of other functions to be developed in Section 6.4). In fact this is wasteful, since storing this weight may require a floating point number. Two ideas help alleviate this space problem. First, if we are using inverse document frequency, we need not precompute idft ; it suffices to store N/dft at the head of the postings for t. Second, we store the term frequency tft,d for each postings entry. Finally, Step 12 extracts the top K scores – this requires a priority queue
DOCUMENT- AT- A - TIME
data structure, often implemented using a heap. Such a heap takes no more than 2N comparisons to construct, following which each of the K top scores can be extracted from the heap at a cost of O(log N ) comparisons. Note that the general algorithm of Figure 6.14 does not prescribe a specific implementation of how we traverse the postings lists of the various query terms; we may traverse them one term at a time as in the loop beginning at Step 3, or we could in fact traverse them concurrently as in Figure 1.6. In such a concurrent postings traversal we compute the scores of one document at a time, so that it is sometimes called document-at-a-time scoring. We will say more about this in Section 7.1.5. Exercise 6.14 If we were to stem jealous and jealousy to a common stem before setting up the vector space, detail how the definitions of tf and idf should be modified. Exercise 6.15 Recall the tf-idf weights computed in Exercise 6.10. Compute the Euclidean normalized document vectors for each of the documents, where each vector has four components, one for each of the four terms. Exercise 6.16 Verify that the sum of the squares of the components of each of the document vectors in Exercise 6.15 is 1 (to within rounding error). Why is this the case? Exercise 6.17 With term weights as computed in Exercise 6.15, rank the three documents by computed score for the query car insurance, for each of the following cases of term weighting in the query: 1. The weight of a term is 1 if present in the query, 0 otherwise. 2. Euclidean normalized idf.
Variant tf-idf functions For assigning a weight for each term in each document, a number of alternatives to tf and tf-idf have been considered. We discuss some of the principal ones here; a more complete development is deferred to Chapter 11. We will summarize these alternatives in Section 6.4.3 (page 128).
Sublinear tf scaling It seems unlikely that twenty occurrences of a term in a document truly carry twenty times the significance of a single occurrence. Accordingly, there has been considerable research into variants of term frequency that go beyond counting the number of occurrences of a term. A common modification is
6.4 Variant tf-idf functions
to use instead the logarithm of the term frequency, which assigns a weight given by 1 + log tft,d if tft,d > 0 wft,d = . 0 otherwise
In this form, we may replace tf by some other function wf as in (6.13), to obtain: wf-idft,d = wft,d × idft .
Equation (6.9) can then be modified by replacing tf-idf by wf-idf as defined in (6.14).
Maximum tf normalization One well-studied technique is to normalize the tf weights of all terms occurring in a document by the maximum tf in that document. For each document d, let tfmax (d) = maxτ ∈d tfτ,d , where τ ranges over all terms in d. Then, we compute a normalized term frequency for each term t in document d by
ntft,d = a + (1 − a)
tft,d , tfmax (d)
where a is a value between 0 and 1 and is generally set to 0.4, although some early work used the value 0.5. The term a in (6.15) is a smoothing term whose role is to damp the contribution of the second term – which may be viewed as a scaling down of tf by the largest tf value in d. We will encounter smoothing further in Chapter 13 when discussing classification; the basic idea is to avoid a large swing in ntft,d from modest changes in tft,d (say from 1 to 2). The main idea of maximum tf normalization is to mitigate the following anomaly: we observe higher term frequencies in longer documents, merely because longer documents tend to repeat the same words over and over again. To appreciate this, consider the following extreme example: supposed we were to take a document d and create a new document d′ by simply appending a copy of d to itself. While d′ should be no more relevant to any query than d is, the use of (6.9) would assign it twice as high a score as d. Replacing tf-idft,d in (6.9) by ntf-idft,d eliminates the anomaly in this example. Maximum tf normalization does suffer from the following issues: 1. The method is unstable in the following sense: a change in the stop word list can dramatically alter term weightings (and therefore ranking). Thus, it is hard to tune. 2. A document may contain an outlier term with an unusually large number of occurrences of that term, not representative of the content of that document.
Term frequency n (natural) tft,d
Document frequency n (no) 1
l (logarithm)
t (idf)
p (prob idf)
max{0, log
a (augmented) b (boolean) L (log ave)
1 + log(tft,d ) 0.5×tft,d 0.5 + maxt (tft,d ) 1 if tft,d > 0 0 otherwise
dft N −dft dft }
n (none)
Normalization 1 1 w21 +w22 +...+w2M
c (cosine)
u (pivoted unique)
1/u (Section 6.4.4)
b (byte size)
1/CharLengthα , α < 1
1+log (tft,d ) 1+log(avet ∈ d (tft,d ))
◮ Figure 6.15 SMART notation for tf-idf variants. Here CharLength is the number of characters in the document.
3. More generally, a document in which the most frequent term appears roughly as often as many other terms should be treated differently from one with a more skewed distribution.
Document and query weighting schemes Equation (6.12) is fundamental to information retrieval systems that use any form of vector space scoring. Variations from one vector space scoring method ~ (d) and to another hinge on the specific choices of weights in the vectors V ~ (q). Figure 6.15 lists some of the principal weighting schemes in use for V ~ (d) and V ~ (q), together with a mnemonic for representing a speeach of V cific combination of weights; this system of mnemonics is sometimes called SMART notation, following the authors of an early text retrieval system. The mnemonic for representing a combination of weights takes the form ddd.qqq where the first triplet gives the term weighting of the document vector, while the second triplet gives the weighting in the query vector. The first letter in each triplet specifies the term frequency component of the weighting, the second the document frequency component, and the third the form of normalization used. It is quite common to apply different normalization func~ (d) and V ~ (q). For example, a very standard weighting scheme tions to V is lnc.ltc, where the document vector has log-weighted term frequency, no idf (for both effectiveness and efficiency reasons), and cosine normalization, while the query vector uses log-weighted term frequency, idf weighting, and cosine normalization.
PIVOTED DOCUMENT LENGTH NORMALIZATION
Pivoted normalized document length In Section 6.3.1 we normalized each document vector by the Euclidean length of the vector, so that all document vectors turned into unit vectors. In doing so, we eliminated all information on the length of the original document; this masks some subtleties about longer documents. First, longer documents will – as a result of containing more terms – have higher tf values. Second, longer documents contain more distinct terms. These factors can conspire to raise the scores of longer documents, which (at least for some information needs) is unnatural. Longer documents can broadly be lumped into two categories: (1) verbose documents that essentially repeat the same content – in these, the length of the document does not alter the relative weights of different terms; (2) documents covering multiple different topics, in which the search terms probably match small segments of the document but not all of it – in this case, the relative weights of terms are quite different from a single short document that matches the query terms. Compensating for this phenomenon is a form of document length normalization that is independent of term and document frequencies. To this end, we introduce a form of normalizing the vector representations of documents in the collection, so that the resulting "normalized" documents are not necessarily of unit length. Then, when we compute the dot product score between a (unit) query vector and such a normalized document, the score is skewed to account for the effect of document length on relevance. This form of compensation for document length is known as pivoted document length normalization. Consider a document collection together with an ensemble of queries for that collection. Suppose that we were given, for each query q and for each document d, a Boolean judgment of whether or not d is relevant to the query q; in Chapter 8 we will see how to procure such a set of relevance judgments for a query ensemble and a document collection. Given this set of relevance judgments, we may compute a probability of relevance as a function of document length, averaged over all queries in the ensemble. The resulting plot may look like the curve drawn in thick lines in Figure 6.16. To compute this curve, we bucket documents by length and compute the fraction of relevant documents in each bucket, then plot this fraction against the median document length of each bucket. (Thus even though the "curve" in Figure 6.16 appears to be continuous, it is in fact a histogram of discrete buckets of document length.) On the other hand, the curve in thin lines shows what might happen with the same documents and query ensemble if we were to use relevance as prescribed by cosine normalization Equation (6.12) – thus, cosine normalization has a tendency to distort the computed relevance vis-à-vis the true relevance, at the expense of longer documents. The thin and thick curves crossover at a point p corresponding to document length ℓ p , which we refer to as the pivot
p Document - length ℓp
◮ Figure 6.16 Pivoted document length normalization.
length; dashed lines mark this point on the x − and y− axes. The idea of pivoted document length normalization would then be to "rotate" the cosine normalization curve counter-clockwise about p so that it more closely matches thick line representing the relevance vs. document length curve. As mentioned at the beginning of this section, we do so by using in Equa~ (d) that is not tion (6.12) a normalization factor for each document vector V the Euclidean length of that vector, but instead one that is larger than the Euclidean length for documents of length less than ℓ p , and smaller for longer documents. ~ (d) in the deTo this end, we first note that the normalizing term for V ~ (d)|. In the nominator of Equation (6.12) is its Euclidean length, denoted |V simplest implementation of pivoted document length normalization, we use ~ (d)|, but one a normalization factor in the denominator that is linear in |V ~ (d)|, of slope < 1 as in Figure 6.17. In this figure, the x − axis represents |V while the y−axis represents possible normalization factors we can use. The thin line y = x depicts the use of cosine normalization. Notice the following aspects of the thick line representing pivoted length normalization: 1. It is linear in the document length and has the form (6.16)
~ (d)| + (1 − a)piv, a |V
Pivoted normalization 6
y = x; Cosine
Pivoted -
piv
~ (d)| |V
◮ Figure 6.17 Implementing pivoted document length normalization by linear scaling.
where piv is the cosine normalization value at which the two curves intersect. 2. Its slope is a < 1 and (3) it crosses the y = x line at piv. It has been argued that in practice, Equation (6.16) is well approximated by aud + (1 − a)piv, where ud is the number of unique terms in document d. Of course, pivoted document length normalization is not appropriate for all applications. For instance, in a collection of answers to frequently asked questions (say, at a customer service website), relevance may have little to do with document length. In other cases the dependency may be more complex than can be accounted for by a simple linear pivoted normalization. In such cases, document length can be used as a feature in the machine learning based scoring approach of Section 6.1.2.
E UCLIDEAN DISTANCE
Exercise 6.18 One measure of the similarity of two vectors is the Euclidean distance (or L2 distance) between them: v uM u |~x − ~y | = t ∑ ( xi − yi )2 i =1
word digital video cameras
query df idf 10,000 100,000 50,000
qi = wf-idf
document d i = normalized wf
qi · d i
◮ Table 6.1 Cosine computation for Exercise 6.19.
Given a query q and documents d1 , d2 , . . ., we may rank the documents di in order of increasing Euclidean distance from q. Show that if q and the di are all normalized to unit vectors, then the rank ordering produced by Euclidean distance is identical to that produced by cosine similarities. Exercise 6.19 Compute the vector space similarity between the query "digital cameras" and the document "digital cameras and video cameras" by filling out the empty columns in Table 6.1. Assume N = 10,000,000, logarithmic term weighting (wf columns) for query and document, idf weighting for the query only and cosine normalization for the document only. Treat and as a stop word. Enter term counts in the tf columns. What is the final similarity score? Exercise 6.20 Show that for the query affection, the relative ordering of the scores of the three documents in Figure 6.13 is the reverse of the ordering of the scores for the query jealous gossip. Exercise 6.21 In turning a query into a unit vector in Figure 6.13, we assigned equal weights to each of the query terms. What other principled approaches are plausible? Exercise 6.22 Consider the case of a query term that is not in the set of M indexed terms; thus our ~ (q ) not being in the vector space standard construction of the query vector results in V created from the collection. How would one adapt the vector space representation to handle this case? Exercise 6.23 Refer to the tf and idf values for four terms and three documents in Exercise 6.10. Compute the two top scoring documents on the query best car insurance for each of the following weighing schemes: (i) nnn.atc; (ii) ntc.atc. Exercise 6.24 Suppose that the word coyote does not occur in the collection used in Exercises 6.10 and 6.23. How would one compute ntc.atc scores for the query coyote insurance?
References and further reading Chapter 7 develops the computational aspects of vector space scoring. Luhn (1957; 1958) describes some of the earliest reported applications of term weighting. His paper dwells on the importance of medium frequency terms (terms that are neither too commonplace nor too rare) and may be thought of as anticipating tf-idf and related weighting schemes. Spärck Jones (1972) builds on this intuition through detailed experiments showing the use of inverse document frequency in term weighting. A series of extensions and theoretical justifications of idf are due to Salton and Buckley (1987) Robertson and Jones (1976), Croft and Harper (1979) and Papineni (2001). Robertson maintains a web page (http://www.soi.city.ac.uk/˜ser/idf.html) containing the history of idf, including soft copies of early papers that predated electronic versions of journal article. Singhal et al. (1996a) develop pivoted document length normalization. Probabilistic language models (Chapter 11) develop weighting techniques that are more nuanced than tf-idf; the reader will find this development in Section 11.4.3. We observed that by assigning a weight for each term in a document, a document may be viewed as a vector of term weights, one for each term in the collection. The SMART information retrieval system at Cornell (Salton 1971b) due to Salton and colleagues was perhaps the first to view a document as a vector of weights. The basic computation of cosine scores as described in Section 6.3.3 is due to Zobel and Moffat (2006). The two query evaluation strategies term-at-a-time and document-at-a-time are discussed by Turtle and Flood (1995). The SMART notation for tf-idf term weighting schemes in Figure 6.15 is presented in (Salton and Buckley 1988, Singhal et al. 1995; 1996b). Not all versions of the notation are consistent; we most closely follow (Singhal et al. 1996b). A more detailed and exhaustive notation was developed in Moffat and Zobel (1998), considering a larger palette of schemes for term and document frequency weighting. Beyond the notation, Moffat and Zobel (1998) sought to set up a space of feasible weighting functions through which hillclimbing approaches could be used to begin with weighting schemes that performed well, then make local improvements to identify the best combinations. However, they report that such hill-climbing methods failed to lead to any conclusions on the best weighting schemes.
Computing scores in a complete search system
Chapter 6 developed the theory underlying term weighting in documents for the purposes of scoring, leading up to vector space models and the basic cosine scoring algorithm of Section 6.3.3 (page 124). In this chapter we begin in Section 7.1 with heuristics for speeding up this computation; many of these heuristics achieve their speed at the risk of not finding quite the top K documents matching the query. Some of these heuristics generalize beyond cosine scoring. With Section 7.1 in place, we have essentially all the components needed for a complete search engine. We therefore take a step back from cosine scoring, to the more general problem of computing scores in a search engine. In Section 7.2 we outline a complete search engine, including indexes and structures to support not only cosine scoring but also more general ranking factors such as query term proximity. We describe how all of the various pieces fit together in Section 7.2.4. We conclude this chapter with Section 7.3, where we discuss how the vector space model for free text queries interacts with common query operators.
Efficient scoring and ranking We begin by recapping the algorithm of Figure 6.14. For a query such as q = jealous gossip, two observations are immediate:
1. The unit vector ~v(q) has only two non-zero components. 2. In the absence of any weighting for query terms, these non-zero components are equal – in this case, both equal 0.707. For the purpose of ranking the documents matching this query, we are really interested in the relative (rather than absolute) scores of the documents in the collection. To this end, it suffices to compute the cosine similarity from ~ (q) (in which all non-zero components each document unit vector ~v(d) to V of the query vector are set to 1), rather than to the unit vector ~v(q). For any
7 Computing scores in a complete search system
FAST C OSINE S CORE (q) 1 float Scores[ N ] = 0 2 for each d 3 do Initialize Length[d] to the length of doc d 4 for each query term t 5 do calculate wt,q and fetch postings list for t 6 for each pair(d, tft,d ) in postings list 7 do add wft,d to Scores[d] 8 Read the array Length[d] 9 for each d 10 do Divide Scores[d] by Length[d] 11 return Top K components of Scores[] ◮ Figure 7.1 A faster algorithm for vector space scores.
two documents d1 , d2 (7.1)
~ (q) · ~v(d1 ) > V ~ (q) · ~v(d2 ) ⇔ ~v(q) · ~v(d1 ) > ~v(q) · ~v(d2 ). V ~ (q) · ~v(d) is the weighted sum, For any document d, the cosine similarity V over all terms in the query q, of the weights of those terms in d. This in turn can be computed by a postings intersection exactly as in the algorithm of Figure 6.14, with line 8 altered since we take wt,q to be 1 so that the multiplyadd in that step becomes just an addition; the result is shown in Figure 7.1. We walk through the postings in the inverted index for the terms in q, accumulating the total score for each document – very much as in processing a Boolean query, except we assign a positive score to each document that appears in any of the postings being traversed. As mentioned in Section 6.3.3 we maintain an idf value for each dictionary term and a tf value for each postings entry. This scheme computes a score for every document in the postings of any of the query terms; the total number of such documents may be considerably smaller than N. Given these scores, the final step before presenting results to a user is to pick out the K highest-scoring documents. While one could sort the complete set of scores, a better approach is to use a heap to retrieve only the top K documents in order. Where J is the number of documents with non-zero cosine scores, constructing such a heap can be performed in 2J comparison steps, following which each of the K highest scoring documents can be "read off" the heap with log J comparison steps.
7.1 Efficient scoring and ranking
Inexact top K document retrieval Thus far, we have focused on retrieving precisely the K highest-scoring documents for a query. We now consider schemes by which we produce K documents that are likely to be among the K highest scoring documents for a query. In doing so, we hope to dramatically lower the cost of computing the K documents we output, without materially altering the user's perceived relevance of the top K results. Consequently, in most applications it suffices to retrieve K documents whose scores are very close to those of the K best. In the sections that follow we detail schemes that retrieve K such documents while potentially avoiding computing scores for most of the N documents in the collection. Such inexact top-K retrieval is not necessarily, from the user's perspective, a bad thing. The top K documents by the cosine measure are in any case not necessarily the K best for the query: cosine similarity is only a proxy for the user's perceived relevance. In Sections 7.1.2–7.1.6 below, we give heuristics using which we are likely to retrieve K documents with cosine scores close to those of the top K documents. The principal cost in computing the output stems from computing cosine similarities between the query and a large number of documents. Having a large number of documents in contention also increases the selection cost in the final stage of culling the top K documents from a heap. We now consider a series of ideas designed to eliminate a large number of documents without computing their cosine scores. The heuristics have the following two-step scheme: 1. Find a set A of documents that are contenders, where K < | A| ≪ N. A does not necessarily contain the K top-scoring documents for the query, but is likely to have many documents with scores near those of the top K. 2. Return the K top-scoring documents in A. From the descriptions of these ideas it will be clear that many of them require parameters to be tuned to the collection and application at hand; pointers to experience in setting these parameters may be found at the end of this chapter. It should also be noted that most of these heuristics are well-suited to free text queries, but not for Boolean or phrase queries.
Index elimination For a multi-term query q, it is clear we only consider documents containing at least one of the query terms. We can take this a step further using additional heuristics: 1. We only consider documents containing terms whose idf exceeds a preset threshold. Thus, in the postings traversal, we only traverse the postings
for terms with high idf. This has a fairly significant benefit: the postings lists of low-idf terms are generally long; with these removed from contention, the set of documents for which we compute cosines is greatly reduced. One way of viewing this heuristic: low-idf terms are treated as stop words and do not contribute to scoring. For instance, on the query catcher in the rye, we only traverse the postings for catcher and rye. The cutoff threshold can of course be adapted in a query-dependent manner. 2. We only consider documents that contain many (and as a special case, all) of the query terms. This can be accomplished during the postings traversal; we only compute scores for documents containing all (or many) of the query terms. A danger of this scheme is that by requiring all (or even many) query terms to be present in a document before considering it for cosine computation, we may end up with fewer than K candidate documents in the output. This issue will discussed further in Section 7.2.1.
Champion lists The idea of champion lists (sometimes also called fancy lists or top docs) is to precompute, for each term t in the dictionary, the set of the r documents with the highest weights for t; the value of r is chosen in advance. For tfidf weighting, these would be the r documents with the highest tf values for term t. We call this set of r documents the champion list for term t. Now, given a query q we create a set A as follows: we take the union of the champion lists for each of the terms comprising q. We now restrict cosine computation to only the documents in A. A critical parameter in this scheme is the value r, which is highly application dependent. Intuitively, r should be large compared with K, especially if we use any form of the index elimination described in Section 7.1.2. One issue here is that the value r is set at the time of index construction, whereas K is application dependent and may not be available until the query is received; as a result we may (as in the case of index elimination) find ourselves with a set A that has fewer than K documents. There is no reason to have the same value of r for all terms in the dictionary; it could for instance be set to be higher for rarer terms.
7.1.4 STATIC QUALITY SCORES
Static quality scores and ordering We now further develop the idea of champion lists, in the somewhat more general setting of static quality scores. In many search engines, we have available a measure of quality g(d) for each document d that is query-independent and thus static. This quality measure may be viewed as a number between zero and one. For instance, in the context of news stories on the web, g(d) may be derived from the number of favorable reviews of the story by web
◮ Figure 7.2 A static quality-ordered index. In this example we assume that Doc1, Doc2 and Doc3 respectively have static quality scores g(1) = 0.25, g(2) = 0.5, g(3) = 1.
surfers. Section 4.6 (page 80) provides further discussion on this topic, as does Chapter 21 in the context of web search. The net score for a document d is some combination of g(d) together with the query-dependent score induced (say) by (6.12). The precise combination may be determined by the learning methods of Section 6.1.2, to be developed further in Section 15.4.1; but for the purposes of our exposition here, let us consider a simple sum: (7.2)
net-score(q, d) = g(d) +
In this simple form, the static quality g(d) and the query-dependent score from (6.10) have equal contributions, assuming each is between 0 and 1. Other relative weightings are possible; the effectiveness of our heuristics will depend on the specific relative weighting. First, consider ordering the documents in the postings list for each term by decreasing value of g(d). This allows us to perform the postings intersection algorithm of Figure 1.6. In order to perform the intersection by a single pass through the postings of each query term, the algorithm of Figure 1.6 relied on the postings being ordered by document IDs. But in fact, we only required that all postings be ordered by a single common ordering; here we rely on the g(d) values to provide this common ordering. This is illustrated in Figure 7.2, where the postings are ordered in decreasing order of g(d). The first idea is a direct extension of champion lists: for a well-chosen value r, we maintain for each term t a global champion list of the r documents
with the highest values for g(d) + tf-idft,d . The list itself is, like all the postings lists considered so far, sorted by a common order (either by document IDs or by static quality). Then at query time, we only compute the net scores (7.2) for documents in the union of these global champion lists. Intuitively, this has the effect of focusing on documents likely to have large net scores. We conclude the discussion of global champion lists with one further idea. We maintain for each term t two postings lists consisting of disjoint sets of documents, each sorted by g(d) values. The first list, which we call high, contains the m documents with the highest tf values for t. The second list, which we call low, contains all other documents containing t. When processing a query, we first scan only the high lists of the query terms, computing net scores for any document on the high lists of all (or more than a certain number of) query terms. If we obtain scores for K documents in the process, we terminate. If not, we continue the scanning into the low lists, scoring documents in these postings lists. This idea is developed further in Section 7.2.1.
Impact ordering In all the postings lists described thus far, we order the documents consistently by some common ordering: typically by document ID but in Section 7.1.4 by static quality scores. As noted at the end of Section 6.3.3, such a common ordering supports the concurrent traversal of all of the query terms' postings lists, computing the score for each document as we encounter it. Computing scores in this manner is sometimes referred to as document-at-atime scoring. We will now introduce a technique for inexact top-K retrieval in which the postings are not all ordered by a common ordering, thereby precluding such a concurrent traversal. We will therefore require scores to be "accumulated" one term at a time as in the scheme of Figure 6.14, so that we have term-at-a-time scoring. The idea is to order the documents d in the postings list of term t by decreasing order of tft,d . Thus, the ordering of documents will vary from one postings list to another, and we cannot compute scores by a concurrent traversal of the postings lists of all query terms. Given postings lists ordered by decreasing order of tft,d , two ideas have been found to significantly lower the number of documents for which we accumulate scores: (1) when traversing the postings list for a query term t, we stop after considering a prefix of the postings list – either after a fixed number of documents r have been seen, or after the value of tft,d has dropped below a threshold; (2) when accumulating scores in the outer loop of Figure 6.14, we consider the query terms in decreasing order of idf, so that the query terms likely to contribute the most to the final scores are considered first. This latter idea too can be adaptive at the time of processing a query: as we get to query terms with lower idf, we can determine whether to proceed based on the changes in
document scores from processing the previous query term. If these changes are minimal, we may omit accumulation from the remaining query terms, or alternatively process shorter prefixes of their postings lists. These ideas form a common generalization of the methods introduced in Sections 7.1.2–7.1.4. We may also implement a version of static ordering in which each postings list is ordered by an additive combination of static and query-dependent scores. We would again lose the consistency of ordering across postings, thereby having to process query terms one at time accumulating scores for all documents as we go along. Depending on the particular scoring function, the postings list for a document may be ordered by other quantities than term frequency; under this more general setting, this idea is known as impact ordering.
Cluster pruning In cluster pruning we have a preprocessing step during which we cluster the document vectors. Then at query time, we consider only documents in a small number of clusters as candidates for which we compute cosine scores. Specifically, the preprocessing step is as follows: √ 1. Pick N documents at random from the collection. Call these leaders. 2. For each document that is not a leader, we compute its nearest leader. We refer to documents that are not leaders as followers. Intuitively, in the par√ tition of the followers induced by the use of N randomly √ chosen √ leaders, the expected number of followers for each leader is ≈ N/ N = N. Next, query processing proceeds as follows: 1. Given a query q, find the leader L that is closest √ to q. This entails computing cosine similarities from q to each of the N leaders. 2. The candidate set A consists of L together with its followers. We compute the cosine scores for all documents in this candidate set. The use of randomly chosen leaders for clustering is fast and likely to reflect the distribution of the document vectors in the vector space: a region of the vector space that is dense in documents is likely to produce multiple leaders and thus a finer partition into sub-regions. This illustrated in Figure 7.3. Variations of cluster pruning introduce additional parameters b1 and b2 , both of which are positive integers. In the pre-processing step we attach each follower to its b1 closest leaders, rather than a single closest leader. At query time we consider the b2 leaders closest to the query q. Clearly, the basic scheme above corresponds to the case b1 = b2 = 1. Further, increasing b1 or
◮ Figure 7.3 Cluster pruning.
b2 increases the likelihood of finding K documents that are more likely to be in the set of true top-scoring K documents, at the expense of more computation. We reiterate this approach when describing clustering in Chapter 16 (page 354).
Exercise 7.1 We suggested above (Figure 7.2) that the postings for static quality ordering be in decreasing order of g(d). Why do we use the decreasing rather than the increasing order? Exercise 7.2 When discussing champion lists, we simply used the r documents with the largest tf values to create the champion list for t. But when considering global champion lists, we used idf as well, identifying documents with the largest values of g(d) + tf-idft,d . Why do we differentiate between these two cases? Exercise 7.3 If we were to only have one-term queries, explain why the use of global champion lists with r = K suffices for identifying the K highest scoring documents. What is a simple modification to this idea if we were to only have s-term queries for any fixed integer s > 1? Exercise 7.4 Explain how the common global ordering by g(d) values in all high and low lists helps make the score computation efficient.
7.2 Components of an information retrieval system
Exercise 7.5 Consider again the data of Exercise 6.23 with nnn.atc for the query-dependent scoring. Suppose that we were given static quality scores of 1 for Doc1 and 2 for Doc2. Determine under Equation (7.2) what ranges of static quality score for Doc3 result in it being the first, second or third result for the query best car insurance. Exercise 7.6 Sketch the frequency-ordered postings for the data in Figure 6.9. Exercise 7.7 Let the static quality scores for Doc1, Doc2 and Doc3 in Figure 6.11 be respectively 0.25, 0.5 and 1. Sketch the postings for impact ordering when each postings list is ordered by the sum of the static quality score and the Euclidean normalized tf values in Figure 6.11. Exercise 7.8 The nearest-neighbor problem in the plane is the following: given a set of N data points on the plane, we preprocess them into some data structure such that, given a query point Q, we seek the point in N that is closest to Q in Euclidean distance. Clearly cluster pruning can be used as an approach to the nearest-neighbor problem in the plane, if we wished to avoid computing the distance from Q to every one of the query points. Devise a simple example on the plane so that with two leaders, the answer returned by cluster pruning is incorrect (it is not the data point closest to Q).
Components of an information retrieval system In this section we combine the ideas developed so far to describe a rudimentary search system that retrieves and scores documents. We first develop further ideas for scoring, beyond vector spaces. Following this, we will put together all of these elements to outline a complete system. Because we consider a complete system, we do not restrict ourselves to vector space retrieval in this section. Indeed, our complete system will have provisions for vector space as well as other query operators and forms of retrieval. In Section 7.3 we will return to how vector space queries interact with other query operators.
TIERED INDEXES
Tiered indexes We mentioned in Section 7.1.2 that when using heuristics such as index elimination for inexact top-K retrieval, we may occasionally find ourselves with a set A of contenders that has fewer than K documents. A common solution to this issue is the user of tiered indexes, which may be viewed as a generalization of champion lists. We illustrate this idea in Figure 7.4, where we represent the documents and terms of Figure 6.9. In this example we set a tf threshold of 20 for tier 1 and 10 for tier 2, meaning that the tier 1 index only has postings entries with tf values exceeding 20, while the tier 2 index only
◮ Figure 7.4 Tiered indexes. If we fail to get K results from tier 1, query processing "falls back" to tier 2, and so on. Within each tier, postings are ordered by document ID.
has postings entries with tf values exceeding 10. In this example we have chosen to order the postings entries within a tier by document ID.
Query-term proximity Especially for free text queries on the web (Chapter 19), users prefer a document in which most or all of the query terms appear close to each other,
PROXIMITY WEIGHTING
because this is evidence that the document has text focused on their query intent. Consider a query with two or more query terms, t1 , t2 , . . . , tk . Let ω be the width of the smallest window in a document d that contains all the query terms, measured in the number of words in the window. For instance, if the document were to simply consist of the sentence The quality of mercy is not strained, the smallest window for the query strained mercy would be 4. Intuitively, the smaller that ω is, the better that d matches the query. In cases where the document does not contain all of the query terms, we can set ω to be some enormous number. We could also consider variants in which only words that are not stop words are considered in computing ω. Such proximity-weighted scoring functions are a departure from pure cosine similarity and closer to the "soft conjunctive" semantics that Google and other web search engines evidently use. How can we design such a proximity-weighted scoring function to depend on ω? The simplest answer relies on a "hand coding" technique we introduce below in Section 7.2.3. A more scalable approach goes back to Section 6.1.2 – we treat the integer ω as yet another feature in the scoring function, whose importance is assigned by machine learning, as will be developed further in Section 15.4.1.
Designing parsing and scoring functions Common search interfaces, particularly for consumer-facing search applications on the web, tend to mask query operators from the end user. The intent is to hide the complexity of these operators from the largely non-technical audience for such applications, inviting free text queries. Given such interfaces, how should a search equipped with indexes for various retrieval operators treat a query such as rising interest rates? More generally, given the various factors we have studied that could affect the score of a document, how should we combine these features? The answer of course depends on the user population, the query distribution and the collection of documents. Typically, a query parser is used to translate the user-specified keywords into a query with various operators that is executed against the underlying indexes. Sometimes, this execution can entail multiple queries against the underlying indexes; for example, the query parser may issue a stream of queries: 1. Run the user-generated query string as a phrase query. Rank them by vector space scoring using as query the vector consisting of the 3 terms rising interest rates. 2. If fewer than ten documents contain the phrase rising interest rates, run the two 2-term phrase queries rising interest and interest rates; rank these using vector space scoring, as well.
3. If we still have fewer than ten results, run the vector space query consisting of the three individual query terms.
EVIDENCE ACCUMULATION
Each of these steps (if invoked) may yield a list of scored documents, for each of which we compute a score. This score must combine contributions from vector space scoring, static quality, proximity weighting and potentially other factors – particularly since a document may appear in the lists from multiple steps. This demands an aggregate scoring function that accumulates evidence of a document's relevance from multiple sources. How do we devise a query parser and how do we devise the aggregate scoring function? The answer depends on the setting. In many enterprise settings we have application builders who make use of a toolkit of available scoring operators, along with a query parsing layer, with which to manually configure the scoring function as well as the query parser. Such application builders make use of the available zones, metadata and knowledge of typical documents and queries to tune the parsing and scoring. In collections whose characteristics change infrequently (in an enterprise application, significant changes in collection and query characteristics typically happen with infrequent events such as the introduction of new document formats or document management systems, or a merger with another company). Web search on the other hand is faced with a constantly changing document collection with new characteristics being introduced all the time. It is also a setting in which the number of scoring factors can run into the hundreds, making hand-tuned scoring a difficult exercise. To address this, it is becoming increasingly common to use machine-learned scoring, extending the ideas we introduced in Section 6.1.2, as will be discussed further in Section 15.4.1.
Putting it all together We have now studied all the components necessary for a basic search system that supports free text queries as well as Boolean, zone and field queries. We briefly review how the various pieces fit together into an overall system; this is depicted in Figure 7.5. In this figure, documents stream in from the left for parsing and linguistic processing (language and format detection, tokenization and stemming). The resulting stream of tokens feeds into two modules. First, we retain a copy of each parsed document in a document cache. This will enable us to generate results snippets: snippets of text accompanying each document in the results list for a query. This snippet tries to give a succinct explanation to the user of why the document matches the query. The automatic generation of such snippets is the subject of Section 8.7. A second copy of the tokens is fed to a bank of indexers that create a bank of indexes including zone and field indexes that store the metadata for each document,
7.3 Vector space scoring and query operator interaction
◮ Figure 7.5 A complete search system. Data paths are shown primarily for a free text query.
(tiered) positional indexes, indexes for spelling correction and other tolerant retrieval, and structures for accelerating inexact top-K retrieval. A free text user query (top center) is sent down to the indexes both directly and through a module for generating spelling-correction candidates. As noted in Chapter 3 the latter may optionally be invoked only when the original query fails to retrieve enough results. Retrieved documents (dark arrow) are passed to a scoring module that computes scores based on machine-learned ranking (MLR), a technique that builds on Section 6.1.2 (to be further developed in Section 15.4.1) for scoring and ranking documents. Finally, these ranked documents are rendered as a results page.
Exercise 7.9 Explain how the postings intersection algorithm first introduced in Section 1.3 can be adapted to find the smallest integer ω that contains all query terms. Exercise 7.10 Adapt this procedure to work when not all query terms are present in a document.
Vector space scoring and query operator interaction We introduced the vector space model as a paradigm for free text queries. We conclude this chapter by discussing how the vector space scoring model
relates to the query operators we have studied in earlier chapters. The relationship should be viewed at two levels: in terms of the expressiveness of queries that a sophisticated user may pose, and in terms of the index that supports the evaluation of the various retrieval methods. In building a search engine, we may opt to support multiple query operators for an end user. In doing so we need to understand what components of the index can be shared for executing various query operators, as well as how to handle user queries that mix various query operators. Vector space scoring supports so-called free text retrieval, in which a query is specified as a set of words without any query operators connecting them. It allows documents matching the query to be scored and thus ranked, unlike the Boolean, wildcard and phrase queries studied earlier. Classically, the interpretation of such free text queries was that at least one of the query terms be present in any retrieved document. However more recently, web search engines such as Google have popularized the notion that a set of terms typed into their query boxes (thus on the face of it, a free text query) carries the semantics of a conjunctive query that only retrieves documents containing all or most query terms. Boolean retrieval Clearly a vector space index can be used to answer Boolean queries, as long as the weight of a term t in the document vector for d is non-zero whenever t occurs in d. The reverse is not true, since a Boolean index does not by default maintain term weight information. There is no easy way of combining vector space and Boolean queries from a user's standpoint: vector space queries are fundamentally a form of evidence accumulation, where the presence of more query terms in a document adds to the score of a document. Boolean retrieval on the other hand, requires a user to specify a formula for selecting documents through the presence (or absence) of specific combinations of keywords, without inducing any relative ordering among them. Mathematically, it is in fact possible to invoke so-called p-norms to combine Boolean and vector space queries, but we know of no system that makes use of this fact. Wildcard queries Wildcard and vector space queries require different indexes, except at the basic level that both can be implemented using postings and a dictionary (e.g., a dictionary of trigrams for wildcard queries). If a search engine allows a user to specify a wildcard operator as part of a free text query (for instance, the query rom* restaurant), we may interpret the wildcard component of the query as spawning multiple terms in the vector space (in this example, rome
and roman would be two such terms) all of which are added to the query vector. The vector space query is then executed as usual, with matching documents being scored and ranked; thus a document containing both rome and roma is likely to be scored higher than another containing only one of them. The exact score ordering will of course depend on the relative weights of each term in matching documents. Phrase queries The representation of documents as vectors is fundamentally lossy: the relative order of terms in a document is lost in the encoding of a document as a vector. Even if we were to try and somehow treat every biword as a term (and thus an axis in the vector space), the weights on different axes not independent: for instance the phrase German shepherd gets encoded in the axis german shepherd, but immediately has a non-zero weight on the axes german and shepherd. Further, notions such as idf would have to be extended to such biwords. Thus an index built for vector space retrieval cannot, in general, be used for phrase queries. Moreover, there is no way of demanding a vector space score for a phrase query — we only know the relative weights of each term in a document. On the query german shepherd, we could use vector space retrieval to identify documents heavy in these two terms, with no way of prescribing that they occur consecutively. Phrase retrieval, on the other hand, tells us of the existence of the phrase german shepherd in a document, without any indication of the relative frequency or weight of this phrase. While these two retrieval paradigms (phrase and vector space) consequently have different implementations in terms of indexes and retrieval algorithms, they can in some cases be combined usefully, as in the three-step example of query parsing in Section 7.2.3.
References and further reading Heuristics for fast query processing with early termination are described by Anh et al. (2001), Garcia et al. (2004), Anh and Moffat (2006b), Persin et al. (1996). Cluster pruning is investigated by Singitham et al. (2004) and by Chierichetti et al. (2007); see also Section 16.6 (page 372). Champion lists are described in Persin (1994) and (under the name top docs) in Brown (1995), and further developed in Brin and Page (1998), Long and Suel (2003). While these heuristics are well-suited to free text queries that can be viewed as vectors, they complicate phrase queries; see Anh and Moffat (2006c) for an index structure that supports both weighted and Boolean/phrase searches. Carmel et al. (2001) Clarke et al. (2000) and Song et al. (2005) treat the use of query
term proximity in assessing relevance. Pioneering work on learning of ranking functions was done by Fuhr (1989), Fuhr and Pfeifer (1994), Cooper et al. (1994), Bartell (1994), Bartell et al. (1998) and by Cohen et al. (1998).
Evaluation in information retrieval
We have seen in the preceding chapters many alternatives in designing an IR system. How do we know which of these techniques are effective in which applications? Should we use stop lists? Should we stem? Should we use inverse document frequency weighting? Information retrieval has developed as a highly empirical discipline, requiring careful and thorough evaluation to demonstrate the superior performance of novel techniques on representative document collections. In this chapter we begin with a discussion of measuring the effectiveness of IR systems (Section 8.1) and the test collections that are most often used for this purpose (Section 8.2). We then present the straightforward notion of relevant and nonrelevant documents and the formal evaluation methodology that has been developed for evaluating unranked retrieval results (Section 8.3). This includes explaining the kinds of evaluation measures that are standardly used for document retrieval and related tasks like text classification and why they are appropriate. We then extend these notions and develop further measures for evaluating ranked retrieval results (Section 8.4) and discuss developing reliable and informative test collections (Section 8.5). We then step back to introduce the notion of user utility, and how it is approximated by the use of document relevance (Section 8.6). The key utility measure is user happiness. Speed of response and the size of the index are factors in user happiness. It seems reasonable to assume that relevance of results is the most important factor: blindingly fast, useless answers do not make a user happy. However, user perceptions do not always coincide with system designers' notions of quality. For example, user happiness commonly depends very strongly on user interface design issues, including the layout, clarity, and responsiveness of the user interface, which are independent of the quality of the results returned. We touch on other measures of the quality of a system, in particular the generation of high-quality result summary snippets, which strongly influence user utility, but are not measured in the basic relevance ranking paradigm (Section 8.7).
8 Evaluation in information retrieval
Information retrieval system evaluation To measure ad hoc information retrieval effectiveness in the standard way, we need a test collection consisting of three things: 1. A document collection 2. A test suite of information needs, expressible as queries 3. A set of relevance judgments, standardly a binary assessment of either relevant or nonrelevant for each query-document pair.
GOLD STANDARD GROUND TRUTH
INFORMATION NEED
The standard approach to information retrieval system evaluation revolves around the notion of relevant and nonrelevant documents. With respect to a user information need, a document in the test collection is given a binary classification as either relevant or nonrelevant. This decision is referred to as the gold standard or ground truth judgment of relevance. The test document collection and suite of information needs have to be of a reasonable size: you need to average performance over fairly large test sets, as results are highly variable over different documents and information needs. As a rule of thumb, 50 information needs has usually been found to be a sufficient minimum. Relevance is assessed relative to an information need, not a query. For example, an information need might be: Information on whether drinking red wine is more effective at reducing your risk of heart attacks than white wine. This might be translated into a query such as: wine AND red AND white AND heart AND attack AND effective
A document is relevant if it addresses the stated information need, not because it just happens to contain all the words in the query. This distinction is often misunderstood in practice, because the information need is not overt. But, nevertheless, an information need is present. If a user types python into a web search engine, they might be wanting to know where they can purchase a pet python. Or they might be wanting information on the programming language Python. From a one word query, it is very difficult for a system to know what the information need is. But, nevertheless, the user has one, and can judge the returned results on the basis of their relevance to it. To evaluate a system, we require an overt expression of an information need, which can be used for judging returned documents as relevant or nonrelevant. At this point, we make a simplification: relevance can reasonably be thought of as a scale, with some documents highly relevant and others marginally so. But for the moment, we will use just a binary decision of relevance. We
8.2 Standard test collections
DEVELOPMENT TEST COLLECTION
discuss the reasons for using binary relevance judgments and alternatives in Section 8.5.1. Many systems contain various weights (often known as parameters) that can be adjusted to tune system performance. It is wrong to report results on a test collection which were obtained by tuning these parameters to maximize performance on that collection. That is because such tuning overstates the expected performance of the system, because the weights will be set to maximize performance on one particular set of queries rather than for a random sample of queries. In such cases, the correct procedure is to have one or more development test collections, and to tune the parameters on the development test collection. The tester then runs the system with those weights on the test collection and reports the results on that collection as an unbiased estimate of performance.
Standard test collections Here is a list of the most standard test collections and evaluation series. We focus particularly on test collections for ad hoc information retrieval system evaluation, but also mention a couple of similar test collections for text classification.
C RANFIELD
The Cranfield collection. This was the pioneering test collection in allowing precise quantitative measures of information retrieval effectiveness, but is nowadays too small for anything but the most elementary pilot experiments. Collected in the United Kingdom starting in the late 1950s, it contains 1398 abstracts of aerodynamics journal articles, a set of 225 queries, and exhaustive relevance judgments of all (query, document) pairs.
Text Retrieval Conference (TREC). The U.S. National Institute of Standards and Technology (NIST) has run a large IR test bed evaluation series since 1992. Within this framework, there have been many tracks over a range of different test collections, but the best known test collections are the ones used for the TREC Ad Hoc track during the first 8 TREC evaluations between 1992 and 1999. In total, these test collections comprise 6 CDs containing 1.89 million documents (mainly, but not exclusively, newswire articles) and relevance judgments for 450 information needs, which are called topics and specified in detailed text passages. Individual test collections are defined over different subsets of this data. The early TRECs each consisted of 50 information needs, evaluated over different but overlapping sets of documents. TRECs 6–8 provide 150 information needs over about 528,000 newswire and Foreign Broadcast Information Service articles. This is probably the best subcollection to use in future work, because it is the largest and the topics are more consistent. Because the test
document collections are so large, there are no exhaustive relevance judgments. Rather, NIST assessors' relevance judgments are available only for the documents that were among the top k returned for some system which was entered in the TREC evaluation for which the information need was developed. GOV2
NTCIR CROSS - LANGUAGE INFORMATION RETRIEVAL
In more recent years, NIST has done evaluations on larger document collections, including the 25 million page GOV2 web page collection. From the beginning, the NIST test document collections were orders of magnitude larger than anything available to researchers previously and GOV2 is now the largest Web collection easily available for research purposes. Nevertheless, the size of GOV2 is still more than 2 orders of magnitude smaller than the current size of the document collections indexed by the large web search companies. NII Test Collections for IR Systems (NTCIR). The NTCIR project has built various test collections of similar sizes to the TREC collections, focusing on East Asian language and cross-language information retrieval, where queries are made in one language over a document collection containing documents in one or more other languages. See: http://research.nii.ac.jp/ntcir/data/dataen.html
CLEF
Cross Language Evaluation Forum (CLEF). This evaluation series has concentrated on European languages and cross-language information retrieval. See: http://www.clef-campaign.org/
R EUTERS
Reuters-21578 and Reuters-RCV1. For text classification, the most used test collection has been the Reuters-21578 collection of 21578 newswire articles; see Chapter 13, page 279. More recently, Reuters released the much larger Reuters Corpus Volume 1 (RCV1), consisting of 806,791 documents; see Chapter 4, page 69. Its scale and rich annotation makes it a better basis for future research.
20 N EWSGROUPS
20 Newsgroups. This is another widely used text classification collection, collected by Ken Lang. It consists of 1000 articles from each of 20 Usenet newsgroups (the newsgroup name being regarded as the category). After the removal of duplicate articles, as it is usually used, it contains 18941 articles.
Evaluation of unranked retrieval sets Given these ingredients, how is system effectiveness measured? The two most frequent and basic measures for information retrieval effectiveness are precision and recall. These are first defined for the simple case where an
8.3 Evaluation of unranked retrieval sets
IR system returns a set of documents for a query. We will see later how to extend these notions to ranked retrieval situations. PRECISION
Precision (P) is the fraction of retrieved documents that are relevant Precision =
(8.1) RECALL
#(relevant items retrieved) = P(relevant|retrieved) #(retrieved items)
Recall (R) is the fraction of relevant documents that are retrieved
Recall =
#(relevant items retrieved) = P(retrieved|relevant) #(relevant items)
These notions can be made clear by examining the following contingency table: (8.3) Retrieved Not retrieved
Relevant true positives (tp) false negatives (fn)
Nonrelevant false positives (fp) true negatives (tn)
Then: (8.4)
= tp/(tp + f p) = tp/(tp + f n)
An obvious alternative that may occur to the reader is to judge an information retrieval system by its accuracy, that is, the fraction of its classifications that are correct. In terms of the contingency table above, accuracy = (tp + tn)/(tp + f p + f n + tn). This seems plausible, since there are two actual classes, relevant and nonrelevant, and an information retrieval system can be thought of as a two-class classifier which attempts to label them as such (it retrieves the subset of documents which it believes to be relevant). This is precisely the effectiveness measure often used for evaluating machine learning classification problems. There is a good reason why accuracy is not an appropriate measure for information retrieval problems. In almost all circumstances, the data is extremely skewed: normally over 99.9% of the documents are in the nonrelevant category. A system tuned to maximize accuracy can appear to perform well by simply deeming all documents nonrelevant to all queries. Even if the system is quite good, trying to label some documents as relevant will almost always lead to a high rate of false positives. However, labeling all documents as nonrelevant is completely unsatisfying to an information retrieval system user. Users are always going to want to see some documents, and can be
assumed to have a certain tolerance for seeing some false positives providing that they get some useful information. The measures of precision and recall concentrate the evaluation on the return of true positives, asking what percentage of the relevant documents have been found and how many false positives have also been returned. The advantage of having the two numbers for precision and recall is that one is more important than the other in many circumstances. Typical web surfers would like every result on the first page to be relevant (high precision) but have not the slightest interest in knowing let alone looking at every document that is relevant. In contrast, various professional searchers such as paralegals and intelligence analysts are very concerned with trying to get as high recall as possible, and will tolerate fairly low precision results in order to get it. Individuals searching their hard disks are also often interested in high recall searches. Nevertheless, the two quantities clearly trade off against one another: you can always get a recall of 1 (but very low precision) by retrieving all documents for all queries! Recall is a non-decreasing function of the number of documents retrieved. On the other hand, in a good system, precision usually decreases as the number of documents retrieved is increased. In general we want to get some amount of recall while tolerating only a certain percentage of false positives. A single measure that trades off precision versus recall is the F measure, which is the weighted harmonic mean of precision and recall: F=
1 ( β2 + 1) PR = β2 P + R α P1 + (1 − α) R1
β2 =
1−α α
where α ∈ [0, 1] and thus β2 ∈ [0, ∞]. The default balanced F measure equally weights precision and recall, which means making α = 1/2 or β = 1. It is commonly written as F1 , which is short for Fβ=1 , even though the formulation in terms of α more transparently exhibits the F measure as a weighted harmonic mean. When using β = 1, the formula on the right simplifies to: (8.6)
Fβ=1 =
2PR P+R
However, using an even weighting is not the only choice. Values of β < 1 emphasize precision, while values of β > 1 emphasize recall. For example, a value of β = 3 or β = 5 might be used if recall is to be emphasized. Recall, precision, and the F measure are inherently measures between 0 and 1, but they are also very commonly written as percentages, on a scale between 0 and 100. Why do we use a harmonic mean rather than the simpler average (arithmetic mean)? Recall that we can always get 100% recall by just returning all documents, and therefore we can always get a 50% arithmetic mean by the
◮ Figure 8.1 Graph comparing the harmonic mean to other means. The graph shows a slice through the calculation of various means of precision and recall for the fixed recall value of 70%. The harmonic mean is always less than either the arithmetic or geometric mean, and often quite close to the minimum of the two numbers. When the precision is also 70%, all the measures coincide.
same process. This strongly suggests that the arithmetic mean is an unsuitable measure to use. In contrast, if we assume that 1 document in 10,000 is relevant to the query, the harmonic mean score of this strategy is 0.02%. The harmonic mean is always less than or equal to the arithmetic mean and the geometric mean. When the values of two numbers differ greatly, the harmonic mean is closer to their minimum than to their arithmetic mean; see Figure 8.1.
An IR system returns 8 relevant documents, and 10 nonrelevant documents. There are a total of 20 relevant documents in the collection. What is the precision of the system on this search, and what is its recall? Exercise 8.2
The balanced F measure (a.k.a. F1 ) is defined as the harmonic mean of precision and recall. What is the advantage of using the harmonic mean rather than "averaging" (using the arithmetic mean)?
Recall ◮ Figure 8.2 Precision/recall graph.
Derive the equivalence between the two formulas for F measure shown in Equation (8.5), given that α = 1/( β2 + 1).
PRECISION - RECALL CURVE
INTERPOLATED PRECISION
Evaluation of ranked retrieval results Precision, recall, and the F measure are set-based measures. They are computed using unordered sets of documents. We need to extend these measures (or to define new measures) if we are to evaluate the ranked retrieval results that are now standard with search engines. In a ranked retrieval context, appropriate sets of retrieved documents are naturally given by the top k retrieved documents. For each such set, precision and recall values can be plotted to give a precision-recall curve, such as the one shown in Figure 8.2. Precision-recall curves have a distinctive saw-tooth shape: if the (k + 1)th document retrieved is nonrelevant then recall is the same as for the top k documents, but precision has dropped. If it is relevant, then both precision and recall increase, and the curve jags up and to the right. It is often useful to remove these jiggles and the standard way to do this is with an interpolated precision: the interpolated precision pinter p at a certain recall level r is defined
8.4 Evaluation of ranked retrieval results
Recall 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Interp. Precision 1.00 0.67 0.63 0.55 0.45 0.41 0.36 0.29 0.13 0.10 0.08
◮ Table 8.1 Calculation of 11-point Interpolated Average Precision. This is for the precision-recall curve shown in Figure 8.2.
as the highest precision found for any recall level r ′ ≥ r: (8.7)
11- POINT INTERPOLATED AVERAGE PRECISION
M EAN AVERAGE P RECISION
pinter p(r ) = max p(r ′ ) r ′ ≥r
The justification is that almost anyone would be prepared to look at a few more documents if it would increase the percentage of the viewed set that were relevant (that is, if the precision of the larger set is higher). Interpolated precision is shown by a thinner line in Figure 8.2. With this definition, the interpolated precision at a recall of 0 is well-defined (Exercise 8.4). Examining the entire precision-recall curve is very informative, but there is often a desire to boil this information down to a few numbers, or perhaps even a single number. The traditional way of doing this (used for instance in the first 8 TREC Ad Hoc evaluations) is the 11-point interpolated average precision. For each information need, the interpolated precision is measured at the 11 recall levels of 0.0, 0.1, 0.2, . . . , 1.0. For the precision-recall curve in Figure 8.2, these 11 values are shown in Table 8.1. For each recall level, we then calculate the arithmetic mean of the interpolated precision at that recall level for each information need in the test collection. A composite precisionrecall curve showing 11 points can then be graphed. Figure 8.3 shows an example graph of such results from a representative good system at TREC 8. In recent years, other measures have become more common. Most standard among the TREC community is Mean Average Precision (MAP), which provides a single-figure measure of quality across recall levels. Among evaluation measures, MAP has been shown to have especially good discrimination and stability. For a single information need, Average Precision is the
0.8 0.6 0.4 0.2 0
Recall ◮ Figure 8.3 Averaged 11-point precision/recall graph across 50 queries for a representative TREC system. The Mean Average Precision for this system is 0.2553.
average of the precision value obtained for the set of top k documents existing after each relevant document is retrieved, and this value is then averaged over information needs. That is, if the set of relevant documents for an information need q j ∈ Q is {d1 , . . . d m j } and R jk is the set of ranked retrieval results from the top result until you get to document dk , then (8.8)
MAP( Q) =
1 |Q|
| Q|
∑ m j ∑ Precision( R jk )
j =1
k =1
When a relevant document is not retrieved at all,1 the precision value in the above equation is taken to be 0. For a single information need, the average precision approximates the area under the uninterpolated precision-recall curve, and so the MAP is roughly the average area under the precision-recall curve for a set of queries. Using MAP, fixed recall levels are not chosen, and there is no interpolation. The MAP value for a test collection is the arithmetic mean of average 1. A system may not fully order all documents in the collection in response to a query or at any rate an evaluation exercise may be based on submitting only the top k results for each information need.
PRECISION AT
R- PRECISION
BREAK - EVEN POINT
precision values for individual information needs. (This has the effect of weighting each information need equally in the final reported number, even if many documents are relevant to some queries whereas very few are relevant to other queries.) Calculated MAP scores normally vary widely across information needs when measured within a single system, for instance, between 0.1 and 0.7. Indeed, there is normally more agreement in MAP for an individual information need across systems than for MAP scores for different information needs for the same system. This means that a set of test information needs must be large and diverse enough to be representative of system effectiveness across different queries. The above measures factor in precision at all recall levels. For many prominent applications, particularly web search, this may not be germane to users. What matters is rather how many good results there are on the first page or the first three pages. This leads to measuring precision at fixed low levels of retrieved results, such as 10 or 30 documents. This is referred to as "Precision at k", for example "Precision at 10". It has the advantage of not requiring any estimate of the size of the set of relevant documents but the disadvantages that it is the least stable of the commonly used evaluation measures and that it does not average well, since the total number of relevant documents for a query has a strong influence on precision at k. An alternative, which alleviates this problem, is R-precision. It requires having a set of known relevant documents Rel, from which we calculate the precision of the top Rel documents returned. (The set Rel may be incomplete, such as when Rel is formed by creating relevance judgments for the pooled top k results of particular systems in a set of experiments.) R-precision adjusts for the size of the set of relevant documents: A perfect system could score 1 on this metric for each query, whereas, even a perfect system could only achieve a precision at 20 of 0.4 if there were only 8 documents in the collection relevant to an information need. Averaging this measure across queries thus makes more sense. This measure is harder to explain to naive users than Precision at k but easier to explain than MAP. If there are | Rel | relevant documents for a query, we examine the top | Rel | results of a system, and find that r are relevant, then by definition, not only is the precision (and hence R-precision) r/| Rel |, but the recall of this result set is also r/| Rel |. Thus, R-precision turns out to be identical to the break-even point, another measure which is sometimes used, defined in terms of this equality relationship holding. Like Precision at k, R-precision describes only one point on the precision-recall curve, rather than attempting to summarize effectiveness across the curve, and it is somewhat unclear why you should be interested in the break-even point rather than either the best point on the curve (the point with maximal F-measure) or a retrieval level of interest to a particular application (Precision at k). Nevertheless, R-precision turns out to be highly correlated with MAP empirically, despite measuring only a single point on
sensitivity ( = recall)
1 − specificity
◮ Figure 8.4 The ROC curve corresponding to the precision-recall curve in Figure 8.2.
CUMULATIVE GAIN
the curve. Another concept sometimes used in evaluation is an ROC curve. ("ROC" stands for "Receiver Operating Characteristics", but knowing that doesn't help most people.) An ROC curve plots the true positive rate or sensitivity against the false positive rate or (1 − specificity). Here, sensitivity is just another term for recall. The false positive rate is given by f p/( f p + tn). Figure 8.4 shows the ROC curve corresponding to the precision-recall curve in Figure 8.2. An ROC curve always goes from the bottom left to the top right of the graph. For a good system, the graph climbs steeply on the left side. For unranked result sets, specificity, given by tn/( f p + tn), was not seen as a very useful notion. Because the set of true negatives is always so large, its value would be almost 1 for all information needs (and, correspondingly, the value of the false positive rate would be almost 0). That is, the "interesting" part of Figure 8.2 is 0 < recall < 0.4, a part which is compressed to a small corner of Figure 8.4. But an ROC curve could make sense when looking over the full retrieval spectrum, and it provides another way of looking at the data. In many fields, a common aggregate measure is to report the area under the ROC curve, which is the ROC analog of MAP. Precision-recall curves are sometimes loosely referred to as ROC curves. This is understandable, but not accurate. A final approach that has seen increasing adoption, especially when employed with machine learning approaches to ranking (see Section 15.4, page 341) is measures of cumulative gain, and in particular normalized discounted cumu-
NORMALIZED DISCOUNTED CUMULATIVE GAIN
lative gain (NDCG). NDCG is designed for situations of non-binary notions of relevance (cf. Section 8.5.1). Like precision at k, it is evaluated over some number k of top search results. For a set of queries Q, let R( j, d) be the relevance score assessors gave to document d for query j. Then,
NDCG( Q, k) =
2R( j,m) − 1 , log2 (1 + m) m =1
∑ Zkj ∑
where Zkj is a normalization factor calculated to make it so that a perfect ranking's NDCG at k for query j is 1. For queries for which k′ < k documents are retrieved, the last summation is done up to k′ .
Exercise 8.4 What are the possible values for interpolated precision at a recall level of 0?
Must there always be a break-even point between precision and recall? Either show there must be or give a counter-example. [⋆⋆]
Exercise 8.6 What is the relationship between the value of F1 and the break-even point?
Exercise 8.7 D ICE COEFFICIENT
The Dice coefficient of two sets is a measure of their intersection scaled by their size (giving a value in the range 0 to 1): Dice( X, Y ) =
2| X ∩ Y | | X | + |Y |
Show that the balanced F-measure (F1 ) is equal to the Dice coefficient of the retrieved and relevant document sets. [ ⋆]
Consider an information need for which there are 4 relevant documents in the collection. Contrast two systems run on this collection. Their top 10 results are judged for relevance as follows (the leftmost item is the top ranked search result): System 1
R N R N N
N N N R R
N R N N R
R R N N N
a. What is the MAP of each system? Which has a higher MAP? b. Does this result intuitively make sense? What does it say about what is important in getting a good MAP score? c. What is the R-precision of each system? (Does it rank the systems the same as MAP?)
8 Evaluation in information retrieval [⋆⋆] Exercise 8.9 The following list of Rs and Ns represents relevant (R) and nonrelevant (N) returned documents in a ranked list of 20 documents retrieved in response to a query from a collection of 10,000 documents. The top of the ranked list (the document the system thinks is most likely to be relevant) is on the left of the list. This list shows 6 relevant documents. Assume that there are 8 relevant documents in total in the collection. R R N N N a. b. c. d. e.
N N N R N
R N N N R
N N N N R
What is the precision of the system on the top 20? What is the F1 on the top 20? What is the uninterpolated precision of the system at 25% recall? What is the interpolated precision at 33% recall? Assume that these 20 documents are the complete result set of the system. What is the MAP for the query?
Assume, now, instead, that the system returned the entire 10,000 documents in a ranked list, and these are the first 20 results returned. f. What is the largest possible MAP that this system could have? g. What is the smallest possible MAP that this system could have? h. In a set of experiments, only the top 20 results are evaluated by hand. The result in (e) is used to approximate the range (f)–(g). For this example, how large (in absolute terms) can the error for the MAP be by calculating (e) instead of (f) and (g) for this query?
POOLING
Assessing relevance To properly evaluate a system, your test information needs must be germane to the documents in the test document collection, and appropriate for predicted usage of the system. These information needs are best designed by domain experts. Using random combinations of query terms as an information need is generally not a good idea because typically they will not resemble the actual distribution of information needs. Given information needs and documents, you need to collect relevance assessments. This is a time-consuming and expensive process involving human beings. For tiny collections like Cranfield, exhaustive judgments of relevance for each query and document pair were obtained. For large modern collections, it is usual for relevance to be assessed only for a subset of the documents for each query. The most standard approach is pooling, where relevance is assessed over a subset of the collection that is formed from the top k documents returned by a number of different IR systems (usually the ones to be evaluated), and perhaps other sources such as the results of Boolean keyword searches or documents found by expert searchers in an interactive process.
8.5 Assessing relevance
Judge 1 Relevance
Yes No Total
Judge 2 Relevance Yes No Total 300 20 320 10 70 80 310 90 400
Observed proportion of the times the judges agreed P( A) = (300 + 70)/400 = 370/400 = 0.925 Pooled marginals P(nonrelevant) = (80 + 90)/(400 + 400) = 170/800 = 0.2125 P(relevant) = (320 + 310)/(400 + 400) = 630/800 = 0.7878 Probability that the two judges agreed by chance P( E) = P(nonrelevant)2 + P(relevant)2 = 0.21252 + 0.78782 = 0.665 Kappa statistic κ = ( P( A) − P( E))/(1 − P( E)) = (0.925 − 0.665)/(1 − 0.665) = 0.776 ◮ Table 8.2 Calculating the kappa statistic.
KAPPA STATISTIC
A human is not a device that reliably reports a gold standard judgment of relevance of a document to a query. Rather, humans and their relevance judgments are quite idiosyncratic and variable. But this is not a problem to be solved: in the final analysis, the success of an IR system depends on how good it is at satisfying the needs of these idiosyncratic humans, one information need at a time. Nevertheless, it is interesting to consider and measure how much agreement between judges there is on relevance judgments. In the social sciences, a common measure for agreement between judges is the kappa statistic. It is designed for categorical judgments and corrects a simple agreement rate for the rate of chance agreement. kappa =
P( A) − P( E) 1 − P( E)
where P( A) is the proportion of the times the judges agreed, and P( E) is the proportion of the times they would be expected to agree by chance. There are choices in how the latter is estimated: if we simply say we are making a two-class decision and assume nothing more, then the expected chance agreement rate is 0.5. However, normally the class distribution assigned is skewed, and it is usual to use marginal statistics to calculate expected agreement.2 There are still two ways to do it depending on whether one pools 2. For a contingency table, as in Table 8.2, a marginal statistic is formed by summing a row or column. The marginal ai.k = ∑ j aijk.
the marginal distribution across judges or uses the marginals for each judge separately; both forms have been used, but we present the pooled version because it is more conservative in the presence of systematic differences in assessments across judges. The calculations are shown in Table 8.2. The kappa value will be 1 if two judges always agree, 0 if they agree only at the rate given by chance, and negative if they are worse than random. If there are more than two judges, it is normal to calculate an average pairwise kappa value. As a rule of thumb, a kappa value above 0.8 is taken as good agreement, a kappa value between 0.67 and 0.8 is taken as fair agreement, and agreement below 0.67 is seen as data providing a dubious basis for an evaluation, though the precise cutoffs depend on the purposes for which the data will be used. Interjudge agreement of relevance has been measured within the TREC evaluations and for medical IR collections. Using the above rules of thumb, the level of agreement normally falls in the range of "fair" (0.67–0.8). The fact that human agreement on a binary relevance judgment is quite modest is one reason for not requiring more fine-grained relevance labeling from the test set creator. To answer the question of whether IR evaluation results are valid despite the variation of individual assessors' judgments, people have experimented with evaluations taking one or the other of two judges' opinions as the gold standard. The choice can make a considerable absolute difference to reported scores, but has in general been found to have little impact on the relative effectiveness ranking of either different systems or variants of a single system which are being compared for effectiveness.
Critiques and justifications of the concept of relevance The advantage of system evaluation, as enabled by the standard model of relevant and nonrelevant documents, is that we have a fixed setting in which we can vary IR systems and system parameters to carry out comparative experiments. Such formal testing is much less expensive and allows clearer diagnosis of the effect of changing system parameters than doing user studies of retrieval effectiveness. Indeed, once we have a formal measure that we have confidence in, we can proceed to optimize effectiveness by machine learning methods, rather than tuning parameters by hand. Of course, if the formal measure poorly describes what users actually want, doing this will not be effective in improving user satisfaction. Our perspective is that, in practice, the standard formal measures for IR evaluation, although a simplification, are good enough, and recent work in optimizing formal evaluation measures in IR has succeeded brilliantly. There are numerous examples of techniques developed in formal evaluation settings, which improve effectiveness in operational settings, such as the development of document length normalization methods within the context of TREC (Sections 6.4.4 and 11.4.3)
MARGINAL RELEVANCE
and machine learning methods for adjusting parameter weights in scoring (Section 6.1.2). That is not to say that there are not problems latent within the abstractions used. The relevance of one document is treated as independent of the relevance of other documents in the collection. (This assumption is actually built into most retrieval systems – documents are scored against queries, not against each other – as well as being assumed in the evaluation methods.) Assessments are binary: there aren't any nuanced assessments of relevance. Relevance of a document to an information need is treated as an absolute, objective decision. But judgments of relevance are subjective, varying across people, as we discussed above. In practice, human assessors are also imperfect measuring instruments, susceptible to failures of understanding and attention. We also have to assume that users' information needs do not change as they start looking at retrieval results. Any results based on one collection are heavily skewed by the choice of collection, queries, and relevance judgment set: the results may not translate from one domain to another or to a different user population. Some of these problems may be fixable. A number of recent evaluations, including INEX, some TREC tracks, and NTCIR have adopted an ordinal notion of relevance with documents divided into 3 or 4 classes, distinguishing slightly relevant documents from highly relevant documents. See Section 10.4 (page 210) for a detailed discussion of how this is implemented in the INEX evaluations. One clear problem with the relevance-based assessment that we have presented is the distinction between relevance and marginal relevance: whether a document still has distinctive usefulness after the user has looked at certain other documents (Carbonell and Goldstein 1998). Even if a document is highly relevant, its information can be completely redundant with other documents which have already been examined. The most extreme case of this is documents that are duplicates – a phenomenon that is actually very common on the World Wide Web – but it can also easily occur when several documents provide a similar precis of an event. In such circumstances, marginal relevance is clearly a better measure of utility to the user. Maximizing marginal relevance requires returning documents that exhibit diversity and novelty. One way to approach measuring this is by using distinct facts or entities as evaluation units. This perhaps more directly measures true utility to the user but doing this makes it harder to create a test collection. Exercise 8.10
Below is a table showing how two human judges rated the relevance of a set of 12 documents to a particular information need (0 = nonrelevant, 1 = relevant). Let us assume that you've written an IR system that for this query returns the set of documents {4, 5, 6, 7, 8}.
8 Evaluation in information retrieval docID 1 2 3 4 5 6 7 8 9 10 11 12
Judge 1 0 0 1 1 1 1 1 1 0 0 0 0
a. Calculate the kappa measure between the two judges. b. Calculate precision, recall, and F1 of your system if a document is considered relevant only if the two judges agree. c. Calculate precision, recall, and F1 of your system if a document is considered relevant if either judge thinks it is relevant.
A broader perspective: System quality and user utility Formal evaluation measures are at some distance from our ultimate interest in measures of human utility: how satisfied is each user with the results the system gives for each information need that they pose? The standard way to measure human satisfaction is by various kinds of user studies. These might include quantitative measures, both objective, such as time to complete a task, as well as subjective, such as a score for satisfaction with the search engine, and qualitative measures, such as user comments on the search interface. In this section we will touch on other system aspects that allow quantitative evaluation and the issue of user utility.
System issues There are many practical benchmarks on which to rate an information retrieval system beyond its retrieval quality. These include: • How fast does it index, that is, how many documents per hour does it index for a certain distribution over document lengths? (cf. Chapter 4) • How fast does it search, that is, what is its latency as a function of index size? • How expressive is its query language? How fast is it on complex queries?
8.6 A broader perspective: System quality and user utility
• How large is its document collection, in terms of the number of documents or the collection having information distributed across a broad range of topics? All these criteria apart from query language expressiveness are straightforwardly measurable: we can quantify the speed or size. Various kinds of feature checklists can make query language expressiveness semi-precise.
User utility What we would really like is a way of quantifying aggregate user happiness, based on the relevance, speed, and user interface of a system. One part of this is understanding the distribution of people we wish to make happy, and this depends entirely on the setting. For a web search engine, happy search users are those who find what they want. One indirect measure of such users is that they tend to return to the same engine. Measuring the rate of return of users is thus an effective metric, which would of course be more effective if you could also measure how much these users used other search engines. But advertisers are also users of modern web search engines. They are happy if customers click through to their sites and then make purchases. On an eCommerce web site, a user is likely to be wanting to purchase something. Thus, we can measure the time to purchase, or the fraction of searchers who become buyers. On a shopfront web site, perhaps both the user's and the store owner's needs are satisfied if a purchase is made. Nevertheless, in general, we need to decide whether it is the end user's or the eCommerce site owner's happiness that we are trying to optimize. Usually, it is the store owner who is paying us. For an "enterprise" (company, government, or academic) intranet search engine, the relevant metric is more likely to be user productivity: how much time do users spend looking for information that they need. There are also many other practical criteria concerning such matters as information security, which we mentioned in Section 4.6 (page 80). User happiness is elusive to measure, and this is part of why the standard methodology uses the proxy of relevance of search results. The standard direct way to get at user satisfaction is to run user studies, where people engage in tasks, and usually various metrics are measured, the participants are observed, and ethnographic interview techniques are used to get qualitative information on satisfaction. User studies are very useful in system design, but they are time consuming and expensive to do. They are also difficult to do well, and expertise is required to design the studies and to interpret the results. We will not discuss the details of human usability testing here.
CLICKTHROUGH LOG ANALYSIS CLICKSTREAM MINING
Refining a deployed system If an IR system has been built and is being used by a large number of users, the system's builders can evaluate possible changes by deploying variant versions of the system and recording measures that are indicative of user satisfaction with one variant vs. others as they are being used. This method is frequently used by web search engines. The most common version of this is A/B testing, a term borrowed from the advertising industry. For such a test, precisely one thing is changed between the current system and a proposed system, and a small proportion of traffic (say, 1–10% of users) is randomly directed to the variant system, while most users use the current system. For example, if we wish to investigate a change to the ranking algorithm, we redirect a random sample of users to a variant system and evaluate measures such as the frequency with which people click on the top result, or any result on the first page. (This particular analysis method is referred to as clickthrough log analysis or clickstream mining. It is further discussed as a method of implicit feedback in Section 9.1.7 (page 187).) The basis of A/B testing is running a bunch of single variable tests (either in sequence or in parallel): for each test only one parameter is varied from the control (the current live system). It is therefore easy to see whether varying each parameter has a positive or negative effect. Such testing of a live system can easily and cheaply gauge the effect of a change on users, and, with a large enough user base, it is practical to measure even very small positive and negative effects. In principle, more analytic power can be achieved by varying multiple things at once in an uncorrelated (random) way, and doing standard multivariate statistical analysis, such as multiple linear regression. In practice, though, A/B testing is widely used, because A/B tests are easy to deploy, easy to understand, and easy to explain to management.
Results snippets Having chosen or ranked the documents matching a query, we wish to present a results list that will be informative to the user. In many cases the user will not want to examine all the returned documents and so we want to make the results list informative enough that the user can do a final ranking of the documents for themselves based on relevance to their information need.3 The standard way of doing this is to provide a snippet, a short summary of the document, which is designed so as to allow the user to decide its relevance. Typically, the snippet consists of the document title and a short 3. There are exceptions, in domains where recall is emphasized. For instance, in many legal disclosure cases, a legal associate will review every document that matches a keyword search.
8.7 Results snippets
STATIC SUMMARY DYNAMIC SUMMARY
TEXT SUMMARIZATION
KEYWORD - IN - CONTEXT
summary, which is automatically extracted. The question is how to design the summary so as to maximize its usefulness to the user. The two basic kinds of summaries are static, which are always the same regardless of the query, and dynamic (or query-dependent), which are customized according to the user's information need as deduced from a query. Dynamic summaries attempt to explain why a particular document was retrieved for the query at hand. A static summary is generally comprised of either or both a subset of the document and metadata associated with the document. The simplest form of summary takes the first two sentences or 50 words of a document, or extracts particular zones of a document, such as the title and author. Instead of zones of a document, the summary can instead use metadata associated with the document. This may be an alternative way to provide an author or date, or may include elements which are designed to give a summary, such as the description metadata which can appear in the meta element of a web HTML page. This summary is typically extracted and cached at indexing time, in such a way that it can be retrieved and presented quickly when displaying search results, whereas having to access the actual document content might be a relatively expensive operation. There has been extensive work within natural language processing (NLP) on better ways to do text summarization. Most such work still aims only to choose sentences from the original document to present and concentrates on how to select good sentences. The models typically combine positional factors, favoring the first and last paragraphs of documents and the first and last sentences of paragraphs, with content factors, emphasizing sentences with key terms, which have low document frequency in the collection as a whole, but high frequency and good distribution across the particular document being returned. In sophisticated NLP approaches, the system synthesizes sentences for a summary, either by doing full text generation or by editing and perhaps combining sentences used in the document. For example, it might delete a relative clause or replace a pronoun with the noun phrase that it refers to. This last class of methods remains in the realm of research and is seldom used for search results: it is easier, safer, and often even better to just use sentences from the original document. Dynamic summaries display one or more "windows" on the document, aiming to present the pieces that have the most utility to the user in evaluating the document with respect to their information need. Usually these windows contain one or several of the query terms, and so are often referred to as keyword-in-context (KWIC) snippets, though sometimes they may still be pieces of the text such as the title that are selected for their queryindependent information value just as in the case of static summarization. Dynamic summaries are generated in conjunction with scoring. If the query is found as a phrase, occurrences of the phrase in the document will be
. . . In recent years, Papua New Guinea has faced severe economic difficulties and economic growth has slowed, partly as a result of weak governance and civil war, and partly as a result of external factors such as the Bougainville civil war which led to the closure in 1989 of the Panguna mine (at that time the most important foreign exchange earner and contributor to Government finances), the Asian financial crisis, a decline in the prices of gold and copper, and a fall in the production of oil. PNG's economic development record over the past few years is evidence that governance issues underly many of the country's problems. Good governance, which may be defined as the transparent and accountable management of human, natural, economic and financial resources for the purposes of equitable and sustainable development, flows from proper public sector management, efficient fiscal and accounting mechanisms, and a willingness to make service delivery a priority in practice. . . . ◮ Figure 8.5 An example of selecting text for a dynamic snippet. This snippet was generated for a document in response to the query new guinea economic development. The figure shows in bold italic where the selected snippet text occurred in the original document.
shown as the summary. If not, windows within the document that contain multiple query terms will be selected. Commonly these windows may just stretch some number of words to the left and right of the query terms. This is a place where NLP techniques can usefully be employed: users prefer snippets that read well because they contain complete phrases. Dynamic summaries are generally regarded as greatly improving the usability of IR systems, but they present a complication for IR system design. A dynamic summary cannot be precomputed, but, on the other hand, if a system has only a positional index, then it cannot easily reconstruct the context surrounding search engine hits in order to generate such a dynamic summary. This is one reason for using static summaries. The standard solution to this in a world of large and cheap disk drives is to locally cache all the documents at index time (notwithstanding that this approach raises various legal, information security and control issues that are far from resolved) as shown in Figure 7.5 (page 147). Then, a system can simply scan a document which is about to appear in a displayed results list to find snippets containing the query words. Beyond simply access to the text, producing a good KWIC snippet requires some care. Given a variety of keyword occurrences in a document, the goal is to choose fragments which are: (i) maximally informative about the discussion of those terms in the document, (ii) self-contained enough to be easy to read, and (iii) short enough to fit within the normally strict constraints on the space available for summaries.
Generating snippets must be fast since the system is typically generating many snippets for each query that it handles. Rather than caching an entire document, it is common to cache only a generous but fixed size prefix of the document, such as perhaps 10,000 characters. For most common, short documents, the entire document is thus cached, but huge amounts of local storage will not be wasted on potentially vast documents. Summaries of documents whose length exceeds the prefix size will be based on material in the prefix only, which is in general a useful zone in which to look for a document summary anyway. If a document has been updated since it was last processed by a crawler and indexer, these changes will be neither in the cache nor in the index. In these circumstances, neither the index nor the summary will accurately reflect the current contents of the document, but it is the differences between the summary and the actual document content that will be more glaringly obvious to the end user.
References and further reading Definition and implementation of the notion of relevance to a query got off to a rocky start in 1953. Swanson (1988) reports that in an evaluation in that year between two teams, they agreed that 1390 documents were variously relevant to a set of 98 questions, but disagreed on a further 1577 documents, and the disagreements were never resolved. Rigorous formal testing of IR systems was first completed in the Cranfield experiments, beginning in the late 1950s. A retrospective discussion of the Cranfield test collection and experimentation with it can be found in (Cleverdon 1991). The other seminal series of early IR experiments were those on the SMART system by Gerard Salton and colleagues (Salton 1971b; 1991). The TREC evaluations are described in detail by Voorhees and Harman (2005). Online information is available at http://trec.nist.gov/. Initially, few researchers computed the statistical significance of their experimental results, but the IR community increasingly demands this (Hull 1993). User studies of IR system effectiveness began more recently (Saracevic and Kantor 1988; 1996). The notions of recall and precision were first used by Kent et al. (1955), although the term precision did not appear until later. The F measure (or, rather its complement E = 1 − F) was introduced by van Rijsbergen (1979). He provides an extensive theoretical discussion, which shows how adopting a principle of decreasing marginal relevance (at some point a user will be unwilling to sacrifice a unit of precision for an added unit of recall) leads to the harmonic mean being the appropriate method for combining precision and recall (and hence to its adoption rather than the minimum or geometric mean).
Buckley and Voorhees (2000) compare several evaluation measures, including precision at k, MAP, and R-precision, and evaluate the error rate of each measure. R-precision was adopted as the official evaluation metric in the TREC HARD track (Allan 2005). Aslam and Yilmaz (2005) examine its surprisingly close correlation to MAP, which had been noted in earlier studies (Tague-Sutcliffe and Blustein 1995, Buckley and Voorhees 2000). A standard program for evaluating IR systems which computes many measures of ranked retrieval effectiveness is Chris Buckley's trec_eval program used in the TREC evaluations. It can be downloaded from: http://trec.nist.gov/trec_eval/. Kekäläinen and Järvelin (2002) argue for the superiority of graded relevance judgments when dealing with very large document collections, and Järvelin and Kekäläinen (2002) introduce cumulated gain-based methods for IR system evaluation in this context. Sakai (2007) does a study of the stability and sensitivity of evaluation measures based on graded relevance judgments from NTCIR tasks, and concludes that NDCG is best for evaluating document ranking. Schamber et al. (1990) examine the concept of relevance, stressing its multidimensional and context-specific nature, but also arguing that it can be measured effectively. (Voorhees 2000) is the standard article for examining variation in relevance judgments and their effects on retrieval system scores and ranking for the TREC Ad Hoc task. Voorhees concludes that although the numbers change, the rankings are quite stable. Hersh et al. (1994) present similar analysis for a medical IR collection. In contrast, Kekäläinen (2005) analyze some of the later TRECs, exploring a 4-way relevance judgment and the notion of cumulative gain, arguing that the relevance measure used does substantially affect system rankings. See also Harter (1998). Zobel (1998) studies whether the pooling method used by TREC to collect a subset of documents that will be evaluated for relevance is reliable and fair, and concludes that it is. The kappa statistic and its use for language-related purposes is discussed by Carletta (1996). Many standard sources (e.g., Siegel and Castellan 1988) present pooled calculation of the expected agreement, but Di Eugenio and Glass (2004) argue for preferring the unpooled agreement (though perhaps presenting multiple measures). For further discussion of alternative measures of agreement, which may in fact be better, see Lombard et al. (2002) and Krippendorff (2003). Text summarization has been actively explored for many years. Modern work on sentence selection was initiated by Kupiec et al. (1995). More recent work includes (Barzilay and Elhadad 1997) and (Jing 2000), together with a broad selection of work appearing at the yearly DUC conferences and at other NLP venues. Tombros and Sanderson (1998) demonstrate the advantages of dynamic summaries in the IR context. Turpin et al. (2007) address how to generate snippets efficiently.
Clickthrough log analysis is studied in (Joachims 2002b, Joachims et al. 2005). In a series of papers, Hersh, Turpin and colleagues show how improvements in formal retrieval effectiveness, as evaluated in batch experiments, do not always translate into an improved system for users (Hersh et al. 2000a;b; 2001, Turpin and Hersh 2001; 2002). User interfaces for IR and human factors such as models of human information seeking and usability testing are outside the scope of what we cover in this book. More information on these topics can be found in other textbooks, including (Baeza-Yates and Ribeiro-Neto 1999, ch. 10) and (Korfhage 1997), and collections focused on cognitive aspects (Spink and Cole 2005).
9 SYNONYMY
Relevance feedback and query expansion
In most collections, the same concept may be referred to using different words. This issue, known as synonymy, has an impact on the recall of most information retrieval systems. For example, you would want a search for aircraft to match plane (but only for references to an airplane, not a woodworking plane), and for a search on thermodynamics to match references to heat in appropriate discussions. Users often attempt to address this problem themselves by manually refining a query, as was discussed in Section 1.4; in this chapter we discuss ways in which a system can help with query refinement, either fully automatically or with the user in the loop. The methods for tackling this problem split into two major classes: global methods and local methods. Global methods are techniques for expanding or reformulating query terms independent of the query and results returned from it, so that changes in the query wording will cause the new query to match other semantically similar terms. Global methods include: • Query expansion/reformulation with a thesaurus or WordNet (Section 9.2.2) • Query expansion via automatic thesaurus generation (Section 9.2.3) • Techniques like spelling correction (discussed in Chapter 3) Local methods adjust a query relative to the documents that initially appear to match the query. The basic methods here are: • Relevance feedback (Section 9.1) • Pseudo relevance feedback, also known as Blind relevance feedback (Section 9.1.6) • (Global) indirect relevance feedback (Section 9.1.7) In this chapter, we will mention all of these approaches, but we will concentrate on relevance feedback, which is one of the most used and most successful approaches.
9.1 RELEVANCE FEEDBACK
Relevance feedback and pseudo relevance feedback The idea of relevance feedback (RF) is to involve the user in the retrieval process so as to improve the final result set. In particular, the user gives feedback on the relevance of documents in an initial set of results. The basic procedure is: • The user issues a (short, simple) query. • The system returns an initial set of retrieval results. • The user marks some returned documents as relevant or nonrelevant. • The system computes a better representation of the information need based on the user feedback. • The system displays a revised set of retrieval results. Relevance feedback can go through one or more iterations of this sort. The process exploits the idea that it may be difficult to formulate a good query when you don't know the collection well, but it is easy to judge particular documents, and so it makes sense to engage in iterative query refinement of this sort. In such a scenario, relevance feedback can also be effective in tracking a user's evolving information need: seeing some documents may lead users to refine their understanding of the information they are seeking. Image search provides a good example of relevance feedback. Not only is it easy to see the results at work, but this is a domain where a user can easily have difficulty formulating what they want in words, but can easily indicate relevant or nonrelevant images. After the user enters an initial query for bike on the demonstration system at: http://nayana.ece.ucsb.edu/imsearch/imsearch.html
the initial results (in this case, images) are returned. In Figure 9.1 (a), the user has selected some of them as relevant. These will be used to refine the query, while other displayed results have no effect on the reformulation. Figure 9.1 (b) then shows the new top-ranked results calculated after this round of relevance feedback. Figure 9.2 shows a textual IR example where the user wishes to find out about new applications of space satellites.
The Rocchio algorithm for relevance feedback The Rocchio Algorithm is the classic algorithm for implementing relevance feedback. It models a way of incorporating relevance feedback information into the vector space model of Section 6.3.
9.1 Relevance feedback and pseudo relevance feedback
(b) ◮ Figure 9.1 Relevance feedback searching over images. (a) The user views the initial query results for a query of bike, selects the first, third and fourth result in the top row and the fourth result in the bottom row as relevant, and submits this feedback. (b) The users sees the revised result set. Precision is greatly improved. From http://nayana.ece.ucsb.edu/imsearch/imsearch.html (Newsam et al. 2001).
Query: New space satellite applications
2.074 new 15.106 space 30.816 satellite 5.660 application 5.991 nasa 5.196 eos 4.196 launch 3.972 aster 3.516 instrument 3.446 arianespace 3.004 bundespost 2.806 ss 2.790 rocket 2.053 scientist 2.003 broadcast 1.172 earth 0.836 oil 0.646 measure
1. 0.539, 08/13/91, NASA Hasn't Scrapped Imaging Spectrometer 2. 0.533, 07/09/91, NASA Scratches Environment Gear From Satellite Plan 3. 0.528, 04/04/90, Science Panel Backs NASA Satellite Plan, But Urges Launches of Smaller Probes 4. 0.526, 09/09/91, A NASA Satellite Project Accomplishes Incredible Feat: Staying Within Budget 5. 0.525, 07/24/90, Scientist Who Exposed Global Warming Proposes Satellites for Climate Research 6. 0.524, 08/22/90, Report Provides Support for the Critics Of Using Big Satellites to Study Climate 7. 0.516, 04/13/87, Arianespace Receives Satellite Launch Pact From Telesat Canada 8. 0.509, 12/02/87, Telecommunications Tale of Two Companies
1. 0.513, 07/09/91, NASA Scratches Environment Gear From Satellite Plan 2. 0.500, 08/13/91, NASA Hasn't Scrapped Imaging Spectrometer 3. 0.493, 08/07/89, When the Pentagon Launches a Secret Satellite, Space Sleuths Do Some Spy Work of Their Own 4. 0.493, 07/31/89, NASA Uses 'Warm' Superconductors For Fast Circuit 5. 0.492, 12/02/87, Telecommunications Tale of Two Companies 6. 0.491, 07/09/91, Soviets May Adapt Parts of SS-20 Missile For Commercial Use 7. 0.490, 07/12/88, Gaping Gap: Pentagon Lags in Race To Match the Soviets In Rocket Launchers 8. 0.490, 06/14/90, Rescue of Satellite By Space Agency To Cost $90 Million
◮ Figure 9.2 Example of relevance feedback on a text collection. (a) The initial query (a). (b) The user marks some relevant documents (shown with a plus sign). (c) The query is then expanded by 18 terms with weights as shown. (d) The revised top results are then shown. A * marks the documents which were judged relevant in the relevance feedback phase.
◮ Figure 9.3 The Rocchio optimal query for separating relevant and nonrelevant documents.
The underlying theory. We want to find a query vector, denoted as ~q, that maximizes similarity with relevant documents while minimizing similarity with nonrelevant documents. If Cr is the set of relevant documents and Cnr is the set of nonrelevant documents, then we wish to find:1 (9.1)
~qopt = arg max[sim(~q, Cr ) − sim(~q, Cnr )], ~q
where sim is defined as in Equation 6.10. Under cosine similarity, the optimal query vector ~qopt for separating the relevant and nonrelevant documents is: (9.2)
~qopt =
1 |Cr |
∑ d~ j ∈ Cr
d~j −
1 |Cnr |
d~j
d~ j ∈ Cnr
That is, the optimal query is the vector difference between the centroids of the relevant and nonrelevant documents; see Figure 9.3. However, this observation is not terribly useful, precisely because the full set of relevant documents is not known: it is what we want to find. R OCCHIO ALGORITHM
The Rocchio (1971) algorithm. This was the relevance feedback mecha1. In the equation, arg maxx f ( x ) returns a value of x which maximizes the value of the function f ( x ). Similarly, arg min x f ( x ) returns a value of x which minimizes the value of the function f ( x ).
◮ Figure 9.4 An application of Rocchio's algorithm. Some documents have been labeled as relevant and nonrelevant and the initial query vector is moved in response to this feedback.
nism introduced in and popularized by Salton's SMART system around 1970. In a real IR query context, we have a user query and partial knowledge of known relevant and nonrelevant documents. The algorithm proposes using the modified query ~qm : (9.3)
~qm = α~q0 + β
1 | Dr |
∑ d~ j ∈ Dr
d~j − γ
1 | Dnr |
d~j ∈ Dnr
where q0 is the original query vector, Dr and Dnr are the set of known relevant and nonrelevant documents respectively, and α, β, and γ are weights attached to each term. These control the balance between trusting the judged document set versus the query: if we have a lot of judged documents, we would like a higher β and γ. Starting from q0 , the new query moves you some distance toward the centroid of the relevant documents and some distance away from the centroid of the nonrelevant documents. This new query can be used for retrieval in the standard vector space model (see Section 6.3). We can easily leave the positive quadrant of the vector space by subtracting off a nonrelevant document's vector. In the Rocchio algorithm, negative term weights are ignored. That is, the term weight is set to 0. Figure 9.4 shows the effect of applying relevance feedback. Relevance feedback can improve both recall and precision. But, in practice, it has been shown to be most useful for increasing recall in situations
I DE DEC - HI
where recall is important. This is partly because the technique expands the query, but it is also partly an effect of the use case: when they want high recall, users can be expected to take time to review results and to iterate on the search. Positive feedback also turns out to be much more valuable than negative feedback, and so most IR systems set γ < β. Reasonable values might be α = 1, β = 0.75, and γ = 0.15. In fact, many systems, such as the image search system in Figure 9.1, allow only positive feedback, which is equivalent to setting γ = 0. Another alternative is to use only the marked nonrelevant document which received the highest ranking from the IR system as negative feedback (here, | Dnr | = 1 in Equation (9.3)). While many of the experimental results comparing various relevance feedback variants are rather inconclusive, some studies have suggested that this variant, called Ide dec-hi is the most effective or at least the most consistent performer.
Probabilistic relevance feedback Rather than reweighting the query in a vector space, if a user has told us some relevant and nonrelevant documents, then we can proceed to build a classifier. One way of doing this is with a Naive Bayes probabilistic model. If R is a Boolean indicator variable expressing the relevance of a document, then we can estimate P( xt = 1| R), the probability of a term t appearing in a document, depending on whether it is relevant or not, as:
Pˆ ( x t = 1| R = 1) Pˆ ( x t = 1| R = 0)
= |VRt |/|VR| = (d f t − |VRt |)/( N − |VR|)
where N is the total number of documents, d f t is the number that contain t, VR is the set of known relevant documents, and VRt is the subset of this set containing t. Even though the set of known relevant documents is a perhaps small subset of the true set of relevant documents, if we assume that the set of relevant documents is a small subset of the set of all documents then the estimates given above will be reasonable. This gives a basis for another way of changing the query term weights. We will discuss such probabilistic approaches more in Chapters 11 and 13, and in particular outline the application to relevance feedback in Section 11.3.4 (page 228). For the moment, observe that using just Equation (9.4) as a basis for term-weighting is likely insufficient. The equations use only collection statistics and information about the term distribution within the documents judged relevant. They preserve no memory of the original query.
When does relevance feedback work? The success of relevance feedback depends on certain assumptions. Firstly, the user has to have sufficient knowledge to be able to make an initial query
which is at least somewhere close to the documents they desire. This is needed anyhow for successful information retrieval in the basic case, but it is important to see the kinds of problems that relevance feedback cannot solve alone. Cases where relevance feedback alone is not sufficient include: • Misspellings. If the user spells a term in a different way to the way it is spelled in any document in the collection, then relevance feedback is unlikely to be effective. This can be addressed by the spelling correction techniques of Chapter 3. • Cross-language information retrieval. Documents in another language are not nearby in a vector space based on term distribution. Rather, documents in the same language cluster more closely together. • Mismatch of searcher's vocabulary versus collection vocabulary. If the user searches for laptop but all the documents use the term notebook computer, then the query will fail, and relevance feedback is again most likely ineffective. Secondly, the relevance feedback approach requires relevant documents to be similar to each other. That is, they should cluster. Ideally, the term distribution in all relevant documents will be similar to that in the documents marked by the users, while the term distribution in all nonrelevant documents will be different from those in relevant documents. Things will work well if all relevant documents are tightly clustered around a single prototype, or, at least, if there are different prototypes, if the relevant documents have significant vocabulary overlap, while similarities between relevant and nonrelevant documents are small. Implicitly, the Rocchio relevance feedback model treats relevant documents as a single cluster, which it models via the centroid of the cluster. This approach does not work as well if the relevant documents are a multimodal class, that is, they consist of several clusters of documents within the vector space. This can happen with: • Subsets of the documents using different vocabulary, such as Burma vs. Myanmar
• A query for which the answer set is inherently disjunctive, such as Pop stars who once worked at Burger King. • Instances of a general concept, which often appear as a disjunction of more specific concepts, for example, felines. Good editorial content in the collection can often provide a solution to this problem. For example, an article on the attitudes of different groups to the situation in Burma could introduce the terminology used by different parties, thus linking the document clusters.
Relevance feedback is not necessarily popular with users. Users are often reluctant to provide explicit feedback, or in general do not wish to prolong the search interaction. Furthermore, it is often harder to understand why a particular document was retrieved after relevance feedback is applied. Relevance feedback can also have practical problems. The long queries that are generated by straightforward application of relevance feedback techniques are inefficient for a typical IR system. This results in a high computing cost for the retrieval and potentially long response times for the user. A partial solution to this is to only reweight certain prominent terms in the relevant documents, such as perhaps the top 20 terms by term frequency. Some experimental results have also suggested that using a limited number of terms like this may give better results (Harman 1992) though other work has suggested that using more terms is better in terms of retrieved document quality (Buckley et al. 1994b).
Relevance feedback on the web Some web search engines offer a similar/related pages feature: the user indicates a document in the results set as exemplary from the standpoint of meeting his information need and requests more documents like it. This can be viewed as a particular simple form of relevance feedback. However, in general relevance feedback has been little used in web search. One exception was the Excite web search engine, which initially provided full relevance feedback. However, the feature was in time dropped, due to lack of use. On the web, few people use advanced search interfaces and most would like to complete their search in a single interaction. But the lack of uptake also probably reflects two other factors: relevance feedback is hard to explain to the average user, and relevance feedback is mainly a recall enhancing strategy, and web search users are only rarely concerned with getting sufficient recall. Spink et al. (2000) present results from the use of relevance feedback in the Excite search engine. Only about 4% of user query sessions used the relevance feedback option, and these were usually exploiting the "More like this" link next to each result. About 70% of users only looked at the first page of results and did not pursue things any further. For people who used relevance feedback, results were improved about two thirds of the time. An important more recent thread of work is the use of clickstream data (what links a user clicks on) to provide indirect relevance feedback. Use of this data is studied in detail in (Joachims 2002b, Joachims et al. 2005). The very successful use of web link structure (see Chapter 21) can also be viewed as implicit feedback, but provided by page authors rather than readers (though in practice most authors are also readers).
Exercise 9.1 In Rocchio's algorithm, what weight setting for α/β/γ does a "Find pages like this one" search correspond to? Exercise 9.2
Give three reasons why relevance feedback has been little used in web search.
Evaluation of relevance feedback strategies Interactive relevance feedback can give very substantial gains in retrieval performance. Empirically, one round of relevance feedback is often very useful. Two rounds is sometimes marginally more useful. Successful use of relevance feedback requires enough judged documents, otherwise the process is unstable in that it may drift away from the user's information need. Accordingly, having at least five judged documents is recommended. There is some subtlety to evaluating the effectiveness of relevance feedback in a sound and enlightening way. The obvious first strategy is to start with an initial query q0 and to compute a precision-recall graph. Following one round of feedback from the user, we compute the modified query qm and again compute a precision-recall graph. Here, in both rounds we assess performance over all documents in the collection, which makes comparisons straightforward. If we do this, we find spectacular gains from relevance feedback: gains on the order of 50% in mean average precision. But unfortunately it is cheating. The gains are partly due to the fact that known relevant documents (judged by the user) are now ranked higher. Fairness demands that we should only evaluate with respect to documents not seen by the user. A second idea is to use documents in the residual collection (the set of documents minus those assessed relevant) for the second round of evaluation. This seems like a more realistic evaluation. Unfortunately, the measured performance can then often be lower than for the original query. This is particularly the case if there are few relevant documents, and so a fair proportion of them have been judged by the user in the first round. The relative performance of variant relevance feedback methods can be validly compared, but it is difficult to validly compare performance with and without relevance feedback because the collection size and the number of relevant documents changes from before the feedback to after it. Thus neither of these methods is fully satisfactory. A third method is to have two collections, one which is used for the initial query and relevance judgments, and the second that is then used for comparative evaluation. The performance of both q0 and qm can be validly compared on the second collection. Perhaps the best evaluation of the utility of relevance feedback is to do user studies of its effectiveness, in particular by doing a time-based comparison:
Term weighting lnc.ltc Lnu.ltu
Precision at k = 50 no RF pseudo RF 64.2% 72.7% 74.2% 87.0%
◮ Figure 9.5 Results showing pseudo relevance feedback greatly improving performance. These results are taken from the Cornell SMART system at TREC 4 (Buckley et al. 1995), and also contrast the use of two different length normalization schemes (L vs. l); cf. Figure 6.15 (page 128). Pseudo relevance feedback consisted of adding 20 terms to each query.
how fast does a user find relevant documents with relevance feedback vs. another strategy (such as query reformulation), or alternatively, how many relevant documents does a user find in a certain amount of time. Such notions of user utility are fairest and closest to real system usage.
9.1.6 PSEUDO RELEVANCE FEEDBACK BLIND RELEVANCE FEEDBACK
9.1.7 IMPLICIT RELEVANCE FEEDBACK
Pseudo relevance feedback Pseudo relevance feedback, also known as blind relevance feedback, provides a method for automatic local analysis. It automates the manual part of relevance feedback, so that the user gets improved retrieval performance without an extended interaction. The method is to do normal retrieval to find an initial set of most relevant documents, to then assume that the top k ranked documents are relevant, and finally to do relevance feedback as before under this assumption. This automatic technique mostly works. Evidence suggests that it tends to work better than global analysis (Section 9.2). It has been found to improve performance in the TREC ad hoc task. See for example the results in Figure 9.5. But it is not without the dangers of an automatic process. For example, if the query is about copper mines and the top several documents are all about mines in Chile, then there may be query drift in the direction of documents on Chile.
Indirect relevance feedback We can also use indirect sources of evidence rather than explicit feedback on relevance as the basis for relevance feedback. This is often called implicit (relevance) feedback. Implicit feedback is less reliable than explicit feedback, but is more useful than pseudo relevance feedback, which contains no evidence of user judgments. Moreover, while users are often reluctant to provide explicit feedback, it is easy to collect implicit feedback in large quantities for a high volume system, such as a web search engine.
CLICKSTREAM MINING
On the web, DirectHit introduced the idea of ranking more highly documents that users chose to look at more often. In other words, clicks on links were assumed to indicate that the page was likely relevant to the query. This approach makes various assumptions, such as that the document summaries displayed in results lists (on whose basis users choose which documents to click on) are indicative of the relevance of these documents. In the original DirectHit search engine, the data about the click rates on pages was gathered globally, rather than being user or query specific. This is one form of the general area of clickstream mining. Today, a closely related approach is used in ranking the advertisements that match a web search query (Chapter 19).
Summary Relevance feedback has been shown to be very effective at improving relevance of results. Its successful use requires queries for which the set of relevant documents is medium to large. Full relevance feedback is often onerous for the user, and its implementation is not very efficient in most IR systems. In many cases, other types of interactive retrieval may improve relevance by about as much with less work. Beyond the core ad hoc retrieval scenario, other uses of relevance feedback include: • Following a changing information need (e.g., names of car models of interest change over time) • Maintaining an information filter (e.g., for a news feed). Such filters are discussed further in Chapter 13. • Active learning (deciding which examples it is most useful to know the class of to reduce annotation costs).
Exercise 9.3 Under what conditions would the modified query q m in Equation 9.3 be the same as the original query q0 ? In all other cases, is q m closer than q0 to the centroid of the relevant documents? Exercise 9.4 Why is positive feedback likely to be more useful than negative feedback to an IR system? Why might only using one nonrelevant document be more effective than using several? Exercise 9.5 Suppose that a user's initial query is cheap CDs cheap DVDs extremely cheap CDs. The user examines two documents, d1 and d2 . She judges d1 , with the content CDs cheap software cheap CDs relevant and d2 with content cheap thrills DVDs nonrelevant. Assume that we are using direct term frequency (with no scaling and no document
9.2 Global methods for query reformulation
frequency). There is no need to length-normalize vectors. Using Rocchio relevance feedback as in Equation (9.3) what would the revised query vector be after relevance feedback? Assume α = 1, β = 0.75, γ = 0.25. Exercise 9.6
Omar has implemented a relevance feedback web search system, where he is going to do relevance feedback based only on words in the title text returned for a page (for efficiency). The user is going to rank 3 results. The first user, Jinxing, queries for: banana slug
and the top three titles returned are: banana slug Ariolimax columbianus Santa Cruz mountains banana slug Santa Cruz Campus Mascot Jinxing judges the first two documents relevant, and the third nonrelevant. Assume that Omar's search engine uses term frequency but no length normalization nor IDF. Assume that he is using the Rocchio relevance feedback mechanism, with α = β = γ = 1. Show the final revised query that would be run. (Please list the vector elements in alphabetical order.)
Global methods for query reformulation In this section we more briefly discuss three global methods for expanding a query: by simply aiding the user in doing so, by using a manual thesaurus, and through building a thesaurus automatically.
Vocabulary tools for query reformulation Various user supports in the search process can help the user see how their searches are or are not working. This includes information about words that were omitted from the query because they were on stop lists, what words were stemmed to, the number of hits on each term or phrase, and whether words were dynamically turned into phrases. The IR system might also suggest search terms by means of a thesaurus or a controlled vocabulary. A user can also be allowed to browse lists of the terms that are in the inverted index, and thus find good terms that appear in the collection.
QUERY EXPANSION
Query expansion In relevance feedback, users give additional input on documents (by marking documents in the results set as relevant or not), and this input is used to reweight the terms in the query for documents. In query expansion on the other hand, users give additional input on query words or phrases, possibly suggesting additional query terms. Some search engines (especially on the
◮ Figure 9.6 An example of query expansion in the interface of the Yahoo! web search engine in 2006. The expanded query suggestions appear just below the "Search Results" bar.
web) suggest related queries in response to a query; the users then opt to use one of these alternative query suggestions. Figure 9.6 shows an example of query suggestion options being presented in the Yahoo! web search engine. The central question in this form of query expansion is how to generate alternative or expanded queries for the user. The most common form of query expansion is global analysis, using some form of thesaurus. For each term t in a query, the query can be automatically expanded with synonyms and related words of t from the thesaurus. Use of a thesaurus can be combined with ideas of term weighting: for instance, one might weight added terms less than original query terms. Methods for building a thesaurus for query expansion include: • Use of a controlled vocabulary that is maintained by human editors. Here, there is a canonical term for each concept. The subject headings of traditional library subject indexes, such as the Library of Congress Subject Headings, or the Dewey Decimal system are examples of a controlled vocabulary. Use of a controlled vocabulary is quite common for wellresourced domains. A well-known example is the Unified Medical Language System (UMLS) used with MedLine for querying the biomedical research literature. For example, in Figure 9.7, neoplasms was added to a
• User query: cancer • PubMed query: ("neoplasms"[TIAB] NOT Medline[SB]) OR "neoplasms"[MeSH Terms] OR cancer[Text Word]
• User query: skin itch • PubMed query: ("skin"[MeSH Terms] OR ("integumentary system"[TIAB] NOT Medline[SB]) OR "integumentary system"[MeSH Terms] OR skin[Text Word]) AND (("pruritus"[TIAB] NOT Medline[SB]) OR "pruritus"[MeSH Terms] OR itch[Text Word])
◮ Figure 9.7 Examples of query expansion via the PubMed thesaurus. When a user issues a query on the PubMed interface to Medline at http://www.ncbi.nlm.nih.gov/entrez/, their query is mapped on to the Medline vocabulary as shown.
search for cancer. This Medline query expansion also contrasts with the Yahoo! example. The Yahoo! interface is a case of interactive query expansion, whereas PubMed does automatic query expansion. Unless the user chooses to examine the submitted query, they may not even realize that query expansion has occurred. • A manual thesaurus. Here, human editors have built up sets of synonymous names for concepts, without designating a canonical term. The UMLS metathesaurus is one example of a thesaurus. Statistics Canada maintains a thesaurus of preferred terms, synonyms, broader terms, and narrower terms for matters on which the government collects statistics, such as goods and services. This thesaurus is also bilingual English and French. • An automatically derived thesaurus. Here, word co-occurrence statistics over a collection of documents in a domain are used to automatically induce a thesaurus; see Section 9.2.3. • Query reformulations based on query log mining. Here, we exploit the manual query reformulations of other users to make suggestions to a new user. This requires a huge query volume, and is thus particularly appropriate to web search. Thesaurus-based query expansion has the advantage of not requiring any user input. Use of query expansion generally increases recall and is widely used in many science and engineering fields. As well as such global analysis techniques, it is also possible to do query expansion by local analysis, for instance, by analyzing the documents in the result set. User input is now
Word absolutely bottomed captivating doghouse makeup mediating keeping lithographs pathogens senses
Nearest neighbors absurd, whatsoever, totally, exactly, nothing dip, copper, drops, topped, slide, trimmed shimmer, stunningly, superbly, plucky, witty dog, porch, crawling, beside, downstairs repellent, lotion, glossy, sunscreen, skin, gel reconciliation, negotiate, case, conciliation hoping, bring, wiping, could, some, would drawings, Picasso, Dali, sculptures, Gauguin toxins, bacteria, organisms, bacterial, parasite grasp, psyche, truly, clumsy, naive, innate
◮ Figure 9.8 An example of an automatically generated thesaurus. This example is based on the work in Schütze (1998), which employs latent semantic indexing (see Chapter 18).
usually required, but a distinction remains as to whether the user is giving feedback on documents or on query terms.
Automatic thesaurus generation As an alternative to the cost of a manual thesaurus, we could attempt to generate a thesaurus automatically by analyzing a collection of documents. There are two main approaches. One is simply to exploit word cooccurrence. We say that words co-occurring in a document or paragraph are likely to be in some sense similar or related in meaning, and simply count text statistics to find the most similar words. The other approach is to use a shallow grammatical analysis of the text and to exploit grammatical relations or grammatical dependencies. For example, we say that entities that are grown, cooked, eaten, and digested, are more likely to be food items. Simply using word cooccurrence is more robust (it cannot be misled by parser errors), but using grammatical relations is more accurate. The simplest way to compute a co-occurrence thesaurus is based on termterm similarities. We begin with a term-document matrix A, where each cell At,d is a weighted count wt,d for term t and document d, with weighting so A has length-normalized rows. If we then calculate C = AA T , then Cu,v is a similarity score between terms u and v, with a larger number being better. Figure 9.8 shows an example of a thesaurus derived in basically this manner, but with an extra step of dimensionality reduction via Latent Semantic Indexing, which we discuss in Chapter 18. While some of the thesaurus terms are good or at least suggestive, others are marginal or bad. The quality of the associations is typically a problem. Term ambiguity easily introduces irrel-
evant statistically correlated terms. For example, a query for Apple computer may expand to Apple red fruit computer. In general these thesauri suffer from both false positives and false negatives. Moreover, since the terms in the automatic thesaurus are highly correlated in documents anyway (and often the collection used to derive the thesaurus is the same as the one being indexed), this form of query expansion may not retrieve many additional documents. Query expansion is often effective in increasing recall. However, there is a high cost to manually producing a thesaurus and then updating it for scientific and terminological developments within a field. In general a domainspecific thesaurus is required: general thesauri and dictionaries give far too little coverage of the rich domain-particular vocabularies of most scientific fields. However, query expansion may also significantly decrease precision, particularly when the query contains ambiguous terms. For example, if the user searches for interest rate, expanding the query to interest rate fascinate evaluate is unlikely to be useful. Overall, query expansion is less successful than relevance feedback, though it may be as good as pseudo relevance feedback. It does, however, have the advantage of being much more understandable to the system user.
Exercise 9.7 If A is simply a Boolean cooccurrence matrix, then what do you get as the entries in C?
References and further reading Work in information retrieval quickly confronted the problem of variant expression which meant that the words in a query might not appear in a document, despite it being relevant to the query. An early experiment about 1960 cited by Swanson (1988) found that only 11 out of 23 documents properly indexed under the subject toxicity had any use of a word containing the stem toxi. There is also the issue of translation, of users knowing what terms a document will use. Blair and Maron (1985) conclude that "it is impossibly difficult for users to predict the exact words, word combinations, and phrases that are used by all (or most) relevant documents and only (or primarily) by those documents". The main initial papers on relevance feedback using vector space models all appear in Salton (1971b), including the presentation of the Rocchio algorithm (Rocchio 1971) and the Ide dec-hi variant along with evaluation of several variants (Ide 1971). Another variant is to regard all documents in the collection apart from those judged relevant as nonrelevant, rather than only ones that are explicitly judged nonrelevant. However, Schütze et al. (1995) and Singhal et al. (1997) show that better results are obtained for routing by using only documents close to the query of interest rather than all
documents. Other later work includes Salton and Buckley (1990), Riezler et al. (2007) (a statistical NLP approach to RF) and the recent survey paper Ruthven and Lalmas (2003). The effectiveness of interactive relevance feedback systems is discussed in (Salton 1989, Harman 1992, Buckley et al. 1994b). Koenemann and Belkin (1996) do user studies of the effectiveness of relevance feedback. Traditionally Roget's thesaurus has been the best known English language thesaurus (Roget 1946). In recent computational work, people almost always use WordNet (Fellbaum 1998), not only because it is free, but also because of its rich link structure. It is available at: http://wordnet.princeton.edu. Qiu and Frei (1993) and Schütze (1998) discuss automatic thesaurus generation. Xu and Croft (1996) explore using both local and global query expansion.
XML retrieval
Information retrieval systems are often contrasted with relational databases. Traditionally, IR systems have retrieved information from unstructured text – by which we mean "raw" text without markup. Databases are designed for querying relational data: sets of records that have values for predefined attributes such as employee number, title and salary. There are fundamental differences between information retrieval and database systems in terms of retrieval model, data structures and query language as shown in Table 10.1.1 Some highly structured text search problems are most efficiently handled by a relational database, for example, if the employee table contains an attribute for short textual job descriptions and you want to find all employees who are involved with invoicing. In this case, the SQL query: select lastname from employees where job_desc like 'invoic%';
STRUCTURED RETRIEVAL
may be sufficient to satisfy your information need with high precision and recall. However, many structured data sources containing text are best modeled as structured documents rather than relational data. We call the search over such structured documents structured retrieval. Queries in structured retrieval can be either structured or unstructured, but we will assume in this chapter that the collection consists only of structured documents. Applications of structured retrieval include digital libraries, patent databases, blogs, text in which entities like persons and locations have been tagged (in a process called named entity tagging) and output from office suites like OpenOffice that save documents as marked up text. In all of these applications, we want to be able to run queries that combine textual criteria with structural criteria. Examples of such queries are give me a full-length article on fast fourier transforms (digital libraries), give me patents whose claims mention RSA public key encryption 1. In most modern database systems, one can enable full-text search for text columns. This usually means that an inverted index is created and Boolean or vector space search enabled, effectively combining core database with information retrieval technologies.
10 XML retrieval
objects model main data structure queries
RDB search records relational model table SQL
unstructured retrieval unstructured documents vector space & others inverted index free text queries
structured retrieval trees with text at leaves ? ? ?
◮ Table 10.1 RDB (relational database) search, unstructured information retrieval and structured information retrieval. There is no consensus yet as to which methods work best for structured retrieval although many researchers believe that XQuery (page 215) will become the standard for structured queries.
and that cite US patent 4,405,829 (patents), or give me articles about sightseeing tours of the Vatican and the Coliseum (entity-tagged text). These three queries
DATA - CENTRIC XML
are structured queries that cannot be answered well by an unranked retrieval system. As we argued in Example 1.1 (page 15) unranked retrieval models like the Boolean model suffer from low recall. For instance, an unranked system would return a potentially large number of articles that mention the Vatican, the Coliseum and sightseeing tours without ranking the ones that are most relevant for the query first. Most users are also notoriously bad at precisely stating structural constraints. For instance, users may not know for which structured elements the search system supports search. In our example, the user may be unsure whether to issue the query as sightseeing AND ( COUNTRY:Vatican OR LANDMARK :Coliseum) , as sightseeing AND ( STATE :Vatican OR BUILDING :Coliseum) or in some other form. Users may also be completely unfamiliar with structured search and advanced search interfaces or unwilling to use them. In this chapter, we look at how ranked retrieval methods can be adapted to structured documents to address these problems. We will only look at one standard for encoding structured documents: Extensible Markup Language or XML, which is currently the most widely used such standard. We will not cover the specifics that distinguish XML from other types of markup such as HTML and SGML. But most of what we say in this chapter is applicable to markup languages in general. In the context of information retrieval, we are only interested in XML as a language for encoding text and documents. A perhaps more widespread use of XML is to encode non-text data. For example, we may want to export data in XML format from an enterprise resource planning system and then read them into an analytics program to produce graphs for a presentation. This type of application of XML is called data-centric because numerical and non-text attribute-value data dominate and text is usually a small fraction of the overall data. Most data-centric XML is stored in databases – in contrast to the inverted index-based methods for text-centric XML that we present in this chapter.
10.1 Basic XML concepts
SEMISTRUCTURED RETRIEVAL
10.1 XML ELEMENT XML ATTRIBUTE
XML DOM
We call XML retrieval structured retrieval in this chapter. Some researchers prefer the term semistructured retrieval to distinguish XML retrieval from database querying. We have adopted the terminology that is widespread in the XML retrieval community. For instance, the standard way of referring to XML queries is structured queries, not semistructured queries. The term structured retrieval is rarely used for database querying and it always refers to XML retrieval in this book. There is a second type of information retrieval problem that is intermediate between unstructured retrieval and querying a relational database: parametric and zone search, which we discussed in Section 6.1 (page 110). In the data model of parametric and zone search, there are parametric fields (relational attributes like date or file-size) and zones – text attributes that each take a chunk of unstructured text as value, e.g., author and title in Figure 6.1 (page 111). The data model is flat, that is, there is no nesting of attributes. The number of attributes is small. In contrast, XML documents have the more complex tree structure that we see in Figure 10.2 in which attributes are nested. The number of attributes and nodes is greater than in parametric and zone search. After presenting the basic concepts of XML in Section 10.1, this chapter first discusses the challenges we face in XML retrieval (Section 10.2). Next we describe a vector space model for XML retrieval (Section 10.3). Section 10.4 presents INEX, a shared task evaluation that has been held for a number of years and currently is the most important venue for XML retrieval research. We discuss the differences between data-centric and text-centric approaches to XML in Section 10.5.
Basic XML concepts An XML document is an ordered, labeled tree. Each node of the tree is an XML element and is written with an opening and closing tag. An element can have one or more XML attributes. In the XML document in Figure 10.1, the scene element is enclosed by the two tags and . It has an attribute number with value vii and two child elements, title and verse. Figure 10.2 shows Figure 10.1 as a tree. The leaf nodes of the tree consist of text, e.g., Shakespeare, Macbeth, and Macbeth's castle. The tree's internal nodes encode either the structure of the document (title, act, and scene) or metadata functions (author). The standard for accessing and processing XML documents is the XML Document Object Model or DOM. The DOM represents elements, attributes and text within elements as nodes in a tree. Figure 10.2 is a simplified DOM representation of the XML document in Figure 10.1.2 With a DOM API, we 2. The representation is simplified in a number of respects. For example, we do not show the
Shakespeare Macbeth
Macbeth's castle Will I with wine and wassail ...
◮ Figure 10.1 An XML document.
root element play element
number="I"
number="vii"
Will I with ...
Macbeth's castle
◮ Figure 10.2 The XML document in Figure 10.1 as a simplified DOM object.
//article [.//yr = 2001 or .//yr = 2002] //section [about(.,summer holidays)]
◮ Figure 10.3 An XML query in NEXI format and its partial representation as a tree.
XPATH XML CONTEXT
XML DTD XML S CHEMA
can process an XML document by starting at the root element and then descending down the tree from parents to children. XPath is a standard for enumerating paths in an XML document collection. We will also refer to paths as XML contexts or simply contexts in this chapter. Only a small subset of XPath is needed for our purposes. The XPath expression node selects all nodes of that name. Successive elements of a path are separated by slashes, so act/scene selects all scene elements whose parent is an act element. Double slashes indicate that an arbitrary number of elements can intervene on a path: play//scene selects all scene elements occurring in a play element. In Figure 10.2 this set consists of a single scene element, which is accessible via the path play, act, scene from the top. An initial slash starts the path at the root element. /play/title selects the play's title in Figure 10.1, /play//title selects a set with two members (the play's title and the scene's title), and /scene/title selects no elements. For notational convenience, we allow the final element of a path to be a vocabulary term and separate it from the element path by the symbol #, even though this does not conform to the XPath standard. For example, title#"Macbeth" selects all titles containing the term Macbeth. We also need the concept of schema in this chapter. A schema puts constraints on the structure of allowable XML documents for a particular application. A schema for Shakespeare's plays may stipulate that scenes can only occur as children of acts and that only acts and scenes have the number attribute. Two standards for schemas for XML documents are XML DTD (document type definition) and XML Schema. Users can only write structured queries for an XML retrieval system if they have some minimal knowledge about the schema of the collection. root node and text is not embedded in text nodes. See http://www.w3.org/DOM/.
Will I . . .
M's castle
Gallic war
◮ Figure 10.4 Tree representation of XML documents and queries.
A common format for XML queries is NEXI (Narrowed Extended XPath I). We give an example in Figure 10.3. We display the query on four lines for typographical convenience, but it is intended to be read as one unit without line breaks. In particular, //section is embedded under //article. The query in Figure 10.3 specifies a search for sections about the summer holidays that are part of articles from 2001 or 2002. As in XPath double slashes indicate that an arbitrary number of elements can intervene on a path. The dot in a clause in square brackets refers to the element the clause modifies. The clause [.//yr = 2001 or .//yr = 2002] modifies //article. Thus, the dot refers to //article in this case. Similarly, the dot in [about(., summer holidays)] refers to the section that the clause modifies. The two yr conditions are relational attribute constraints. Only articles whose yr attribute is 2001 or 2002 (or that contain an element whose yr attribute is 2001 or 2002) are to be considered. The about clause is a ranking constraint: Sections that occur in the right type of article are to be ranked according to how relevant they are to the topic summer holidays. We usually handle relational attribute constraints by prefiltering or postfiltering: We simply exclude all elements from the result set that do not meet the relational attribute constraints. In this chapter, we will not address how to do this efficiently and instead focus on the core information retrieval problem in XML retrieval, namely how to rank documents according to the relevance criteria expressed in the about conditions of the NEXI query. If we discard relational attributes, we can represent documents as trees with only one type of node: element nodes. In other words, we remove all attribute nodes from the XML document, such as the number attribute in Figure 10.1. Figure 10.4 shows a subtree of the document in Figure 10.1 as an element-node tree (labeled d1 ).
10.2 Challenges in XML retrieval
We can represent queries as trees in the same way. This is a query-byexample approach to query language design because users pose queries by creating objects that satisfy the same formal description as documents. In Figure 10.4, q1 is a search for books whose titles score highly for the keywords Julius Caesar. q2 is a search for books whose author elements score highly for Julius Caesar and whose title elements score highly for Gallic war.3
STRUCTURED DOCUMENT RETRIEVAL PRINCIPLE
INDEXING UNIT
Challenges in XML retrieval In this section, we discuss a number of challenges that make structured retrieval more difficult than unstructured retrieval. Recall from page 195 the basic setting we assume in structured retrieval: the collection consists of structured documents and queries are either structured (as in Figure 10.3) or unstructured (e.g., summer holidays). The first challenge in structured retrieval is that users want us to return parts of documents (i.e., XML elements), not entire documents as IR systems usually do in unstructured retrieval. If we query Shakespeare's plays for Macbeth's castle, should we return the scene, the act or the entire play in Figure 10.2? In this case, the user is probably looking for the scene. On the other hand, an otherwise unspecified search for Macbeth should return the play of this name, not a subunit. One criterion for selecting the most appropriate part of a document is the structured document retrieval principle: Structured document retrieval principle. A system should always retrieve the most specific part of a document answering the query. This principle motivates a retrieval strategy that returns the smallest unit that contains the information sought, but does not go below this level. However, it can be hard to implement this principle algorithmically. Consider the query title#"Macbeth" applied to Figure 10.2. The title of the tragedy, Macbeth, and the title of Act I, Scene vii, Macbeth's castle, are both good hits because they contain the matching term Macbeth. But in this case, the title of the tragedy, the higher node, is preferred. Deciding which level of the tree is right for answering a query is difficult. Parallel to the issue of which parts of a document to return to the user is the issue of which parts of a document to index. In Section 2.1.2 (page 20), we discussed the need for a document unit or indexing unit in indexing and retrieval. In unstructured retrieval, it is usually clear what the right document 3. To represent the semantics of NEXI queries fully we would also need to designate one node in the tree as a "target node", for example, the section in the tree in Figure 10.3. Without the designation of a target node, the tree in Figure 10.3 is not a search for sections embedded in articles (as specified by NEXI), but a search for articles that contain sections.
◮ Figure 10.5 Partitioning an XML document into non-overlapping indexing units.
unit is: files on your desktop, email messages, web pages on the web etc. In structured retrieval, there are a number of different approaches to defining the indexing unit. One approach is to group nodes into non-overlapping pseudodocuments as shown in Figure 10.5. In the example, books, chapters and sections have been designated to be indexing units, but without overlap. For example, the leftmost dashed indexing unit contains only those parts of the tree dominated by book that are not already part of other indexing units. The disadvantage of this approach is that pseudodocuments may not make sense to the user because they are not coherent units. For instance, the leftmost indexing unit in Figure 10.5 merges three disparate elements, the class, author and title elements. We can also use one of the largest elements as the indexing unit, for example, the book element in a collection of books or the play element for Shakespeare's works. We can then postprocess search results to find for each book or play the subelement that is the best hit. For example, the query Macbeth's castle may return the play Macbeth, which we can then postprocess to identify act I, scene vii as the best-matching subelement. Unfortunately, this twostage retrieval process fails to return the best subelement for many queries because the relevance of a whole book is often not a good predictor of the relevance of small subelements within it. Instead of retrieving large units and identifying subelements (top down), we can also search all leaves, select the most relevant ones and then extend them to larger units in postprocessing (bottom up). For the query Macbeth's castle in Figure 10.1, we would retrieve the title Macbeth's castle in the first pass and then decide in a postprocessing step whether to return the title, the scene, the act or the play. This approach has a similar problem as the last one: The relevance of a leaf element is often not a good predictor of the relevance
NESTED ELEMENTS
of elements it is contained in. The least restrictive approach is to index all elements. This is also problematic. Many XML elements are not meaningful search results, e.g., typographical elements like definitely or an ISBN number which cannot be interpreted without context. Also, indexing all elements means that search results will be highly redundant. For the query Macbeth's castle and the document in Figure 10.1, we would return all of the play, act, scene and title elements on the path between the root node and Macbeth's castle. The leaf node would then occur four times in the result set, once directly and three times as part of other elements. We call elements that are contained within each other nested. Returning redundant nested elements in a list of returned hits is not very user-friendly. Because of the redundancy caused by nested elements it is common to restrict the set of elements that are eligible to be returned. Restriction strategies include: • discard all small elements • discard all element types that users do not look at (this requires a working XML retrieval system that logs this information) • discard all element types that assessors generally do not judge to be relevant (if relevance assessments are available) • only keep element types that a system designer or librarian has deemed to be useful search results In most of these approaches, result sets will still contain nested elements. Thus, we may want to remove some elements in a postprocessing step to reduce redundancy. Alternatively, we can collapse several nested elements in the results list and use highlighting of query terms to draw the user's attention to the relevant passages. If query terms are highlighted, then scanning a medium-sized element (e.g., a section) takes little more time than scanning a small subelement (e.g., a paragraph). Thus, if the section and the paragraph both occur in the results list, it is sufficient to show the section. An additional advantage of this approach is that the paragraph is presented together with its context (i.e., the embedding section). This context may be helpful in interpreting the paragraph (e.g., the source of the information reported) even if the paragraph on its own satisfies the query. If the user knows the schema of the collection and is able to specify the desired type of element, then the problem of redundancy is alleviated as few nested elements have the same type. But as we discussed in the introduction, users often don't know what the name of an element in the collection is (Is the Vatican a country or a city?) or they may not know how to compose structured queries at all.
◮ Figure 10.6 Schema heterogeneity: intervening nodes and mismatched names.
SCHEMA HETEROGENEITY
A challenge in XML retrieval related to nesting is that we may need to distinguish different contexts of a term when we compute term statistics for ranking, in particular inverse document frequency (idf) statistics as defined in Section 6.2.1 (page 117). For example, the term Gates under the node author is unrelated to an occurrence under a content node like section if used to refer to the plural of gate. It makes little sense to compute a single document frequency for Gates in this example. One solution is to compute idf for XML-context/term pairs, e.g., to compute different idf weights for author#"Gates" and section#"Gates". Unfortunately, this scheme will run into sparse data problems – that is, many XML-context pairs occur too rarely to reliably estimate df (see Section 13.2, page 260, for a discussion of sparseness). A compromise is only to consider the parent node x of the term and not the rest of the path from the root to x to distinguish contexts. There are still conflations of contexts that are harmful in this scheme. For instance, we do not distinguish names of authors and names of corporations if both have the parent node name. But most important distinctions, like the example contrast author#"Gates" vs. section#"Gates", will be respected. In many cases, several different XML schemas occur in a collection since the XML documents in an IR application often come from more than one source. This phenomenon is called schema heterogeneity or schema diversity and presents yet another challenge. As illustrated in Figure 10.6 comparable elements may have different names: creator in d2 vs. author in d3 . In other cases, the structural organization of the schemas may be different: Author
EXTENDED QUERY
names are direct descendants of the node author in q3 , but there are the intervening nodes firstname and lastname in d3 . If we employ strict matching of trees, then q3 will retrieve neither d2 nor d3 although both documents are relevant. Some form of approximate matching of element names in combination with semi-automatic matching of different document structures can help here. Human editing of correspondences of elements in different schemas will usually do better than automatic methods. Schema heterogeneity is one reason for query-document mismatches like q3 /d2 and q3 /d3 . Another reason is that users often are not familiar with the element names and the structure of the schemas of collections they search as mentioned. This poses a challenge for interface design in XML retrieval. Ideally, the user interface should expose the tree structure of the collection and allow users to specify the elements they are querying. If we take this approach, then designing the query interface in structured retrieval is more complex than a search box for keyword queries in unstructured retrieval. We can also support the user by interpreting all parent-child relationships in queries as descendant relationships with any number of intervening nodes allowed. We call such queries extended queries. The tree in Figure 10.3 and q4 in Figure 10.6 are examples of extended queries. We show edges that are interpreted as descendant relationships as dashed arrows. In q4 , a dashed arrow connects book and Gates. As a pseudo-XPath notation for q4 , we adopt book//#"Gates": a book that somewhere in its structure contains the word Gates where the path from the book node to Gates can be arbitrarily long. The pseudo-XPath notation for the extended query that in addition specifies that Gates occurs in a section of the book is book//section//#"Gates". It is convenient for users to be able to issue such extended queries without having to specify the exact structural configuration in which a query term should occur – either because they do not care about the exact configuration or because they do not know enough about the schema of the collection to be able to specify it. In Figure 10.7, the user is looking for a chapter entitled FFT (q5 ). Suppose there is no such chapter in the collection, but that there are references to books on FFT (d4 ). A reference to a book on FFT is not exactly what the user is looking for, but it is better than returning nothing. Extended queries do not help here. The extended query q6 also returns nothing. This is a case where we may want to interpret the structural constraints specified in the query as hints as opposed to as strict conditions. As we will discuss in Section 10.4, users prefer a relaxed interpretation of structural constraints: Elements that do not meet structural constraints perfectly should be ranked lower, but they should not be omitted from search results.
◮ Figure 10.7 A structural mismatch between two queries and a document.
A vector space model for XML retrieval In this section, we present a simple vector space model for XML retrieval. It is not intended to be a complete description of a state-of-the-art system. Instead, we want to give the reader a flavor of how documents can be represented and retrieved in XML retrieval. To take account of structure in retrieval in Figure 10.4, we want a book entitled Julius Caesar to be a match for q1 and no match (or a lower weighted match) for q2 . In unstructured retrieval, there would be a single dimension of the vector space for Caesar. In XML retrieval, we must separate the title word Caesar from the author name Caesar. One way of doing this is to have each dimension of the vector space encode a word together with its position within the XML tree. Figure 10.8 illustrates this representation. We first take each text node (which in our setup is always a leaf) and break it into multiple nodes, one for each word. So the leaf node Bill Gates is split into two leaves Bill and Gates. Next we define the dimensions of the vector space to be lexicalized subtrees of documents – subtrees that contain at least one vocabulary term. A subset of these possible lexicalized subtrees is shown in the figure, but there are others – e.g., the subtree corresponding to the whole document with the leaf node Gates removed. We can now represent queries and documents as vectors in this space of lexicalized subtrees and compute matches between them. This means that we can use the vector space formalism from Chapter 6 for XML retrieval. The main difference is that the dimensions of vector space
10.3 A vector space model for XML retrieval
◮ Figure 10.8 A mapping of an XML document (left) to a set of lexicalized subtrees (right).
STRUCTURAL TERM
in unstructured retrieval are vocabulary terms whereas they are lexicalized subtrees in XML retrieval. There is a tradeoff between the dimensionality of the space and accuracy of query results. If we trivially restrict dimensions to vocabulary terms, then we have a standard vector space retrieval system that will retrieve many documents that do not match the structure of the query (e.g., Gates in the title as opposed to the author element). If we create a separate dimension for each lexicalized subtree occurring in the collection, the dimensionality of the space becomes too large. A compromise is to index all paths that end in a single vocabulary term, in other words, all XML-context/term pairs. We call such an XML-context/term pair a structural term and denote it by hc, ti: a pair of XML-context c and vocabulary term t. The document in Figure 10.8 has nine structural terms. Seven are shown (e.g., "Bill" and Author#"Bill") and two are not shown: /Book/Author#"Bill" and /Book/Author#"Gates". The tree with the leaves Bill and Gates is a lexicalized subtree that is not a structural term. We use the previously introduced pseudo-XPath notation for structural terms. As we discussed in the last section users are bad at remembering details about the schema and at constructing queries that comply with the schema. We will therefore interpret all queries as extended queries – that is, there can be an arbitrary number of intervening nodes in the document for any parentchild node pair in the query. For example, we interpret q5 in Figure 10.7 as q6 . But we still prefer documents that match the query structure closely by
CONTEXT RESEMBLANCE
inserting fewer additional nodes. We ensure that retrieval results respect this preference by computing a weight for each match. A simple measure of the similarity of a path cq in a query and a path cd in a document is the following context resemblance function C R: ( 1+|c q | if cq matches cd 1+|c d | C R(cq , cd ) = 0 if cq does not match cd where |cq | and |cd | are the number of nodes in the query path and document path, respectively, and cq matches cd iff we can transform cq into cd by inserting additional nodes. Two examples from Figure 10.6 are C R (cq4 , cd2 ) = 3/4 = 0.75 and C R (cq4 , cd3 ) = 3/5 = 0.6 where cq4 , cd2 and cd3 are the relevant paths from top to leaf node in q4 , d2 and d3 , respectively. The value of C R (cq , cd ) is 1.0 if q and d are identical. The final score for a document is computed as a variant of the cosine measure (Equation (6.10), page 121), which we call S IM N O M ERGE for reasons that will become clear shortly. S IM N O M ERGE is defined as follows:
S IM N O M ERGE (q, d) =
∑ ∑ ck ∈ B cl ∈ B
C R(ck , cl )
∑ weight(q, t, ck ) q
t ∈V
weight(d, t, cl ) ∑c∈ B,t∈V weight2 (d, t, c)
where V is the vocabulary of non-structural terms; B is the set of all XML contexts; and weight(q, t, c) and weight(d, t, c) are the weights of term t in XML context c in query q and document d, respectively. We compute the weights using one of the weightings from Chapter 6, such as idft · wft,d . The inverse document frequency idft depends on which elements we use to compute dft as discussed in Section 10.2. The similarity measure S IM N O M ERGE (q, d) is not a true cosine measure q since its value can be larger than 1.0 (Exercise 10.11). We divide by ∑c∈ B,t∈V weight2 (d, t, c) to normalize for document length (Section 6.3.1, page 121). We have omitted query length normalization to simplify the formula. It has no effect on ranking since, for q
a given query, the normalizer ∑c∈ B,t∈V weight2 (q, t, c) is the same for all documents. The algorithm for computing S IM N O M ERGE for all documents in the collection is shown in Figure 10.9. The array normalizer in Figure 10.9 contains q
2 ∑c∈ B,t∈V weight (d, t, c) from Equation (10.2) for each document. We give an example of how S IM N O M ERGE computes query-document similarities in Figure 10.10. hc1 , ti is one of the structural terms in the query. We successively retrieve all postings lists for structural terms hc′ , ti with the same vocabulary term t. Three example postings lists are shown. For the first one, we have C R (c1 , c1 ) = 1.0 since the two contexts are identical. The
S CORE D OCUMENTS W ITH S IM N O M ERGE (q, B, V, N, normalizer ) 1 for n ← 1 to N 2 do score[n] ← 0 3 for each hcq , ti ∈ q 4 do wq ← W EIGHT (q, t, cq ) 5 for each c ∈ B 6 do if C R (cq , c) > 0 7 then postings ← G ET P OSTINGS (hc, ti) 8 for each posting ∈ postings 9 do x ← C R (cq , c) ∗ wq ∗ weight( posting) 10 score[docID ( posting)] += x 11 for n ← 1 to N 12 do score[n] ← score[n]/normalizer [n] 13 return score ◮ Figure 10.9 The algorithm for scoring documents with S IM N O M ERGE .
query h c1 , t i inverted index C R ( c1 , c1 ) = 1.0 C R ( c1 , c2 ) = 0 C R ( c1, c3 ) = 0.63
h c1 , t i
hd1 , 0.5i
hd2 , 0.25i
hd12, 0.9i
◮ Figure 10.10
Scoring of a query with one structural term in S IM N O M ERGE .
next context has no context resemblance with c1 : C R (c1 , c2 ) = 0 and the corresponding postings list is ignored. The context match of c1 with c3 is 0.63>0 and it will be processed. In this example, the highest ranking document is d9 with a similarity of 1.0 × 0.2 + 0.63 × 0.6 = 0.578. To simplify the figure, the query weight of hc1 , ti is assumed to be 1.0. The query-document similarity function in Figure 10.9 is called S IM N O M ERGE because different XML contexts are kept separate for the purpose of weighting. An alternative similarity function is S IM M ERGE which relaxes the matching conditions of query and document further in the following three ways.
• We collect the statistics used for computing weight(q, t, c) and weight(d, t, c) from all contexts that have a non-zero resemblance to c (as opposed to just from c as in S IM N O M ERGE). For instance, for computing the document frequency of the structural term atl#"recognition", we also count occurrences of recognition in XML contexts fm/atl, article//atl etc. • We modify Equation (10.2) by merging all structural terms in the document that have a non-zero context resemblance to a given query structural term. For example, the contexts /play/act/scene/title and /play/title in the document will be merged when matching against the query term /play/title#"Macbeth". • The context resemblance function is further relaxed: Contexts have a nonzero resemblance in many cases where the definition of C R in Equation (10.1) returns 0. See the references in Section 10.6 for details. These three changes alleviate the problem of sparse term statistics discussed in Section 10.2 and increase the robustness of the matching function against poorly posed structural queries. The evaluation of S IM N O M ERGE and S IM M ERGE in the next section shows that the relaxed matching conditions of S IM M ERGE increase the effectiveness of XML retrieval.
Exercise 10.1 Consider computing df for a structural term as the number of times that the structural term occurs under a particular parent node. Assume the following: the structural term hc, ti = author#"Herbert" occurs once as the child of the node squib; there are 10 squib nodes in the collection; hc, ti occurs 1000 times as the child of article; there are 1,000,000 article nodes in the collection. The idf weight of hc, ti then is log2 10/1 ≈ 3.3 when occurring as the child of squib and log2 1,000,000/1000 ≈ 10.0 when occurring as the child of article. (i) Explain why this is not an appropriate weighting for hc, ti. Why should hc, ti not receive a weight that is three times higher in articles than in squibs? (ii) Suggest a better way of computing idf. Exercise 10.2 Write down all the structural terms occurring in the XML document in Figure 10.8. Exercise 10.3 How many structural terms does the document in Figure 10.1 yield?
Evaluation of XML retrieval
INEX
The premier venue for research on XML retrieval is the INEX (INitiative for the Evaluation of XML retrieval) program, a collaborative effort that has produced reference collections, sets of queries, and relevance judgments. A yearly INEX meeting is held to present and discuss research results. The
10.4 Evaluation of XML retrieval
12,107 494 MB 1995–2002 1,532 6.9 30 30
number of documents size time of publication of articles average number of XML nodes per document average depth of a node number of CAS topics number of CO topics
◮ Table 10.2 INEX 2002 collection statistics. article
IEEE Transaction on Pattern Analysis
Activity recognition
This work focuses on . . .
◮ Figure 10.11 Simplified schema of the documents in the INEX collection.
CO TOPICS CAS TOPICS
INEX 2002 collection consisted of about 12,000 articles from IEEE journals. We give collection statistics in Table 10.2 and show part of the schema of the collection in Figure 10.11. The IEEE journal collection was expanded in 2005. Since 2006 INEX uses the much larger English Wikipedia as a test collection. The relevance of documents is judged by human assessors using the methodology introduced in Section 8.1 (page 152), appropriately modified for structured documents as we will discuss shortly. Two types of information needs or topics in INEX are content-only or CO topics and content-and-structure (CAS) topics. CO topics are regular keyword queries as in unstructured information retrieval. CAS topics have structural constraints in addition to keywords. We already encountered an exam-
COMPONENT COVERAGE
ple of a CAS topic in Figure 10.3. The keywords in this case are summer and holidays and the structural constraints specify that the keywords occur in a section that in turn is part of an article and that this article has an embedded year attribute with value 2001 or 2002. Since CAS queries have both structural and content criteria, relevance assessments are more complicated than in unstructured retrieval. INEX 2002 defined component coverage and topical relevance as orthogonal dimensions of relevance. The component coverage dimension evaluates whether the element retrieved is "structurally" correct, i.e., neither too low nor too high in the tree. We distinguish four cases: • Exact coverage (E). The information sought is the main topic of the component and the component is a meaningful unit of information. • Too small (S). The information sought is the main topic of the component, but the component is not a meaningful (self-contained) unit of information. • Too large (L). The information sought is present in the component, but is not the main topic. • No coverage (N). The information sought is not a topic of the component.
TOPICAL RELEVANCE
The topical relevance dimension also has four levels: highly relevant (3), fairly relevant (2), marginally relevant (1) and nonrelevant (0). Components are judged on both dimensions and the judgments are then combined into a digit-letter code. 2S is a fairly relevant component that is too small and 3E is a highly relevant component that has exact coverage. In theory, there are 16 combinations of coverage and relevance, but many cannot occur. For example, a nonrelevant component cannot have exact coverage, so the combination 3N is not possible. The relevance-coverage combinations are quantized as follows:  1.00 if (rel, cov) = 3E      0.75 if (rel, cov) ∈ {2E, 3L} 0.50 if (rel, cov) ∈ {1E, 2L, 2S} Q(rel, cov) =   0.25 if (rel, cov) ∈ {1S, 1L}    0.00 if (rel, cov) = 0N
This evaluation scheme takes account of the fact that binary relevance judgments, which are standard in unstructured information retrieval (Section 8.5.1, page 166), are not appropriate for XML retrieval. A 2S component provides incomplete information and may be difficult to interpret without more context, but it does answer the query partially. The quantization function Q does not impose a binary choice relevant/nonrelevant and instead allows us to grade the component as partially relevant.
algorithm S IM N O M ERGE S IM M ERGE
average precision 0.242 0.271
◮ Table 10.3 INEX 2002 results of the vector space model in Section 10.3 for contentand-structure (CAS) queries and the quantization function Q.
The number of relevant components in a retrieved set A of components can then be computed as: #(relevant items retrieved) =
∑ Q(rel (c), cov(c)) c∈ A
As an approximation, the standard definitions of precision, recall and F from Chapter 8 can be applied to this modified definition of relevant items retrieved, with some subtleties because we sum graded as opposed to binary relevance assessments. See the references on focused retrieval in Section 10.6 for further discussion. One flaw of measuring relevance this way is that overlap is not accounted for. We discussed the concept of marginal relevance in the context of unstructured retrieval in Section 8.5.1 (page 166). This problem is worse in XML retrieval because of the problem of multiple nested elements occurring in a search result as we discussed on page 203. Much of the recent focus at INEX has been on developing algorithms and evaluation measures that return non-redundant results lists and evaluate them properly. See the references in Section 10.6. Table 10.3 shows two INEX 2002 runs of the vector space system we described in Section 10.3. The better run is the S IM M ERGE run, which incorporates few structural constraints and mostly relies on keyword matching. S IM M ERGE's median average precision (where the median is with respect to average precision numbers over topics) is only 0.147. Effectiveness in XML retrieval is often lower than in unstructured retrieval since XML retrieval is harder. Instead of just finding a document, we have to find the subpart of a document that is most relevant to the query. Also, XML retrieval effectiveness – when evaluated as described here – can be lower than unstructured retrieval effectiveness on a standard evaluation because graded judgments lower measured performance. Consider a system that returns a document with graded relevance 0.6 and binary relevance 1 at the top of the retrieved list. Then, interpolated precision at 0.00 recall (cf. page 158) is 1.0 on a binary evaluation, but can be as low as 0.6 on a graded evaluation. Table 10.3 gives us a sense of the typical performance of XML retrieval, but it does not compare structured with unstructured retrieval. Table 10.4 directly shows the effect of using structure in retrieval. The results are for a
precision at 5 precision at 10 precision at 20 precision at 30
content only 0.2000 0.1820 0.1700 0.1527
full structure 0.3265 0.2531 0.1796 0.1531
improvement 63.3% 39.1% 5.6% 0.3%
◮ Table 10.4 A comparison of content-only and full-structure search in INEX 2003/2004.
language-model-based system (cf. Chapter 12) that is evaluated on a subset of CAS topics from INEX 2003 and 2004. The evaluation metric is precision at k as defined in Chapter 8 (page 161). The discretization function used for the evaluation maps highly relevant elements (roughly corresponding to the 3E elements defined for Q) to 1 and all other elements to 0. The contentonly system treats queries and documents as unstructured bags of words. The full-structure model ranks elements that satisfy structural constraints higher than elements that do not. For instance, for the query in Figure 10.3 an element that contains the phrase summer holidays in a section will be rated higher than one that contains it in an abstract. The table shows that structure helps increase precision at the top of the results list. There is a large increase of precision at k = 5 and at k = 10. There is almost no improvement at k = 30. These results demonstrate the benefits of structured retrieval. Structured retrieval imposes additional constraints on what to return and documents that pass the structural filter are more likely to be relevant. Recall may suffer because some relevant documents will be filtered out, but for precision-oriented tasks structured retrieval is superior.
TEXT- CENTRIC XML
Text-centric vs. data-centric XML retrieval In the type of structured retrieval we cover in this chapter, XML structure serves as a framework within which we match the text of the query with the text of the XML documents. This exemplifies a system that is optimized for text-centric XML. While both text and structure are important, we give higher priority to text. We do this by adapting unstructured retrieval methods to handling additional structural constraints. The premise of our approach is that XML document retrieval is characterized by (i) long text fields (e.g., sections of a document), (ii) inexact matching, and (iii) relevance-ranked results. Relational databases do not deal well with this use case. In contrast, data-centric XML mainly encodes numerical and non-text attributevalue data. When querying data-centric XML, we want to impose exact match conditions in most cases. This puts the emphasis on the structural aspects of XML documents and queries. An example is:
10.5 Text-centric vs. data-centric XML retrieval
Find employees whose salary is the same this month as it was 12 months ago. This query requires no ranking. It is purely structural and an exact matching of the salaries in the two time periods is probably sufficient to meet the user's information need. Text-centric approaches are appropriate for data that are essentially text documents, marked up as XML to capture document structure. This is becoming a de facto standard for publishing text databases since most text documents have some form of interesting structure – paragraphs, sections, footnotes etc. Examples include assembly manuals, issues of journals, Shakespeare's collected works and newswire articles. Data-centric approaches are commonly used for data collections with complex structures that mainly contain non-text data. A text-centric retrieval engine will have a hard time with proteomic data in bioinformatics or with the representation of a city map that (together with street names and other textual descriptions) forms a navigational database. Two other types of queries that are difficult to handle in a text-centric structured retrieval model are joins and ordering constraints. The query for employees with unchanged salary requires a join. The following query imposes an ordering constraint: Retrieve the chapter of the book Introduction to algorithms that follows the chapter Binomial heaps. This query relies on the ordering of elements in XML – in this case the ordering of chapter elements underneath the book node. There are powerful query languages for XML that can handle numerical attributes, joins and ordering constraints. The best known of these is XQuery, a language proposed for standardization by the W3C. It is designed to be broadly applicable in all areas where XML is used. Due to its complexity, it is challenging to implement an XQuery-based ranked retrieval system with the performance characteristics that users have come to expect in information retrieval. This is currently one of the most active areas of research in XML retrieval. Relational databases are better equipped to handle many structural constraints, particularly joins (but ordering is also difficult in a database framework – the tuples of a relation in the relational calculus are not ordered). For this reason, most data-centric XML retrieval systems are extensions of relational databases (see the references in Section 10.6). If text fields are short, exact matching meets user needs and retrieval results in form of unordered sets are acceptable, then using a relational database for XML retrieval is appropriate.
XML FRAGMENT
References and further reading There are many good introductions to XML, including (Harold and Means 2004). Table 10.1 is inspired by a similar table in (van Rijsbergen 1979). Section 10.4 follows the overview of INEX 2002 by Gövert and Kazai (2003), published in the proceedings of the meeting (Fuhr et al. 2003a). The proceedings of the four following INEX meetings were published as Fuhr et al. (2003b), Fuhr et al. (2005), Fuhr et al. (2006), and Fuhr et al. (2007). An uptodate overview article is Fuhr and Lalmas (2007). The results in Table 10.4 are from (Kamps et al. 2006). Chu-Carroll et al. (2006) also present evidence that XML queries increase precision compared with unstructured queries. Instead of coverage and relevance, INEX now evaluates on the related but different dimensions of exhaustivity and specificity (Lalmas and Tombros 2007). Trotman et al. (2006) relate the tasks investigated at INEX to real world uses of structured retrieval such as structured book search on internet bookstore sites. The structured document retrieval principle is due to Chiaramella et al. (1996). Figure 10.5 is from (Fuhr and Großjohann 2004). Rahm and Bernstein (2001) give a survey of automatic schema matching that is applicable to XML. The vector-space based XML retrieval method in Section 10.3 is essentially IBM Haifa's JuruXML system as presented by Mass et al. (2003) and Carmel et al. (2003). Schlieder and Meuss (2002) and Grabs and Schek (2002) describe similar approaches. Carmel et al. (2003) represent queries as XML fragments. The trees that represent XML queries in this chapter are all XML fragments, but XML fragments also permit the operators +, − and phrase on content nodes. We chose to present the vector space model for XML retrieval because it is simple and a natural extension of the unstructured vector space model in Chapter 6. But many other unstructured retrieval methods have been applied to XML retrieval with at least as much success as the vector space model. These methods include language models (cf. Chapter 12, e.g., Kamps et al. (2004), List et al. (2005), Ogilvie and Callan (2005)), systems that use a relational database as a backend (Mihajlovi´c et al. 2005, Theobald et al. 2005; 2008), probabilistic weighting (Lu et al. 2007), and fusion (Larson 2005). There is currently no consensus as to what the best approach to XML retrieval is. Most early work on XML retrieval accomplished relevance ranking by focusing on individual terms, including their structural contexts, in query and document. As in unstructured information retrieval, there is a trend in more recent work to model relevance ranking as combining evidence from disparate measurements about the query, the document and their match. The combination function can be tuned manually (Arvola et al. 2005, Sigurbjörnsson et al. 2004) or trained using machine learning methods (Vittaut and Gal-
10.7 Exercises
FOCUSED RETRIEVAL
PASSAGE RETRIEVAL
linari (2006), cf. Section 15.4.1, page 341). An active area of XML retrieval research is focused retrieval (Trotman et al. 2007), which aims to avoid returning nested elements that share one or more common subelements (cf. discussion in Section 10.2, page 203). There is evidence that users dislike redundancy caused by nested elements (Betsi et al. 2006). Focused retrieval requires evaluation measures that penalize redundant results lists (Kazai and Lalmas 2006, Lalmas et al. 2007). Trotman and Geva (2006) argue that XML retrieval is a form of passage retrieval. In passage retrieval (Salton et al. 1993, Hearst and Plaunt 1993, Zobel et al. 1995, Hearst 1997, Kaszkiel and Zobel 1997), the retrieval system returns short passages instead of documents in response to a user query. While element boundaries in XML documents are cues for identifying good segment boundaries between passages, the most relevant passage often does not coincide with an XML element. In the last several years, the query format at INEX has been the NEXI standard proposed by Trotman and Sigurbjörnsson (2004). Figure 10.3 is from their paper. O'Keefe and Trotman (2004) give evidence that users cannot reliably distinguish the child and descendant axes. This justifies only permitting descendant axes in NEXI (and XML fragments). These structural constraints were only treated as "hints" in recent INEXes. Assessors can judge an element highly relevant even though it violates one of the structural constraints specified in a NEXI query. An alternative to structured query languages like NEXI is a more sophisticated user interface for query formulation (Tannier and Geva 2005, van Zwol et al. 2006, Woodley and Geva 2006). A broad overview of XML retrieval that covers database as well as IR approaches is given by Amer-Yahia and Lalmas (2006) and an extensive reference list on the topic can be found in (Amer-Yahia et al. 2005). Chapter 6 of Grossman and Frieder (2004) is a good introduction to structured text retrieval from a database perspective. The proposed standard for XQuery is available at http://www.w3.org/TR/xquery/ including an extension for full-text queries (Amer-Yahia et al. 2006): http://www.w3.org/TR/xquery-full-text/. Work that has looked at combining the relational database and the unstructured information retrieval approaches includes (Fuhr and Rölleke 1997), (Navarro and Baeza-Yates 1997), (Cohen 1998), and (Chaudhuri et al. 2006).
Exercise 10.4 Find a reasonably sized XML document collection (or a collection using a markup language different from XML like HTML) on the web and download it. Jon Bosak's XML edition of Shakespeare and of various religious works at http://www.ibiblio.org/bosak/ or the first 10,000 documents of the Wikipedia are good choices. Create three CAS topics
10 XML retrieval of the type shown in Figure 10.3 that you would expect to do better than analogous CO topics. Explain why an XML retrieval system would be able to exploit the XML structure of the documents to achieve better retrieval results on the topics than an unstructured retrieval system. Exercise 10.5 For the collection and the topics in Exercise 10.4, (i) are there pairs of elements e1 and e2 , with e2 a subelement of e1 such that both answer one of the topics? Find one case each where (ii) e1 (iii) e2 is the better answer to the query. Exercise 10.6 Implement the (i) S IM M ERGE (ii) S IM N O M ERGE algorithm in Section 10.3 and run it for the collection and the topics in Exercise 10.4. (iii) Evaluate the results by assigning binary relevance judgments to the first five documents of the three retrieved lists for each algorithm. Which algorithm performs better? Exercise 10.7 Are all of the elements in Exercise 10.4 appropriate to be returned as hits to a user or are there elements (as in the example definitely on page 203) that you would exclude? Exercise 10.8 We discussed the tradeoff between accuracy of results and dimensionality of the vector space on page 207. Give an example of an information need that we can answer correctly if we index all lexicalized subtrees, but cannot answer if we only index structural terms. Exercise 10.9 If we index all structural terms, what is the size of the index as a function of text size? Exercise 10.10 If we index all lexicalized subtrees, what is the size of the index as a function of text size? Exercise 10.11 Give an example of a query-document pair for which S IM N O M ERGE(q, d) is larger than 1.0.
Probabilistic information retrieval
During the discussion of relevance feedback in Section 9.1.2, we observed that if we have some known relevant and nonrelevant documents, then we can straightforwardly start to estimate the probability of a term t appearing in a relevant document P(t| R = 1), and that this could be the basis of a classifier that decides whether documents are relevant or not. In this chapter, we more systematically introduce this probabilistic approach to IR, which provides a different formal basis for a retrieval model and results in different techniques for setting term weights. Users start with information needs, which they translate into query representations. Similarly, there are documents, which are converted into document representations (the latter differing at least by how text is tokenized, but perhaps containing fundamentally less information, as when a non-positional index is used). Based on these two representations, a system tries to determine how well documents satisfy information needs. In the Boolean or vector space models of IR, matching is done in a formally defined but semantically imprecise calculus of index terms. Given only a query, an IR system has an uncertain understanding of the information need. Given the query and document representations, a system has an uncertain guess of whether a document has content relevant to the information need. Probability theory provides a principled foundation for such reasoning under uncertainty. This chapter provides one answer as to how to exploit this foundation to estimate how likely it is that a document is relevant to an information need. There is more than one possible retrieval model which has a probabilistic basis. Here, we will introduce probability theory and the Probability Ranking Principle (Sections 11.1–11.2), and then concentrate on the Binary Independence Model (Section 11.3), which is the original and still most influential probabilistic retrieval model. Finally, we will introduce related but extended methods which use term counts, including the empirically successful Okapi BM25 weighting scheme, and Bayesian Network models for IR (Section 11.4). In Chapter 12, we then present the alternative probabilistic language model-
11 Probabilistic information retrieval
ing approach to IR, which has been developed with considerable success in recent years.
RANDOM VARIABLE
CHAIN RULE
Review of basic probability theory We hope that the reader has seen a little basic probability theory previously. We will give a very quick review; some references for further reading appear at the end of the chapter. A variable A represents an event (a subset of the space of possible outcomes). Equivalently, we can represent the subset via a random variable, which is a function from outcomes to real numbers; the subset is the domain over which the random variable A has a particular value. Often we will not know with certainty whether an event is true in the world. We can ask the probability of the event 0 ≤ P( A) ≤ 1. For two events A and B, the joint event of both events occurring is described by the joint probability P( A, B). The conditional probability P( A| B) expresses the probability of event A given that event B occurred. The fundamental relationship between joint and conditional probabilities is given by the chain rule: P( A, B) = P( A ∩ B) = P( A| B) P( B) = P( B| A) P( A) Without making any assumptions, the probability of a joint event equals the probability of one of the events multiplied by the probability of the other event conditioned on knowing the first event happened. Writing P( A) for the complement of an event, we similarly have:
(11.2) PARTITION RULE
(11.3) B AYES ' R ULE
PRIOR PROBABILITY POSTERIOR PROBABILITY
P( A, B) = P( B| A) P( A) Probability theory also has a partition rule, which says that if an event B can be divided into an exhaustive set of disjoint subcases, then the probability of B is the sum of the probabilities of the subcases. A special case of this rule gives that: P( B) = P( A, B) + P( A, B) From these we can derive Bayes' Rule for inverting conditional probabilities: # " P( B| A) P( B| A) P( A) P( A) = P( A| B) = P( B) ∑ X ∈{ A,A} P( B| X ) P( X ) This equation can also be thought of as a way of updating probabilities. We start off with an initial estimate of how likely the event A is when we do not have any other information; this is the prior probability P( A). Bayes' rule lets us derive a posterior probability P( A| B) after having seen the evidence B,
11.2 The Probability Ranking Principle
based on the likelihood of B occurring in the two cases that A does or does not hold.1 Finally, it is often useful to talk about the odds of an event, which provide a kind of multiplier for how probabilities change: P( A) P( A) = 1 − P( A) P( A)
The Probability Ranking Principle
P ROBABILITY R ANKING P RINCIPLE
1/0 LOSS
B AYES O PTIMAL D ECISION R ULE
O( A) =
The 1/0 loss case We assume a ranked retrieval setup as in Section 6.3, where there is a collection of documents, the user issues a query, and an ordered list of documents is returned. We also assume a binary notion of relevance as in Chapter 8. For a query q and a document d in the collection, let Rd,q be an indicator random variable that says whether d is relevant with respect to a given query q. That is, it takes on a value of 1 when the document is relevant and 0 otherwise. In context we will often write just R for Rd,q . Using a probabilistic model, the obvious order in which to present documents to the user is to rank documents by their estimated probability of relevance with respect to the information need: P( R = 1|d, q). This is the basis of the Probability Ranking Principle (PRP) (van Rijsbergen 1979, 113–114): "If a reference retrieval system's response to each request is a ranking of the documents in the collection in order of decreasing probability of relevance to the user who submitted the request, where the probabilities are estimated as accurately as possible on the basis of whatever data have been made available to the system for this purpose, the overall effectiveness of the system to its user will be the best that is obtainable on the basis of those data." In the simplest case of the PRP, there are no retrieval costs or other utility concerns that would differentially weight actions or errors. You lose a point for either returning a nonrelevant document or failing to return a relevant document (such a binary situation where you are evaluated on your accuracy is called 1/0 loss). The goal is to return the best possible results as the top k documents, for any value of k the user chooses to examine. The PRP then says to simply rank all documents in decreasing order of P( R = 1|d, q). If a set of retrieval results is to be returned, rather than an ordering, the Bayes 1. The term likelihood is just a synonym of probability. It is the probability of an event or data according to a model. The term is usually used when people are thinking of holding the data fixed, while varying the model.
Optimal Decision Rule, the decision which minimizes the risk of loss, is to simply return documents that are more likely relevant than nonrelevant: (11.6) B AYES RISK
d is relevant iff P( R = 1|d, q) > P( R = 0|d, q) Theorem 11.1. The PRP is optimal, in the sense that it minimizes the expected loss (also known as the Bayes risk) under 1/0 loss. The proof can be found in Ripley (1996). However, it requires that all probabilities are known correctly. This is never the case in practice. Nevertheless, the PRP still provides a very useful foundation for developing models of IR.
The PRP with retrieval costs Suppose, instead, that we assume a model of retrieval costs. Let C1 be the cost of not retrieving a relevant document and C0 the cost of retrieval of a nonrelevant document. Then the Probability Ranking Principle says that if for a specific document d and for all documents d′ not yet retrieved
C0 · P( R = 0|d) − C1 · P( R = 1|d) ≤ C0 · P( R = 0|d′ ) − C1 · P( R = 1|d′ ) then d is the next document to be retrieved. Such a model gives a formal framework where we can model differential costs of false positives and false negatives and even system performance issues at the modeling stage, rather than simply at the evaluation stage, as we did in Section 8.6 (page 168). However, we will not further consider loss/utility models in this chapter.
11.3 B INARY I NDEPENDENCE M ODEL
The Binary Independence Model The Binary Independence Model (BIM) we present in this section is the model that has traditionally been used with the PRP. It introduces some simple assumptions, which make estimating the probability function P( R|d, q) practical. Here, "binary" is equivalent to Boolean: documents and queries are both represented as binary term incidence vectors. That is, a document d is represented by the vector ~x = ( x1 , . . . , x M ) where xt = 1 if term t is present in document d and xt = 0 if t is not present in d. With this representation, many possible documents have the same vector representation. Similarly, we represent q by the incidence vector ~q (the distinction between q and ~q is less central since commonly q is in the form of a set of words). "Independence" means that terms are modeled as occurring in documents independently. The model recognizes no association between terms. This assumption is far from correct, but it nevertheless often gives satisfactory results in practice; it is the "naive" assumption of Naive Bayes models, discussed further in Section 13.4 (page 265). Indeed, the Binary Independence
11.3 The Binary Independence Model
Model is exactly the same as the multivariate Bernoulli Naive Bayes model presented in Section 13.3 (page 263). In a sense this assumption is equivalent to an assumption of the vector space model, where each term is a dimension that is orthogonal to all other terms. We will first present a model which assumes that the user has a single step information need. As discussed in Chapter 9, seeing a range of results might let the user refine their information need. Fortunately, as mentioned there, it is straightforward to extend the Binary Independence Model so as to provide a framework for relevance feedback, and we present this model in Section 11.3.4. To make a probabilistic retrieval strategy precise, we need to estimate how terms in documents contribute to relevance, specifically, we wish to know how term frequency, document frequency, document length, and other statistics that we can compute influence judgments about document relevance, and how they can be reasonably combined to estimate the probability of document relevance. We then order documents by decreasing estimated probability of relevance. We assume here that the relevance of each document is independent of the relevance of other documents. As we noted in Section 8.5.1 (page 166), this is incorrect: the assumption is especially harmful in practice if it allows a system to return duplicate or near duplicate documents. Under the BIM, we model the probability P( R|d, q) that a document is relevant via the probability in terms of term incidence vectors P( R|~x, ~q). Then, using Bayes rule, we have: (11.8)
P( R = 1|~x, ~q)
P(~x| R = 1, ~q) P( R = 1|~q) P(~x|~q ) P(~x| R = 0, ~q) P( R = 0|~q) P(~x|~q )
Here, P(~x| R = 1, ~q) and P(~x| R = 0, ~q) are the probability that if a relevant or nonrelevant, respectively, document is retrieved, then that document's representation is ~x. You should think of this quantity as defined with respect to a space of possible documents in a domain. How do we compute all these probabilities? We never know the exact probabilities, and so we have to use estimates: Statistics about the actual document collection are used to estimate these probabilities. P( R = 1|~q) and P( R = 0|~q) indicate the prior probability of retrieving a relevant or nonrelevant document respectively for a query ~q. Again, if we knew the percentage of relevant documents in the collection, then we could use this number to estimate P( R = 1|~q ) and P( R = 0|~q). Since a document is either relevant or nonrelevant to a query, we must have that: (11.9)
P( R = 1|~x, ~q ) + P( R = 0|~x, ~q) = 1
Deriving a ranking function for query terms Given a query q, we wish to order returned documents by descending P( R = 1|d, q). Under the BIM, this is modeled as ordering by P( R = 1|~x, ~q). Rather than estimating this probability directly, because we are interested only in the ranking of documents, we work with some other quantities which are easier to compute and which give the same ordering of documents. In particular, we can rank documents by their odds of relevance (as the odds of relevance is monotonic with the probability of relevance). This makes things easier, because we can ignore the common denominator in (11.8), giving:
N AIVE B AYES ASSUMPTION
P( R = 1|~x, ~q) O( R|~x, ~q) = = P( R = 0|~x, ~q)
P ( R =1|~q) P (~x| R =1,~q) P (~x|~q) P ( R =0|~q) P (~x| R =0,~q) P (~x|~q)
P( R = 1|~q) P(~x| R = 1, ~q) · P( R = 0|~q) P(~x| R = 0, ~q)
The left term in the rightmost expression of Equation (11.10) is a constant for a given query. Since we are only ranking documents, there is thus no need for us to estimate it. The right-hand term does, however, require estimation, and this initially appears to be difficult: How can we accurately estimate the probability of an entire term incidence vector occurring? It is at this point that we make the Naive Bayes conditional independence assumption that the presence or absence of a word in a document is independent of the presence or absence of any other word (given the query): P(~x| R = 1, ~q) = P(~x| R = 0, ~q)
(11.11) So: (11.12)
P( xt | R = 1, ~q)
∏ P(xt | R = 0, ~q)
t =1
O( R|~x, ~q) = O( R|~q) · ∏
P( xt | R = 1, ~q) P( xt | R = 0, ~q)
Since each xt is either 0 or 1, we can separate the terms to give: (11.13)
O( R|~x, ~q) = O( R|~q) ·
∏ t:x t =1
P( xt = 1| R = 1, ~q) P( xt = 0| R = 1, ~q) · ∏ P( xt = 1| R = 0, ~q) t:xt =0 P( xt = 0| R = 0, ~q)
Henceforth, let pt = P( xt = 1| R = 1, ~q) be the probability of a term appearing in a document relevant to the query, and ut = P( xt = 1| R = 0, ~q) be the probability of a term appearing in a nonrelevant document. These quantities can be visualized in the following contingency table where the columns add to 1: (11.14) Term present Term absent
document xt = 1 xt = 0
relevant (R = 1) pt 1 − pt
nonrelevant (R = 0) ut 1 − ut
Let us make an additional simplifying assumption that terms not occurring in the query are equally likely to occur in relevant and nonrelevant documents: that is, if qt = 0 then pt = ut . (This assumption can be changed, as when doing relevance feedback in Section 11.3.4.) Then we need only consider terms in the products that appear in the query, and so, (11.15)
O( R|~q, ~x) = O( R|~q) ·
1 − pt pt · ∏ u 1 − ut t:x =q =1 t t:x =0,q =1
The left product is over query terms found in the document and the right product is over query terms not found in the document. We can manipulate this expression by including the query terms found in the document into the right product, but simultaneously dividing through by them in the left product, so the value is unchanged. Then we have: (11.16)
R ETRIEVAL S TATUS VALUE
O( R|~q, ~x ) = O( R|~q) ·
p t (1 − u t ) 1 − pt · ∏ u (1 − pt ) t:q =1 1 − ut t:x =q =1 t
The left product is still over query terms found in the document, but the right product is now over all query terms. That means that this right product is a constant for a particular query, just like the odds O( R|~q). So the only quantity that needs to be estimated to rank documents for relevance to a query is the left product. We can equally rank documents by the logarithm of this term, since log is a monotonic function. The resulting quantity used for ranking is called the Retrieval Status Value (RSV) in this model: RSVd = log
p t (1 − u t ) p t (1 − u t ) = ∑ log u (1 − p t ) u t (1 − p t ) t:x =q =1 t:x =q =1 t
So everything comes down to computing the RSV. Define ct : (11.18)
ct = log
pt 1 − ut p t (1 − u t ) = log + log u t (1 − p t ) (1 − p t ) ut
The ct terms are log odds ratios for the terms in the query. We have the odds of the term appearing if the document is relevant (pt /(1 − pt )) and the odds of the term appearing if the document is nonrelevant (ut /(1 − ut )). The odds ratio is the ratio of two such odds, and then we finally take the log of that quantity. The value will be 0 if a term has equal odds of appearing in relevant and nonrelevant documents, and positive if it is more likely to appear in relevant documents. The ct quantities function as term weights in the model, and the document score for a query is RSVd = ∑ xt =qt =1 ct . Operationally, we sum them in accumulators for query terms appearing in documents, just as for the vector space model calculations discussed in Section 7.1 (page 135). We now turn to how we estimate these ct quantities for a particular collection and query.
Probability estimates in theory For each term t, what would these ct numbers look like for the whole collection? (11.19) gives a contingency table of counts of documents in the collection, where dft is the number of documents that contain term t:
(11.19) Term present Term absent
documents xt = 1 xt = 0 Total
relevant s S−s S
nonrelevant dft − s ( N − dft ) − (S − s) N−S
Total dft N − dft N
Using this, pt = s/S and ut = (dft − s)/( N − S) and (11.20)
ct = K ( N, dft , S, s) = log
s/(S − s) (dft − s)/(( N − dft ) − (S − s))
To avoid the possibility of zeroes (such as if every or no relevant document has a particular term) it is fairly standard to add 12 to each of the quantities in the center 4 terms of (11.19), and then to adjust the marginal counts (the totals) accordingly (so, the bottom right cell totals N + 2). Then we have: (11.21)
RELATIVE FREQUENCY MAXIMUM LIKELIHOOD ESTIMATE
PSEUDOCOUNTS
B AYESIAN PRIOR
cˆt = K ( N, dft , S, s) = log
(s + 21 )/(S − s + 12 ) (dft − s + 21 )/( N − dft − S + s + 12 )
Adding 12 in this way is a simple form of smoothing. For trials with categorical outcomes (such as noting the presence or absence of a term), one way to estimate the probability of an event from data is simply to count the number of times an event occurred divided by the total number of trials. This is referred to as the relative frequency of the event. Estimating the probability as the relative frequency is the maximum likelihood estimate (or MLE), because this value makes the observed data maximally likely. However, if we simply use the MLE, then the probability given to events we happened to see is usually too high, whereas other events may be completely unseen and giving them as a probability estimate their relative frequency of 0 is both an underestimate, and normally breaks our models, since anything multiplied by 0 is 0. Simultaneously decreasing the estimated probability of seen events and increasing the probability of unseen events is referred to as smoothing. One simple way of smoothing is to add a number α to each of the observed counts. These pseudocounts correspond to the use of a uniform distribution over the vocabulary as a Bayesian prior, following Equation (11.4). We initially assume a uniform distribution over events, where the size of α denotes the strength of our belief in uniformity, and we then update the probability based on observed events. Since our belief in uniformity is weak, we use
MAXIMUM A POSTERIORI
α = 12 . This is a form of maximum a posteriori (MAP) estimation, where we choose the most likely point value for probabilities based on the prior and the observed evidence, following Equation (11.4). We will further discuss methods of smoothing estimated counts to give probability models in Section 12.2.2 (page 243); the simple method of adding 21 to each observed count will do for now.
Probability estimates in practice Under the assumption that relevant documents are a very small percentage of the collection, it is plausible to approximate statistics for nonrelevant documents by statistics from the whole collection. Under this assumption, ut (the probability of term occurrence in nonrelevant documents for a query) is dft /N and
log[(1 − ut )/ut ] = log[( N − dft )/dft ] ≈ log N/dft In other words, we can provide a theoretical justification for the most frequently used form of idf weighting, which we saw in Section 6.2.1. The approximation technique in Equation (11.22) cannot easily be extended to relevant documents. The quantity pt can be estimated in various ways: 1. We can use the frequency of term occurrence in known relevant documents (if we know some). This is the basis of probabilistic approaches to relevance feedback weighting in a feedback loop, discussed in the next subsection. 2. Croft and Harper (1979) proposed using a constant in their combination match model. For instance, we might assume that pt is constant over all terms x t in the query and that pt = 0.5. This means that each term has even odds of appearing in a relevant document, and so the pt and (1 − pt ) factors cancel out in the expression for RSV. Such an estimate is weak, but doesn't disagree violently with our hopes for the search terms appearing in many but not all relevant documents. Combining this method with our earlier approximation for ut , the document ranking is determined simply by which query terms occur in documents scaled by their idf weighting. For short documents (titles or abstracts) in situations in which iterative searching is undesirable, using this weighting term alone can be quite satisfactory, although in many other circumstances we would like to do better. 3. Greiff (1998) argues that the constant estimate of pt in the Croft and Harper (1979) model is theoretically problematic and not observed empirically: as might be expected, pt is shown to rise with dft . Based on his data analysis, a plausible proposal would be to use the estimate p t = 13 + 23 dft /N.
Iterative methods of estimation, which combine some of the above ideas, are discussed in the next subsection.
Probabilistic approaches to relevance feedback We can use (pseudo-)relevance feedback, perhaps in an iterative process of estimation, to get a more accurate estimate of pt . The probabilistic approach to relevance feedback works as follows: 1. Guess initial estimates of p t and ut . This can be done using the probability estimates of the previous section. For instance, we can assume that pt is constant over all xt in the query, in particular, perhaps taking p t = 21 . 2. Use the current estimates of p t and ut to determine a best guess at the set of relevant documents R = {d : Rd,q = 1}. Use this model to retrieve a set of candidate relevant documents, which we present to the user. 3. We interact with the user to refine the model of R. We do this by learning from the user relevance judgments for some subset of documents V. Based on relevance judgments, V is partitioned into two subsets: VR = {d ∈ V, Rd,q = 1} ⊂ R and VNR = {d ∈ V, Rd,q = 0}, which is disjoint from R.
4. We reestimate pt and ut on the basis of known relevant and nonrelevant documents. If the sets VR and VNR are large enough, we may be able to estimate these quantities directly from these documents as maximum likelihood estimates: pt = |VRt |/|VR| (where VRt is the set of documents in VR containing xt ). In practice, we usually need to smooth these estimates. We can do this by adding 1 2 to both the count |VRt | and to the number of relevant documents not containing the term, giving:
pt =
|VRt | + 12 |VR| + 1
However, the set of documents judged by the user (V) is usually very small, and so the resulting statistical estimate is quite unreliable (noisy), even if the estimate is smoothed. So it is often better to combine the new information with the original guess in a process of Bayesian updating. In this case we have: (k) |VRt | + κ pt ( k +1 ) pt = |VR| + κ
11.3 The Binary Independence Model (k)
Here pt is the kth estimate for pt in an iterative updating process and is used as a Bayesian prior in the next iteration with a weighting of κ. Relating this equation back to Equation (11.4) requires a bit more probability theory than we have presented here (we need to use a beta distribution prior, conjugate to the Bernoulli random variable Xt ). But the form of the resulting equation is quite straightforward: rather than uniformly distributing pseudocounts, we now distribute a total of κ pseudocounts according to the previous estimate, which acts as the prior distribution. In the absence of other evidence (and assuming that the user is perhaps indicating roughly 5 relevant or nonrelevant documents) then a value of around κ = 5 is perhaps appropriate. That is, the prior is strongly weighted so that the estimate does not change too much from the evidence provided by a very small number of documents. 5. Repeat the above process from step 2, generating a succession of approximations to R and hence p t , until the user is satisfied. It is also straightforward to derive a pseudo-relevance feedback version of this algorithm, where we simply pretend that VR = V. More briefly: 1. Assume initial estimates for pt and ut as above. 2. Determine a guess for the size of the relevant document set. If unsure, a conservative (too small) guess is likely to be best. This motivates use of a fixed size set V of highest ranked documents. 3. Improve our guesses for pt and ut . We choose from the methods of Equations (11.23) and (11.25) for re-estimating pt , except now based on the set V instead of VR. If we let Vt be the subset of documents in V containing xt and use add 21 smoothing, we get: (11.26)
|Vt | + 21 |V | + 1
and if we assume that documents that are not retrieved are nonrelevant then we can update our ut estimates as: (11.27)
ut =
dft − |Vt | + 21 N − |V | + 1
4. Go to step 2 until the ranking of the returned results converges. Once we have a real estimate for pt then the ct weights used in the RSV value look almost like a tf-idf value. For instance, using Equation (11.18),
Equation (11.22), and Equation (11.26), we have: " # |Vt | + 12 1 − ut N pt ≈ log · · ct = log 1 − pt ut |V | − |Vt | + 1 dft But things aren't quite the same: p t /(1 − p t ) measures the (estimated) proportion of relevant documents that the term t occurs in, not term frequency. Moreover, if we apply log identities:
|Vt | + 21 N + log |V | − |Vt | + 1 dft
we see that we are now adding the two log scaled components rather than multiplying them.
Exercise 11.1 Work through the derivation of Equation (11.20) from Equations (11.18) and (11.19). Exercise 11.2 What are the differences between standard vector space tf-idf weighting and the BIM probabilistic retrieval model (in the case where no document relevance information is available)? Exercise 11.3 [⋆⋆] Let Xt be a random variable indicating whether the term t appears in a document. Suppose we have | R| relevant documents in the document collection and that Xt = 1 in s of the documents. Take the observed data to be just these observations of Xt for each document in R. Show that the MLE for the parameter pt = P ( Xt = 1| R = 1, ~q), that is, the value for pt which maximizes the probability of the observed data, is pt = s/| R|. Exercise 11.4 Describe the differences between vector space relevance feedback and probabilistic relevance feedback.
An appraisal and some extensions
An appraisal of probabilistic models Probabilistic methods are one of the oldest formal models in IR. Already in the 1970s they were held out as an opportunity to place IR on a firmer theoretical footing, and with the resurgence of probabilistic methods in computational linguistics in the 1990s, that hope has returned, and probabilistic methods are again one of the currently hottest topics in IR. Traditionally, probabilistic IR has had neat ideas but the methods have never won on performance. Getting reasonable approximations of the needed probabilities for
11.4 An appraisal and some extensions
a probabilistic IR model is possible, but it requires some major assumptions. In the BIM these are: • a Boolean representation of documents/queries/relevance • term independence • terms not in the query don't affect the outcome • document relevance values are independent It is perhaps the severity of the modeling assumptions that makes achieving good performance difficult. A general problem seems to be that probabilistic models either require partial relevance information or else only allow for deriving apparently inferior term weighting models. Things started to change in the 1990s when the BM25 weighting scheme, which we discuss in the next section, showed very good performance, and started to be adopted as a term weighting scheme by many groups. The difference between "vector space" and "probabilistic" IR systems is not that great: in either case, you build an information retrieval scheme in the exact same way that we discussed in Chapter 7. For a probabilistic IR system, it's just that, at the end, you score queries not by cosine similarity and tf-idf in a vector space, but by a slightly different formula motivated by probability theory. Indeed, sometimes people have changed an existing vector-space IR system into an effectively probabilistic system simply by adopted term weighting formulas from probabilistic models. In this section, we briefly present three extensions of the traditional probabilistic model, and in the next chapter, we look at the somewhat different probabilistic language modeling approach to IR.
Tree-structured dependencies between terms Some of the assumptions of the BIM can be removed. For example, we can remove the assumption that terms are independent. This assumption is very far from true in practice. A case that particularly violates this assumption is term pairs like Hong and Kong, which are strongly dependent. But dependencies can occur in various complex configurations, such as between the set of terms New, York, England, City, Stock, Exchange, and University. van Rijsbergen (1979) proposed a simple, plausible model which allowed a tree structure of term dependencies, as in Figure 11.1. In this model each term can be directly dependent on only one other term, giving a tree structure of dependencies. When it was invented in the 1970s, estimation problems held back the practical success of this model, but the idea was reinvented as the Tree Augmented Naive Bayes model by Friedman and Goldszmidt (1996), who used it with some success on various machine learning data sets.
◮ Figure 11.1 A tree of dependencies between terms. In this graphical model representation, a term xi is directly dependent on a term xk if there is an arrow xk → xi .
BM25 WEIGHTS O KAPI WEIGHTING
Okapi BM25: a non-binary model The BIM was originally designed for short catalog records and abstracts of fairly consistent length, and it works reasonably in these contexts, but for modern full-text search collections, it seems clear that a model should pay attention to term frequency and document length, as in Chapter 6. The BM25 weighting scheme, often called Okapi weighting, after the system in which it was first implemented, was developed as a way of building a probabilistic model sensitive to these quantities while not introducing too many additional parameters into the model (Spärck Jones et al. 2000). We will not develop the full theory behind the model here, but just present a series of forms that build up to the standard form now used for document scoring. The simplest score for document d is just idf weighting of the query terms present, as in Equation (11.22): N RSVd = ∑ log dft t∈q Sometimes, an alternative version of idf is used. If we start with the formula in Equation (11.21) but in the absence of relevance feedback information we estimate that S = s = 0, then we get an alternative idf formulation as follows:
RSVd =
∑ log t∈q
N − dft + dft +
This variant behaves slightly strangely: if a term occurs in over half the documents in the collection then this model gives a negative term weight, which is presumably undesirable. But, assuming the use of a stop list, this normally doesn't happen, and the value for each summand can be given a floor of 0. We can improve on Equation (11.30) by factoring in the frequency of each term and document length: (k1 + 1)tftd N · RSVd = ∑ log df k (( 1 − b ) + b × ( Ld /Lave )) + tftd t 1 t∈q Here, tftd is the frequency of term t in document d, and Ld and Lave are the length of document d and the average document length for the whole collection. The variable k1 is a positive tuning parameter that calibrates the document term frequency scaling. A k1 value of 0 corresponds to a binary model (no term frequency), and a large value corresponds to using raw term frequency. b is another tuning parameter (0 ≤ b ≤ 1) which determines the scaling by document length: b = 1 corresponds to fully scaling the term weight by the document length, while b = 0 corresponds to no length normalization. If the query is long, then we might also use similar weighting for query terms. This is appropriate if the queries are paragraph long information needs, but unnecessary for short queries. (k3 + 1)tftq N (k1 + 1)tftd RSVd = ∑ log · · df k (( 1 − b ) + b × ( L /L )) + tf k3 + tftq t ave 1 d td t∈q with tftq being the frequency of term t in the query q, and k3 being another positive tuning parameter that this time calibrates term frequency scaling of the query. In the equation presented, there is no length normalization of queries (it is as if b = 0 here). Length normalization of the query is unnecessary because retrieval is being done with respect to a single fixed query. The tuning parameters of these formulas should ideally be set to optimize performance on a development test collection (see page 153). That is, we can search for values of these parameters that maximize performance on a separate development test collection (either manually or with optimization methods such as grid search or something more advanced), and then use these parameters on the actual test collection. In the absence of such optimization, experiments have shown reasonable values are to set k1 and k3 to a value between 1.2 and 2 and b = 0.75. If we have relevance judgments available, then we can use the full form of (11.21) in place of the approximation log( N/dft ) introduced in (11.22): # "" (|VRt | + 21 )/(|VNRt | + 12 ) RSVd = ∑ log (dft − |VRt | + 12 )/( N − dft − |VR| + |VRt | + 12 ) t∈q
(k3 + 1)tftq (k1 + 1)tftd × × k1 ((1 − b) + b( Ld /Lave )) + tftd k3 + tftq
Here, VRt , NVRt , and VR are used as in Section 11.3.4. The first part of the expression reflects relevance feedback (or just idf weighting if no relevance information is available), the second implements document term frequency and document length scaling, and the third considers term frequency in the query. Rather than just providing a term weighting method for terms in a user's query, relevance feedback can also involve augmenting the query (automatically or with manual review) with some (say, 10–20) of the top terms in the known-relevant documents as ordered by the relevance factor cˆt from Equation (11.21), and the above formula can then be used with such an augmented query vector ~q. The BM25 term weighting formulas have been used quite widely and quite successfully across a range of collections and search tasks. Especially in the TREC evaluations, they performed well and were widely adopted by many groups. See Spärck Jones et al. (2000) for extensive motivation and discussion of experimental results.
11.4.4 B AYESIAN NETWORKS
Bayesian network approaches to IR Turtle and Croft (1989; 1991) introduced into information retrieval the use of Bayesian networks (Jensen and Jensen 2001), a form of probabilistic graphical model. We skip the details because fully introducing the formalism of Bayesian networks would require much too much space, but conceptually, Bayesian networks use directed graphs to show probabilistic dependencies between variables, as in Figure 11.1, and have led to the development of sophisticated algorithms for propagating influence so as to allow learning and inference with arbitrary knowledge within arbitrary directed acyclic graphs. Turtle and Croft used a sophisticated network to better model the complex dependencies between a document and a user's information need. The model decomposes into two parts: a document collection network and a query network. The document collection network is large, but can be precomputed: it maps from documents to terms to concepts. The concepts are a thesaurus-based expansion of the terms appearing in the document. The query network is relatively small but a new network needs to be built each time a query comes in, and then attached to the document network. The query network maps from query terms, to query subexpressions (built using probabilistic or "noisy" versions of AND and OR operators), to the user's information need. The result is a flexible probabilistic network which can generalize various simpler Boolean and probabilistic models. Indeed, this is the primary
11.5 References and further reading
case of a statistical ranked retrieval model that naturally supports structured query operators. The system allowed efficient large-scale retrieval, and was the basis of the InQuery text retrieval system, built at the University of Massachusetts. This system performed very well in TREC evaluations and for a time was sold commercially. On the other hand, the model still used various approximations and independence assumptions to make parameter estimation and computation possible. There has not been much follow-on work along these lines, but we would note that this model was actually built very early on in the modern era of using Bayesian networks, and there have been many subsequent developments in the theory, and the time is perhaps right for a new generation of Bayesian network-based information retrieval systems.
References and further reading Longer introductions to probability theory can be found in most introductory probability and statistics books, such as (Grinstead and Snell 1997, Rice 2006, Ross 2006). An introduction to Bayesian utility theory can be found in (Ripley 1996). The probabilistic approach to IR originated in the UK in the 1950s. The first major presentation of a probabilistic model is Maron and Kuhns (1960). Robertson and Jones (1976) introduce the main foundations of the BIM and van Rijsbergen (1979) presents in detail the classic BIM probabilistic model. The idea of the PRP is variously attributed to S. E. Robertson, M. E. Maron and W. S. Cooper (the term "Probabilistic Ordering Principle" is used in Robertson and Jones (1976), but PRP dominates in later work). Fuhr (1992) is a more recent presentation of probabilistic IR, which includes coverage of other approaches such as probabilistic logics and Bayesian networks. Crestani et al. (1998) is another survey.Spärck Jones et al. (2000) is the definitive presentation of probabilistic IR experiments by the "London school", and Robertson (2005) presents a retrospective on the group's participation in TREC evaluations, including detailed discussion of the Okapi BM25 scoring function and its development. Robertson et al. (2004) extend BM25 to the case of multiple weighted fields. The open-source Indri search engine, which is distributed with the Lemur toolkit (http://www.lemurproject.org/) merges ideas from Bayesian inference networks and statistical language modeling approaches (see Chapter 12), in particular preserving the former's support for structured query operators.
Language models for information retrieval
A common suggestion to users for coming up with good queries is to think of words that would likely appear in a relevant document, and to use those words as the query. The language modeling approach to IR directly models that idea: a document is a good match to a query if the document model is likely to generate the query, which will in turn happen if the document contains the query words often. This approach thus provides a different realization of some of the basic ideas for document ranking which we saw in Section 6.2 (page 117). Instead of overtly modeling the probability P( R = 1|q, d) of relevance of a document d to a query q, as in the traditional probabilistic approach to IR (Chapter 11), the basic language modeling approach instead builds a probabilistic language model Md from each document d, and ranks documents based on the probability of the model generating the query: P ( q | Md ) . In this chapter, we first introduce the concept of language models (Section 12.1) and then describe the basic and most commonly used language modeling approach to IR, the Query Likelihood Model (Section 12.2). After some comparisons between the language modeling approach and other approaches to IR (Section 12.3), we finish by briefly describing various extensions to the language modeling approach (Section 12.4).
12.1 12.1.1 GENERATIVE MODEL
Language models Finite automata and language models What do we mean by a document model generating a query? A traditional generative model of a language, of the kind familiar from formal language theory, can be used either to recognize or to generate strings. For example, the finite automaton shown in Figure 12.1 can generate strings that include the examples shown. The full set of strings that can be generated is called the language of the automaton.1
12 Language models for information retrieval
I wish I wish I wish I wish I wish I wish I wish I wish I wish I wish I wish I wish ... C ANNOT
GENERATE :
wish I wish
◮ Figure 12.1 A simple finite automaton and some of the strings in the language it generates. → shows the start state of the automaton and a double circle indicates a (possible) finishing state.
q1 P( STOP |q1 ) = 0.2
the a frog toad said likes that ...
0.2 0.1 0.01 0.01 0.03 0.02 0.04 ...
◮ Figure 12.2 A one-state finite automaton that acts as a unigram language model. We show a partial specification of the state emission probabilities.
LANGUAGE MODEL
If instead each node has a probability distribution over generating different terms, we have a language model. The notion of a language model is inherently probabilistic. A language model is a function that puts a probability measure over strings drawn from some vocabulary. That is, for a language model M over an alphabet Σ:
P ( s) = 1
s∈Σ∗
One simple kind of language model is equivalent to a probabilistic finite automaton consisting of just a single node with a single probability distribution over producing different terms, so that ∑t∈V P(t) = 1, as shown in Figure 12.2. After generating each word, we decide whether to stop or to loop around and then produce another word, and so the model also requires a probability of stopping in the finishing state. Such a model places a probability distribution over any sequence of words. By construction, it also provides a model for generating text according to its distribution. 1. Finite automata can have outputs attached to either their states or their arcs; we use states here, because that maps directly on to the way probabilistic automata are usually formalized.
12.1 Language models
Model M1 the 0.2 a 0.1 frog 0.01 toad 0.01 said 0.03 likes 0.02 that 0.04 dog 0.005 cat 0.003 monkey 0.001 ... ...
Model M2 the 0.15 a 0.12 frog 0.0002 toad 0.0001 said 0.03 likes 0.04 that 0.04 dog 0.01 cat 0.015 monkey 0.002 ... ...
◮ Figure 12.3 Partial specification of two unigram language models.
✎ (12.2)
To find the probability of a word sequence, we just multiply the probabilities which the model gives to each word in the sequence, together with the probability of continuing or stopping after producing each word. For example,
Example 12.1:
P (frog said that toad likes frog)
(0.01 × 0.03 × 0.04 × 0.01 × 0.02 × 0.01) ×(0.8 × 0.8 × 0.8 × 0.8 × 0.8 × 0.8 × 0.2) 0.000000000001573
As you can see, the probability of a particular string/document, is usually a very small number! Here we stopped after generating frog the second time. The first line of numbers are the term emission probabilities, and the second line gives the probability of continuing or stopping after generating each word. An explicit stop probability is needed for a finite automaton to be a well-formed language model according to Equation (12.1). Nevertheless, most of the time, we will omit to include STOP and (1 − STOP) probabilities (as do most other authors). To compare two models for a data set, we can calculate their likelihood ratio, which results from simply dividing the probability of the data according to one model by the probability of the data according to the other model. Providing that the stop probability is fixed, its inclusion will not alter the likelihood ratio that results from comparing the likelihood of two language models generating a string. Hence, it will not alter the ranking of documents.2 Nevertheless, formally, the numbers will no longer truly be probabilities, but only proportional to probabilities. See Exercise 12.4.
Example 12.2: Suppose, now, that we have two language models M1 and M2 , shown partially in Figure 12.3. Each gives a probability estimate to a sequence of 2. In the IR context that we are leading up to, taking the stop probability to be fixed across models seems reasonable. This is because we are generating queries, and the length distribution of queries is fixed and independent of the document from which we are generating the language model.
12 Language models for information retrieval terms, as already illustrated in Example 12.1. The language model that gives the higher probability to the sequence of terms is more likely to have generated the term sequence. This time, we will omit STOP probabilities from our calculations. For the sequence shown, we get: (12.3)
s M1 M2
frog 0.01 0.0002
said 0.03 0.03
that 0.04 0.04
toad 0.01 0.0001
likes 0.02 0.04
dog 0.005 0.01
P (s| M1 ) = 0.00000000000048 P (s| M2 ) = 0.000000000000000384
and we see that P (s| M1 ) > P (s| M2 ). We present the formulas here in terms of products of probabilities, but, as is common in probabilistic applications, in practice it is usually best to work with sums of log probabilities (cf. page 258).
Types of language models How do we build probabilities over sequences of terms? We can always use the chain rule from Equation (11.1) to decompose the probability of a sequence of events into the probability of each successive event conditioned on earlier events:
UNIGRAM LANGUAGE MODEL
(12.5) BIGRAM LANGUAGE MODEL
P ( t1 t2 t3 t4 ) = P ( t1 ) P ( t2 | t1 ) P ( t3 | t1 t2 ) P ( t4 | t1 t2 t3 ) The simplest form of language model simply throws away all conditioning context, and estimates each term independently. Such a model is called a unigram language model: Puni (t1 t2 t3 t4 ) = P(t1 ) P(t2 ) P(t3 ) P(t4 ) There are many more complex kinds of language models, such as bigram language models, which condition on the previous term, Pbi (t1 t2 t3 t4 ) = P(t1 ) P(t2 |t1 ) P(t3 |t2 ) P(t4 |t3 ) and even more complex grammar-based language models such as probabilistic context-free grammars. Such models are vital for tasks like speech recognition, spelling correction, and machine translation, where you need the probability of a term conditioned on surrounding context. However, most language-modeling work in IR has used unigram language models. IR is not the place where you most immediately need complex language models, since IR does not directly depend on the structure of sentences to the extent that other tasks like speech recognition do. Unigram models are often sufficient to judge the topic of a text. Moreover, as we shall see, IR language models are frequently estimated from a single document and so it is
questionable whether there is enough training data to do more. Losses from data sparseness (see the discussion on page 260) tend to outweigh any gains from richer models. This is an example of the bias-variance tradeoff (cf. Section 14.6, page 308): With limited training data, a more constrained model tends to perform better. In addition, unigram models are more efficient to estimate and apply than higher-order models. Nevertheless, the importance of phrase and proximity queries in IR in general suggests that future work should make use of more sophisticated language models, and some has begun to (see Section 12.5, page 252). Indeed, making this move parallels the model of van Rijsbergen in Chapter 11 (page 231).
MULTINOMIAL DISTRIBUTION
Multinomial distributions over words Under the unigram language model the order of words is irrelevant, and so such models are often called "bag of words" models, as discussed in Chapter 6 (page 117). Even though there is no conditioning on preceding context, this model nevertheless still gives the probability of a particular ordering of terms. However, any other ordering of this bag of terms will have the same probability. So, really, we have a multinomial distribution over words. So long as we stick to unigram models, the language model name and motivation could be viewed as historical rather than necessary. We could instead just refer to the model as a multinomial model. From this perspective, the equations presented above do not present the multinomial probability of a bag of words, since they do not sum over all possible orderings of those words, as is done by the multinomial coefficient (the first term on the right-hand side) in the standard presentation of a multinomial model: P(d) =
Ld ! tf P(t1 )tft1 ,d P(t2 )tft2 ,d · · · P(t M ) t M ,d tft1 ,d !tft2 ,d ! · · · tft M ,d !
Here, Ld = ∑1≤i≤ M tfti ,d is the length of document d, M is the size of the term vocabulary, and the products are now over the terms in the vocabulary, not the positions in the document. However, just as with STOP probabilities, in practice we can also leave out the multinomial coefficient in our calculations, since, for a particular bag of words, it will be a constant, and so it has no effect on the likelihood ratio of two different models generating a particular bag of words. Multinomial distributions also appear in Section 13.2 (page 258). The fundamental problem in designing language models is that we do not know what exactly we should use as the model Md . However, we do generally have a sample of text that is representative of that model. This problem makes a lot of sense in the original, primary uses of language models. For example, in speech recognition, we have a training sample of (spoken) text. But we have to expect that, in the future, users will use different words and in
different sequences, which we have never observed before, and so the model has to generalize beyond the observed data to allow unknown words and sequences. This interpretation is not so clear in the IR case, where a document is finite and usually fixed. The strategy we adopt in IR is as follows. We pretend that the document d is only a representative sample of text drawn from a model distribution, treating it like a fine-grained topic. We then estimate a language model from this sample, and use that model to calculate the probability of observing any word sequence, and, finally, we rank documents according to their probability of generating the query.
Exercise 12.1 [⋆] Including stop probabilities in the calculation, what will the sum of the probability estimates of all strings in the language of length 1 be? Assume that you generate a word and then decide whether to stop or not (i.e., the null string is not part of the language). Exercise 12.2 [⋆] If the stop probability is omitted from calculations, what will the sum of the scores assigned to strings in the language of length 1 be? Exercise 12.3 [⋆] What is the likelihood ratio of the document according to M1 and M2 in Example 12.2? Exercise 12.4 [⋆] No explicit STOP probability appeared in Example 12.2. Assuming that the STOP probability of each model is 0.1, does this change the likelihood ratio of a document according to the two models? Exercise 12.5 [⋆⋆] How might a language model be used in a spelling correction system? In particular, consider the case of context-sensitive spelling correction, and correcting incorrect usages of words, such as their in Are you their? (See Section 3.5 (page 65) for pointers to some literature on this topic.)
12.2 12.2.1
QUERY LIKELIHOOD MODEL
The query likelihood model Using query likelihood language models in IR Language modeling is a quite general formal approach to IR, with many variant realizations. The original and basic method for using language models in IR is the query likelihood model. In it, we construct from each document d in the collection a language model Md . Our goal is to rank documents by P(d|q), where the probability of a document is interpreted as the likelihood that it is relevant to the query. Using Bayes rule (as introduced in Section 11.1, page 220), we have: P(d|q) = P(q|d) P(d)/P(q)
12.2 The query likelihood model
P(q) is the same for all documents, and so can be ignored. The prior probability of a document P(d) is often treated as uniform across all d and so it can also be ignored, but we could implement a genuine prior which could include criteria like authority, length, genre, newness, and number of previous people who have read the document. But, given these simplifications, we return results ranked by simply P(q|d), the probability of the query q under the language model derived from d. The Language Modeling approach thus attempts to model the query generation process: Documents are ranked by the probability that a query would be observed as a random sample from the respective document model. The most common way to do this is using the multinomial unigram language model, which is equivalent to a multinomial Naive Bayes model (page 263), where the documents are the classes, each treated in the estimation as a separate "language". Under this model, we have that: (12.8)
P ( q | Md ) = K q
∏ P(t| Md )tft,d
where, again Kq = Ld !/(tft1 ,d !tft2 ,d ! · · · tft M ,d !) is the multinomial coefficient for the query q, which we will henceforth ignore, since it is a constant for a particular query. For retrieval based on a language model (henceforth LM), we treat the generation of queries as a random process. The approach is to 1. Infer a LM for each document. 2. Estimate P(q| Mdi ), the probability of generating the query according to each of these document models. 3. Rank the documents according to these probabilities. The intuition of the basic model is that the user has a prototype document in mind, and generates a query based on words that appear in this document. Often, users have a reasonable idea of terms that are likely to occur in documents of interest and they will choose query terms that distinguish these documents from others in the collection.3 Collection statistics are an integral part of the language model, rather than being used heuristically as in many other approaches.
Estimating the query generation probability In this section we describe how to estimate P(q| Md ). The probability of producing the query given the LM Md of document d using maximum likelihood 3. Of course, in other cases, they do not. The answer to this within the language modeling approach is translation language models, as briefly discussed in Section 12.4.
estimation (MLE) and the unigram assumption is: (12.9)
Pˆ (q| Md ) = ∏ Pˆ mle (t| Md ) = t∈q
∏ t∈q
tft,d Ld
where Md is the language model of document d, tft,d is the (raw) term frequency of term t in document d, and Ld is the number of tokens in document d. That is, we just count up how often each word occurred, and divide through by the total number of words in the document d. This is the same method of calculating an MLE as we saw in Section 11.3.2 (page 226), but now using a multinomial over word counts. The classic problem with using language models is one of estimation (the ˆ symbol on the P's is used above to stress that the model is estimated): terms appear very sparsely in documents. In particular, some words will not have appeared in the document at all, but are possible words for the information need, which the user may have used in the query. If we estimate Pˆ (t| Md ) = 0 for a term missing from a document d, then we get a strict conjunctive semantics: documents will only give a query non-zero probability if all of the query terms appear in the document. Zero probabilities are clearly a problem in other uses of language models, such as when predicting the next word in a speech recognition application, because many words will be sparsely represented in the training data. It may seem rather less clear whether this is problematic in an IR application. This could be thought of as a human-computer interface issue: vector space systems have generally preferred more lenient matching, though recent web search developments have tended more in the direction of doing searches with such conjunctive semantics. Regardless of the approach here, there is a more general problem of estimation: occurring words are also badly estimated; in particular, the probability of words occurring once in the document is normally overestimated, since their one occurrence was partly by chance. The answer to this (as we saw in Section 11.3.2, page 226) is smoothing. But as people have come to understand the LM approach better, it has become apparent that the role of smoothing in this model is not only to avoid zero probabilities. The smoothing of terms actually implements major parts of the term weighting component (Exercise 12.8). It is not just that an unsmoothed model has conjunctive semantics; an unsmoothed model works badly because it lacks parts of the term weighting component. Thus, we need to smooth probabilities in our document language models: to discount non-zero probabilities and to give some probability mass to unseen words. There's a wide space of approaches to smoothing probability distributions to deal with this problem. In Section 11.3.2 (page 226), we already discussed adding a number (1, 1/2, or a small α) to the observed
counts and renormalizing to give a probability distribution.4 In this section we will mention a couple of other smoothing methods, which involve combining observed counts with a more general reference probability distribution. The general approach is that a non-occurring term should be possible in a query, but its probability should be somewhat close to but no more likely than would be expected by chance from the whole collection. That is, if tft,d = 0 then Pˆ (t| Md ) ≤ cft /T
where cft is the raw count of the term in the collection, and T is the raw size (number of tokens) of the entire collection. A simple idea that works well in practice is to use a mixture between a document-specific multinomial distribution and a multinomial distribution estimated from the entire collection: (12.10)
B AYESIAN SMOOTHING
Pˆ (t|d) = λ Pˆ mle (t| Md ) + (1 − λ) Pˆ mle (t| Mc ) where 0 < λ < 1 and Mc is a language model built from the entire document collection. This mixes the probability from the document with the general collection frequency of the word. Such a model is referred to as a linear interpolation language model.5 Correctly setting λ is important to the good performance of this model. An alternative is to use a language model built from the whole collection as a prior distribution in a Bayesian updating process (rather than a uniform distribution, as we saw in Section 11.3.2). We then get the following equation: tft,d + α Pˆ (t| Mc ) Pˆ (t|d) = Ld + α Both of these smoothing methods have been shown to perform well in IR experiments; we will stick with the linear interpolation smoothing method for the rest of this section. While different in detail, they are both conceptually similar: in both cases the probability estimate for a word present in the document combines a discounted MLE and a fraction of the estimate of its prevalence in the whole collection, while for words not present in a document, the estimate is just a fraction of the estimate of the prevalence of the word in the whole collection. The role of smoothing in LMs for IR is not simply or principally to avoid estimation problems. This was not clear when the models were first proposed, but it is now understood that smoothing is essential to the good properties 4. In the context of probability theory, (re)normalization refers to summing numbers that cover an event space and dividing them through by their sum, so that the result is a probability distribution which sums to 1. This is distinct from both the concept of term normalization in Chapter 2 and the concept of length normalization in Chapter 6, which is done with a L2 norm. 5. It is also referred to as Jelinek-Mercer smoothing.
of the models. The reason for this is explored in Exercise 12.8. The extent of smoothing in these two models is controlled by the λ and α parameters: a small value of λ or a large value of α means more smoothing. This parameter can be tuned to optimize performance using a line search (or, for the linear interpolation model, by other methods, such as the expectation maximimization algorithm; see Section 16.5, page 368). The value need not be a constant. One approach is to make the value a function of the query size. This is useful because a small amount of smoothing (a "conjunctive-like" search) is more suitable for short queries, while a lot of smoothing is more suitable for long queries. To summarize, the retrieval ranking for a query q under the basic LM for IR we have been considering is given by: (12.12)
P(d|q) ∝ P(d) ∏((1 − λ) P(t| Mc ) + λP(t| Md )) t∈q
This equation captures the probability that the document that the user had in mind was in fact d.
Suppose the document collection contains two documents:
• d1 : Xyzzy reports a profit but revenue is down • d2 : Quorus narrows quarter loss but revenue decreases further The model will be MLE unigram models from the documents and collection, mixed with λ = 1/2. Suppose the query is revenue down. Then:
P ( q | d1 ) P ( q | d2 )
[(1/8 + 2/16)/2] × [(1/8 + 1/16)/2] 1/8 × 3/32 = 3/256 [(1/8 + 2/16)/2] × [(0/8 + 1/16)/2] 1/8 × 1/32 = 1/256
So, the ranking is d1 > d2 .
Ponte and Croft's Experiments Ponte and Croft (1998) present the first experiments on the language modeling approach to information retrieval. Their basic approach is the model that we have presented until now. However, we have presented an approach where the language model is a mixture of two multinomials, much as in (Miller et al. 1999, Hiemstra 2000) rather than Ponte and Croft's multivariate Bernoulli model. The use of multinomials has been standard in most subsequent work in the LM approach and experimental results in IR, as well as evidence from text classification which we consider in Section 13.3
Rec. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 Ave
tf-idf 0.7439 0.4521 0.3514 0.2761 0.2093 0.1558 0.1024 0.0451 0.0160 0.0033 0.0028 0.1868
Precision LM %chg 0.7590 +2.0 0.4910 +8.6 0.4045 +15.1 0.3342 +21.0 0.2572 +22.9 0.2061 +32.3 0.1405 +37.1 0.0760 +68.7 0.0432 +169.6 0.0063 +89.3 0.0050 +76.9 0.2233 +19.55
◮ Figure 12.4 Results of a comparison of tf-idf with language modeling (LM) term weighting by Ponte and Croft (1998). The version of tf-idf from the INQUERY IR system includes length normalization of tf. The table gives an evaluation according to 11-point average precision with significance marked with a * according to a Wilcoxon signed rank test. The language modeling approach always does better in these experiments, but note that where the approach shows significant gains is at higher levels of recall.
(page 263), suggests that it is superior. Ponte and Croft argued strongly for the effectiveness of the term weights that come from the language modeling approach over traditional tf-idf weights. We present a subset of their results in Figure 12.4 where they compare tf-idf to language modeling by evaluating TREC topics 202–250 over TREC disks 2 and 3. The queries are sentencelength natural language queries. The language modeling approach yields significantly better results than their baseline tf-idf based term weighting approach. And indeed the gains shown here have been extended in subsequent work.
Exercise 12.6
Consider making a language model from the following training text: the martian has landed on the latin pop sensation ricky martin a. Under a MLE-estimated unigram probability model, what are P (the) and P (martian)? b. Under a MLE-estimated bigram model, what are P (sensation|pop) and P (pop|the)?
12 Language models for information retrieval [⋆⋆] Exercise 12.7 Suppose we have a collection that consists of the 4 documents given in the below table. docID 1 2 3 4
Document text click go the shears boys click click click click click metal here metal shears click here
Build a query likelihood language model for this document collection. Assume a mixture model between the documents and the collection, with both weighted at 0.5. Maximum likelihood estimation (mle) is used to estimate both as unigram models. Work out the model probabilities of the queries click, shears, and hence click shears for each document, and use those probabilities to rank the documents returned by each query. Fill in these probabilities in the below table: Query
click shears click shears
What is the final ranking of the documents for the query click shears? Exercise 12.8 [⋆⋆] Using the calculations in Exercise 12.7 as inspiration or as examples where appropriate, write one sentence each describing the treatment that the model in Equation (12.10) gives to each of the following quantities. Include whether it is present in the model or not and whether the effect is raw or scaled. a. Term frequency in a document b. Collection frequency of a term c. Document frequency of a term d. Length normalization of a term Exercise 12.9 [⋆⋆] In the mixture model approach to the query likelihood model (Equation (12.12)), the probability estimate of a term is based on the term frequency of a word in a document, and the collection frequency of the word. Doing this certainly guarantees that each term of a query (in the vocabulary) has a non-zero chance of being generated by each document. But it has a more subtle but important effect of implementing a form of term weighting, related to what we saw in Chapter 6. Explain how this works. In particular, include in your answer a concrete numeric example showing this term weighting at work.
Language modeling versus other approaches in IR The language modeling approach provides a novel way of looking at the problem of text retrieval, which links it with a lot of recent work in speech
12.3 Language modeling versus other approaches in IR
and language processing. As Ponte and Croft (1998) emphasize, the language modeling approach to IR provides a different approach to scoring matches between queries and documents, and the hope is that the probabilistic language modeling foundation improves the weights that are used, and hence the performance of the model. The major issue is estimation of the document model, such as choices of how to smooth it effectively. The model has achieved very good retrieval results. Compared to other probabilistic approaches, such as the BIM from Chapter 11, the main difference initially appears to be that the LM approach does away with explicitly modeling relevance (whereas this is the central variable evaluated in the BIM approach). But this may not be the correct way to think about things, as some of the papers in Section 12.5 further discuss. The LM approach assumes that documents and expressions of information needs are objects of the same type, and assesses their match by importing the tools and methods of language modeling from speech and natural language processing. The resulting model is mathematically precise, conceptually simple, computationally tractable, and intuitively appealing. This seems similar to the situation with XML retrieval (Chapter 10): there the approaches that assume queries and documents are objects of the same type are also among the most successful. On the other hand, like all IR models, you can also raise objections to the model. The assumption of equivalence between document and information need representation is unrealistic. Current LM approaches use very simple models of language, usually unigram models. Without an explicit notion of relevance, relevance feedback is difficult to integrate into the model, as are user preferences. It also seems necessary to move beyond a unigram model to accommodate notions of phrase or passage matching or Boolean retrieval operators. Subsequent work in the LM approach has looked at addressing some of these concerns, including putting relevance back into the model and allowing a language mismatch between the query language and the document language. The model has significant relations to traditional tf-idf models. Term frequency is directly represented in tf-idf models, and much recent work has recognized the importance of document length normalization. The effect of doing a mixture of document generation probability with collection generation probability is a little like idf: terms rare in the general collection but common in some documents will have a greater influence on the ranking of documents. In most concrete realizations, the models share treating terms as if they were independent. On the other hand, the intuitions are probabilistic rather than geometric, the mathematical models are more principled rather than heuristic, and the details of how statistics like term frequency and document length are used differ. If you are concerned mainly with performance numbers, recent work has shown the LM approach to be very effective in retrieval experiments, beating tf-idf and BM25 weights. Nevertheless, there is
Query model
P(t|Query)
(a) (c) (b) Document
Doc. model
P(t|Document)
◮ Figure 12.5 Three ways of developing the language modeling approach: (a) query likelihood, (b) document likelihood, and (c) model comparison.
perhaps still insufficient evidence that its performance so greatly exceeds that of a well-tuned traditional vector space retrieval system as to justify changing an existing implementation.
DOCUMENT LIKELIHOOD MODEL
Extended language modeling approaches In this section we briefly mention some of the work that extends the basic language modeling approach. There are other ways to think of using the language modeling idea in IR settings, and many of them have been tried in subsequent work. Rather than looking at the probability of a document language model Md generating the query, you can look at the probability of a query language model Mq generating the document. The main reason that doing things in this direction and creating a document likelihood model is less appealing is that there is much less text available to estimate a language model based on the query text, and so the model will be worse estimated, and will have to depend more on being smoothed with some other language model. On the other hand, it is easy to see how to incorporate relevance feedback into such a model: you can expand the query with terms taken from relevant documents in the usual way and hence update the language model Mq (Zhai and Lafferty 2001a). Indeed, with appropriate modeling choices, this approach leads to the BIM model of Chapter 11. The relevance model of Lavrenko and Croft (2001) is an instance of a document likelihood model, which incorporates pseudo-relevance feedback into a language modeling approach. It achieves very strong empirical results. Rather than directly generating in either direction, we can make a language model from both the document and query, and then ask how different these two language models are from each other. Lafferty and Zhai (2001) lay
12.4 Extended language modeling approaches
K ULLBACK -L EIBLER DIVERGENCE
TRANSLATION MODEL
out these three ways of thinking about the problem, which we show in Figure 12.5, and develop a general risk minimization approach for document retrieval. For instance, one way to model the risk of returning a document d as relevant to a query q is to use the Kullback-Leibler (KL) divergence between their respective language models: R(d; q) = KL( Md k Mq ) =
P ( t | Mq )
∑ P(t| Mq ) log P(t| Md )
KL divergence is an asymmetric divergence measure originating in information theory, which measures how bad the probability distribution Mq is at modeling Md (Cover and Thomas 1991, Manning and Schütze 1999). Lafferty and Zhai (2001) present results suggesting that a model comparison approach outperforms both query-likelihood and document-likelihood approaches. One disadvantage of using KL divergence as a ranking function is that scores are not comparable across queries. This does not matter for ad hoc retrieval, but is important in other applications such as topic tracking. Kraaij and Spitters (2003) suggest an alternative proposal which models similarity as a normalized log-likelihood ratio (or, equivalently, as a difference between cross-entropies). Basic LMs do not address issues of alternate expression, that is, synonymy, or any deviation in use of language between queries and documents. Berger and Lafferty (1999) introduce translation models to bridge this query-document gap. A translation model lets you generate query words not in a document by translation to alternate terms with similar meaning. This also provides a basis for performing cross-language IR. We assume that the translation model can be represented by a conditional probability distribution T (·|·) between vocabulary terms. The form of the translation query generation model is then: P ( q | Md ) = ∏ ∑ P ( v | Md ) T ( t | v ) t ∈ q v ∈V
The term P(v| Md ) is the basic document language model, and the term T (t|v) performs translation. This model is clearly more computationally intensive and we need to build a translation model. The translation model is usually built using separate resources (such as a traditional thesaurus or bilingual dictionary or a statistical machine translation system's translation dictionary), but can be built using the document collection if there are pieces of text that naturally paraphrase or summarize other pieces of text. Candidate examples are documents and their titles or abstracts, or documents and anchor-text pointing to them in a hypertext environment. Building extended LM approaches remains an active area of research. In general, translation models, relevance feedback models, and model compar-
ison approaches have all been demonstrated to improve performance over the basic query likelihood LM.
References and further reading For more details on the basic concepts of probabilistic language models and techniques for smoothing, see either Manning and Schütze (1999, Chapter 6) or Jurafsky and Martin (2008, Chapter 4). The important initial papers that originated the language modeling approach to IR are: (Ponte and Croft 1998, Hiemstra 1998, Berger and Lafferty 1999, Miller et al. 1999). Other relevant papers can be found in the next several years of SIGIR proceedings. (Croft and Lafferty 2003) contains a collection of papers from a workshop on language modeling approaches and Hiemstra and Kraaij (2005) review one prominent thread of work on using language modeling approaches for TREC tasks. Zhai and Lafferty (2001b) clarify the role of smoothing in LMs for IR and present detailed empirical comparisons of different smoothing methods. Zaragoza et al. (2003) advocate using full Bayesian predictive distributions rather than MAP point estimates, but while they outperform Bayesian smoothing, they fail to outperform a linear interpolation. Zhai and Lafferty (2002) argue that a two-stage smoothing model with first Bayesian smoothing followed by linear interpolation gives a good model of the task, and performs better and more stably than a single form of smoothing. A nice feature of the LM approach is that it provides a convenient and principled way to put various kinds of prior information into the model; Kraaij et al. (2002) demonstrate this by showing the value of link information as a prior in improving web entry page retrieval performance. As briefly discussed in Chapter 16 (page 353), Liu and Croft (2004) show some gains by smoothing a document LM with estimates from a cluster of similar documents; Tao et al. (2006) report larger gains by doing document-similarity based smoothing. Hiemstra and Kraaij (2005) present TREC results showing a LM approach beating use of BM25 weights. Recent work has achieved some gains by going beyond the unigram model, providing the higher order models are smoothed with lower order models (Gao et al. 2004, Cao et al. 2005), though the gains to date remain modest. Spärck Jones (2004) presents a critical viewpoint on the rationale for the language modeling approach, but Lafferty and Zhai (2003) argue that a unified account can be given of the probabilistic semantics underlying both the language modeling approach presented in this chapter and the classical probabilistic information retrieval approach of Chapter 11. The Lemur Toolkit (http://www.lemurproject.org/) provides a flexible open source framework for investigating language modeling approaches to IR.
STANDING QUERY
ROUTING FILTERING
Text classification and Naive Bayes
Thus far, this book has mainly discussed the process of ad hoc retrieval, where users have transient information needs that they try to address by posing one or more queries to a search engine. However, many users have ongoing information needs. For example, you might need to track developments in multicore computer chips. One way of doing this is to issue the query multicore AND computer AND chip against an index of recent newswire articles each morning. In this and the following two chapters we examine the question: How can this repetitive task be automated? To this end, many systems support standing queries. A standing query is like any other query except that it is periodically executed on a collection to which new documents are incrementally added over time. If your standing query is just multicore AND computer AND chip, you will tend to miss many relevant new articles which use other terms such as multicore processors. To achieve good recall, standing queries thus have to be refined over time and can gradually become quite complex. In this example, using a Boolean search engine with stemming, you might end up with a query like (multicore OR multi-core) AND (chip OR processor OR microprocessor). To capture the generality and scope of the problem space to which standing queries belong, we now introduce the general notion of a classification problem. Given a set of classes, we seek to determine which class(es) a given object belongs to. In the example, the standing query serves to divide new newswire articles into the two classes: documents about multicore computer chips and documents not about multicore computer chips. We refer to this as two-class classification. Classification using standing queries is also called routing or filteringand will be discussed further in Section 15.3.1 (page 335). A class need not be as narrowly focused as the standing query multicore computer chips. Often, a class is a more general subject area like China or coffee. Such more general classes are usually referred to as topics, and the classification task is then called text classification, text categorization, topic classification, or topic spotting. An example for China appears in Figure 13.1. Standing queries and topics differ in their degree of specificity, but the methods for
13 Text classification and Naive Bayes
solving routing, filtering, and text classification are essentially the same. We therefore include routing and filtering under the rubric of text classification in this and the following chapters. The notion of classification is very general and has many applications within and beyond information retrieval (IR). For instance, in computer vision, a classifier may be used to divide images into classes such as landscape, portrait, and neither. We focus here on examples from information retrieval such as: • Several of the preprocessing steps necessary for indexing as discussed in Chapter 2: detecting a document's encoding (ASCII, Unicode UTF-8 etc; page 20); word segmentation (Is the white space between two letters a word boundary or not? page 24 ) ; truecasing (page 30); and identifying the language of a document (page 46). • The automatic detection of spam pages (which then are not included in the search engine index). • The automatic detection of sexually explicit content (which is included in search results only if the user turns an option such as SafeSearch off). SENTIMENT DETECTION
EMAIL SORTING
VERTICAL SEARCH ENGINE
• Sentiment detection or the automatic classification of a movie or product review as positive or negative. An example application is a user searching for negative reviews before buying a camera to make sure it has no undesirable features or quality problems. • Personal email sorting. A user may have folders like talk announcements, electronic bills, email from family and friends, and so on, and may want a classifier to classify each incoming email and automatically move it to the appropriate folder. It is easier to find messages in sorted folders than in a very large inbox. The most common case of this application is a spam folder that holds all suspected spam messages. • Topic-specific or vertical search. Vertical search engines restrict searches to a particular topic. For example, the query computer science on a vertical search engine for the topic China will return a list of Chinese computer science departments with higher precision and recall than the query computer science China on a general purpose search engine. This is because the vertical search engine does not include web pages in its index that contain the term china in a different sense (e.g., referring to a hard white ceramic), but does include relevant pages even if they do not explicitly mention the term China. • Finally, the ranking function in ad hoc information retrieval can also be based on a document classifier as we will explain in Section 15.4 (page 341).
RULES IN TEXT CLASSIFICATION
STATISTICAL TEXT CLASSIFICATION
This list shows the general importance of classification in IR. Most retrieval systems today contain multiple components that use some form of classifier. The classification task we will use as an example in this book is text classification. A computer is not essential for classification. Many classification tasks have traditionally been solved manually. Books in a library are assigned Library of Congress categories by a librarian. But manual classification is expensive to scale. The multicore computer chips example illustrates one alternative approach: classification by the use of standing queries – which can be thought of as rules – most commonly written by hand. As in our example (multicore OR multi-core) AND (chip OR processor OR microprocessor), rules are sometimes equivalent to Boolean expressions. A rule captures a certain combination of keywords that indicates a class. Hand-coded rules have good scaling properties, but creating and maintaining them over time is labor intensive. A technically skilled person (e.g., a domain expert who is good at writing regular expressions) can create rule sets that will rival or exceed the accuracy of the automatically generated classifiers we will discuss shortly; however, it can be hard to find someone with this specialized skill. Apart from manual classification and hand-crafted rules, there is a third approach to text classification, namely, machine learning-based text classification. It is the approach that we focus on in the next several chapters. In machine learning, the set of rules or, more generally, the decision criterion of the text classifier, is learned automatically from training data. This approach is also called statistical text classification if the learning method is statistical. In statistical text classification, we require a number of good example documents (or training documents) for each class. The need for manual classification is not eliminated because the training documents come from a person who has labeled them – where labeling refers to the process of annotating each document with its class. But labeling is arguably an easier task than writing rules. Almost anybody can look at a document and decide whether or not it is related to China. Sometimes such labeling is already implicitly part of an existing workflow. For instance, you may go through the news articles returned by a standing query each morning and give relevance feedback (cf. Chapter 9) by moving the relevant articles to a special folder like multicore-processors. We begin this chapter with a general introduction to the text classification problem including a formal definition (Section 13.1); we then cover Naive Bayes, a particularly simple and effective classification method (Sections 13.2– 13.4). All of the classification algorithms we study represent documents in high-dimensional spaces. To improve the efficiency of these algorithms, it is generally desirable to reduce the dimensionality of these spaces; to this end, a technique known as feature selection is commonly applied in text clas-
sification as discussed in Section 13.5. Section 13.6 covers evaluation of text classification. In the following chapters, Chapters 14 and 15, we look at two other families of classification methods, vector space classifiers and support vector machines.
13.1 DOCUMENT SPACE CLASS
The text classification problem In text classification, we are given a description d ∈ X of a document, where X is the document space; and a fixed set of classes C = {c1 , c2 , . . . , c J }. Classes are also called categories or labels. Typically, the document space X is some type of high-dimensional space, and the classes are human defined for the needs of an application, as in the examples China and documents that talk about multicore computer chips above. We are given a training set D of labeled documents hd, ci,where hd, ci ∈ X × C. For example:
hd, ci = hBeijing joins the World Trade Organization, Chinai
LEARNING METHOD CLASSIFIER
(13.1) SUPERVISED LEARNING
TEST SET
for the one-sentence document Beijing joins the World Trade Organization and the class (or label) China. Using a learning method or learning algorithm, we then wish to learn a classifier or classification function γ that maps documents to classes: γ:X→C
This type of learning is called supervised learning because a supervisor (the human who defines the classes and labels training documents) serves as a teacher directing the learning process. We denote the supervised learning method by Γ and write Γ(D ) = γ. The learning method Γ takes the training set D as input and returns the learned classification function γ. Most names for learning methods Γ are also used for classifiers γ. We talk about the Naive Bayes (NB) learning method Γ when we say that "Naive Bayes is robust," meaning that it can be applied to many different learning problems and is unlikely to produce classifiers that fail catastrophically. But when we say that "Naive Bayes had an error rate of 20%," we are describing an experiment in which a particular NB classifier γ (which was produced by the NB learning method) had a 20% error rate in an application. Figure 13.1 shows an example of text classification from the Reuters-RCV1 collection, introduced in Section 4.2, page 69. There are six classes (UK, China, . . . , sports), each with three training documents. We show a few mnemonic words for each document's content. The training set provides some typical examples for each class, so that we can learn the classification function γ. Once we have learned γ, we can apply it to the test set (or test data), for example, the new document first private Chinese airline whose class is unknown.
13.1 The text classification problem
γ(d′ ) =China regions
classes: training set:
recount
congestion London
run-off
bird flu
d′ test set:
first private Chinese airline
◮ Figure 13.1 Classes, training set, and test set in text classification .
In Figure 13.1, the classification function assigns the new document to class γ(d) = China, which is the correct assignment. The classes in text classification often have some interesting structure such as the hierarchy in Figure 13.1. There are two instances each of region categories, industry categories, and subject area categories. A hierarchy can be an important aid in solving a classification problem; see Section 15.3.2 for further discussion. Until then, we will make the assumption in the text classification chapters that the classes form a set with no subset relationships between them. Definition (13.1) stipulates that a document is a member of exactly one class. This is not the most appropriate model for the hierarchy in Figure 13.1. For instance, a document about the 2008 Olympics should be a member of two classes: the China class and the sports class. This type of classification problem is referred to as an any-of problem and we will return to it in Section 14.5 (page 306). For the time being, we only consider one-of problems where a document is a member of exactly one class. Our goal in text classification is high accuracy on test data or new data – for example, the newswire articles that we will encounter tomorrow morning in the multicore chip example. It is easy to achieve high accuracy on the training set (e.g., we can simply memorize the labels). But high accuracy on the training set in general does not mean that the classifier will work well on
new data in an application. When we use the training set to learn a classifier for test data, we make the assumption that training data and test data are similar or from the same distribution. We defer a precise definition of this notion to Section 14.6 (page 308).
13.2 MULTINOMIAL N AIVE B AYES
MAXIMUM A POSTERIORI CLASS
Naive Bayes text classification The first supervised learning method we introduce is the multinomial Naive Bayes or multinomial NB model, a probabilistic learning method. The probability of a document d being in class c is computed as P(c|d) ∝ P(c)
∏ 1≤k ≤n d
P(tk |c)
where P(tk |c) is the conditional probability of term tk occurring in a document of class c.1 We interpret P(tk |c) as a measure of how much evidence tk contributes that c is the correct class. P(c) is the prior probability of a document occurring in class c. If a document's terms do not provide clear evidence for one class versus another, we choose the one that has a higher prior probability. ht1 , t2 , . . . , tnd i are the tokens in d that are part of the vocabulary we use for classification and nd is the number of such tokens in d. For example, ht1 , t2 , . . . , tnd i for the one-sentence document Beijing and Taipei join the WTO might be hBeijing, Taipei, join, WTOi, with nd = 4, if we treat the terms and and the as stop words. In text classification, our goal is to find the best class for the document. The best class in NB classification is the most likely or maximum a posteriori (MAP) class cmap : cmap = arg max Pˆ (c|d) = arg max Pˆ (c) c ∈C
c ∈C
Pˆ (tk |c).
We write Pˆ for P because we do not know the true values of the parameters P(c) and P(tk |c), but estimate them from the training set as we will see in a moment. In Equation (13.3), many conditional probabilities are multiplied, one for each position 1 ≤ k ≤ nd . This can result in a floating point underflow. It is therefore better to perform the computation by adding logarithms of probabilities instead of multiplying probabilities. The class with the highest log probability score is still the most probable; log( xy) = log( x ) + log(y) and the logarithm function is monotonic. Hence, the maximization that is 1. We will explain in the next section why P ( c|d) is proportional to (∝), not equal to the quantity on the right.
13.2 Naive Bayes text classification
actually done in most implementations of NB is: (13.4)
cmap = arg max [log Pˆ (c) + c ∈C
∑ 1≤k ≤n d
log Pˆ (tk |c)].
Equation (13.4) has a simple interpretation. Each conditional parameter log Pˆ (tk |c) is a weight that indicates how good an indicator tk is for c. Similarly, the prior log Pˆ (c) is a weight that indicates the relative frequency of c. More frequent classes are more likely to be the correct class than infrequent classes. The sum of log prior and term weights is then a measure of how much evidence there is for the document being in the class, and Equation (13.4) selects the class for which we have the most evidence. We will initially work with this intuitive interpretation of the multinomial NB model and defer a formal derivation to Section 13.4. How do we estimate the parameters Pˆ (c) and Pˆ (tk |c)? We first try the maximum likelihood estimate (MLE; Section 11.3.2, page 226), which is simply the relative frequency and corresponds to the most likely value of each parameter given the training data. For the priors this estimate is: (13.5)
Nc Pˆ (c) = , N where Nc is the number of documents in class c and N is the total number of documents. We estimate the conditional probability Pˆ (t|c) as the relative frequency of term t in documents belonging to class c:
Pˆ (t|c) =
Tct , ∑t′ ∈V Tct′
where Tct is the number of occurrences of t in training documents from class c, including multiple occurrences of a term in a document. We have made the positional independence assumption here, which we will discuss in more detail in the next section: Tct is a count of occurrences in all positions k in the documents in the training set. Thus, we do not compute different estimates for different positions and, for example, if a word occurs twice in a document, in positions k1 and k2 , then Pˆ (tk1 |c) = Pˆ (tk2 |c). The problem with the MLE estimate is that it is zero for a term–class combination that did not occur in the training data. If the term WTO in the training data only occurred in China documents, then the MLE estimates for the other classes, for example UK, will be zero: Pˆ (WTO|UK) = 0.
Now, the one-sentence document Britain is a member of the WTO will get a conditional probability of zero for UK because we are multiplying the conditional probabilities for all terms in Equation (13.2). Clearly, the model should
T RAIN M ULTINOMIAL NB (C, D ) 1 V ← E XTRACT V OCABULARY (D ) 2 N ← C OUNT D OCS (D ) 3 for each c ∈ C 4 do Nc ← C OUNT D OCS I N C LASS (D, c) 5 prior [c] ← Nc /N 6 textc ← C ONCATENATE T EXT O FA LL D OCS I N C LASS(D, c) 7 for each t ∈ V 8 do Tct ← C OUNT T OKENS O F T ERM (textc , t) 9 for each t ∈ V 10 do condprob[t][c] ← ∑ ′T(ctT+′1+1) ct t 11 return V, prior, condprob A PPLY M ULTINOMIAL NB (C, V, prior, condprob, d) 1 W ← E XTRACT T OKENS F ROM D OC (V, d) 2 for each c ∈ C 3 do score[c] ← log prior [c] 4 for each t ∈ W 5 do score[c] += log condprob[t][c] 6 return arg maxc∈C score[c] ◮ Figure 13.2 Naive Bayes algorithm (multinomial model): Training and testing.
ADD - ONE SMOOTHING
assign a high probability to the UK class because the term Britain occurs. The problem is that the zero probability for WTO cannot be "conditioned away," no matter how strong the evidence for the class UK from other features. The estimate is 0 because of sparseness: The training data are never large enough to represent the frequency of rare events adequately, for example, the frequency of WTO occurring in UK documents. To eliminate zeros, we use add-one or Laplace smoothing, which simply adds one to each count (cf. Section 11.3.2): Pˆ (t|c) =
Tct + 1 Tct + 1 , = (∑t′ ∈V Tct′ ) + B ∑t′ ∈V ( Tct′ + 1)
where B = |V | is the number of terms in the vocabulary. Add-one smoothing can be interpreted as a uniform prior (each term occurs once for each class) that is then updated as evidence from the training data comes in. Note that this is a prior probability for the occurrence of a term as opposed to the prior probability of a class which we estimate in Equation (13.5) on the document level.
◮ Table 13.1 Data for parameter estimation examples.
docID 1 2 3 4 5
words in document Chinese Beijing Chinese Chinese Chinese Shanghai Chinese Macao Tokyo Japan Chinese Chinese Chinese Chinese Tokyo Japan
in c = China? yes yes yes no ?
◮ Table 13.2 Training and test times for NB.
mode training testing
time complexity Θ(|D | Lave + |C ||V |) Θ( La + |C | Ma ) = Θ(|C | Ma )
We have now introduced all the elements we need for training and applying an NB classifier. The complete algorithm is described in Figure 13.2.
Example 13.1: For the example in Table 13.1, the multinomial parameters we need to classify the test document are the priors Pˆ (c) = 3/4 and Pˆ (c) = 1/4 and the following conditional probabilities: Pˆ (Chinese| c) Pˆ (Tokyo| c) = Pˆ (Japan| c) Pˆ (Chinese| c) Pˆ (Tokyo| c) = Pˆ (Japan| c)
(5 + 1)/(8 + 6) = 6/14 = 3/7
(0 + 1)/(8 + 6) = 1/14
(1 + 1)/(3 + 6) = 2/9
The denominators are (8 + 6) and (3 + 6) because the lengths of textc and textc are 8 and 3, respectively, and because the constant B in Equation (13.7) is 6 as the vocabulary consists of six terms. We then get: Pˆ (c| d5 ) Pˆ (c | d5 )
∝ ∝
3/4 · (3/7)3 · 1/14 · 1/14 ≈ 0.0003.
1/4 · (2/9)3 · 2/9 · 2/9 ≈ 0.0001.
Thus, the classifier assigns the test document to c = China. The reason for this classification decision is that the three occurrences of the positive indicator Chinese in d5 outweigh the occurrences of the two negative indicators Japan and Tokyo.
What is the time complexity of NB? The complexity of computing the parameters is Θ(|C ||V |) because the set of parameters consists of |C ||V | conditional probabilities and |C | priors. The preprocessing necessary for computing the parameters (extracting the vocabulary, counting terms, etc.) can be done in one pass through the training data. The time complexity of this
component is therefore Θ(|D | Lave ), where |D | is the number of documents and Lave is the average length of a document. We use Θ(|D | Lave ) as a notation for Θ( T ) here, where T is the length of the training collection. This is nonstandard; Θ(.) is not defined for an average. We prefer expressing the time complexity in terms of D and Lave because these are the primary statistics used to characterize training collections. The time complexity of A PPLY M ULTINOMIAL NB in Figure 13.2 is Θ(|C | La ). La and Ma are the numbers of tokens and types, respectively, in the test document. A PPLY M ULTINOMIAL NB can be modified to be Θ( La + |C | Ma ) (Exercise 13.8). Finally, assuming that the length of test documents is bounded, Θ( La + |C | Ma ) = Θ(|C | Ma ) because La < b|C | Ma for a fixed constant b.2 Table 13.2 summarizes the time complexities. In general, we have |C ||V | < |D | Lave , so both training and testing complexity are linear in the time it takes to scan the data. Because we have to look at the data at least once, NB can be said to have optimal time complexity. Its efficiency is one reason why NB is a popular text classification method.
Relation to multinomial unigram language model The multinomial NB model is formally identical to the multinomial unigram language model (Section 12.2.1, page 242). In particular, Equation (13.2) is a special case of Equation (12.12) from page 243, which we repeat here for λ = 1: P ( d | q ) ∝ P ( d ) ∏ P ( t | Md ) .
t∈q
The document d in text classification (Equation (13.2)) takes the role of the query in language modeling (Equation (13.8)) and the classes c in text classification take the role of the documents d in language modeling. We used Equation (13.8) to rank documents according to the probability that they are relevant to the query q. In NB classification, we are usually only interested in the top-ranked class. We also used MLE estimates in Section 12.2.2 (page 243) and encountered the problem of zero estimates owing to sparse data (page 244); but instead of add-one smoothing, we used a mixture of two distributions to address the problem there. Add-one smoothing is closely related to add- 21 smoothing in Section 11.3.4 (page 228).
Exercise 13.1 Why is |C ||V | < |D | Lave in Table 13.2 expected to hold for most text collections? 2. Our assumption here is that the length of test documents is bounded. La would exceed b |C| Ma for extremely long test documents.
13.3 The Bernoulli model
T RAIN B ERNOULLI NB (C, D ) 1 V ← E XTRACT V OCABULARY (D ) 2 N ← C OUNT D OCS (D ) 3 for each c ∈ C 4 do Nc ← C OUNT D OCS I N C LASS (D, c) 5 prior [c] ← Nc /N 6 for each t ∈ V 7 do Nct ← C OUNT D OCS I N C LASS C ONTAINING T ERM (D, c, t) 8 condprob[t][c] ← ( Nct + 1)/( Nc + 2) 9 return V, prior, condprob A PPLY B ERNOULLI NB (C, V, prior, condprob, d) 1 Vd ← E XTRACT T ERMS F ROM D OC (V, d) 2 for each c ∈ C 3 do score[c] ← log prior [c] 4 for each t ∈ V 5 do if t ∈ Vd 6 then score[c] += log condprob[t][c] 7 else score[c] += log(1 − condprob[t][c]) 8 return arg maxc∈C score[c] ◮ Figure 13.3 NB algorithm (Bernoulli model): Training and testing. The add-one smoothing in Line 8 (top) is in analogy to Equation (13.7) with B = 2.
B ERNOULLI MODEL
The Bernoulli model There are two different ways we can set up an NB classifier. The model we introduced in the previous section is the multinomial model. It generates one term from the vocabulary in each position of the document, where we assume a generative model that will be discussed in more detail in Section 13.4 (see also page 237). An alternative to the multinomial model is the multivariate Bernoulli model or Bernoulli model. It is equivalent to the binary independence model of Section 11.3 (page 222), which generates an indicator for each term of the vocabulary, either 1 indicating presence of the term in the document or 0 indicating absence. Figure 13.3 presents training and testing algorithms for the Bernoulli model. The Bernoulli model has the same time complexity as the multinomial model. The different generation models imply different estimation strategies and different classification rules. The Bernoulli model estimates Pˆ (t|c) as the fraction of documents of class c that contain term t (Figure 13.3, T RAIN B ERNOULLI -
NB, line 8). In contrast, the multinomial model estimates Pˆ (t|c) as the fraction of tokens or fraction of positions in documents of class c that contain term t (Equation (13.7)). When classifying a test document, the Bernoulli model uses binary occurrence information, ignoring the number of occurrences, whereas the multinomial model keeps track of multiple occurrences. As a result, the Bernoulli model typically makes many mistakes when classifying long documents. For example, it may assign an entire book to the class China because of a single occurrence of the term China. The models also differ in how nonoccurring terms are used in classification. They do not affect the classification decision in the multinomial model; but in the Bernoulli model the probability of nonoccurrence is factored in when computing P(c|d) (Figure 13.3, A PPLY B ERNOULLI NB, Line 7). This is because only the Bernoulli NB model models absence of terms explicitly.
Applying the Bernoulli model to the example in Table 13.1, we have the same estimates for the priors as before: Pˆ (c) = 3/4, Pˆ (c) = 1/4. The conditional probabilities are:
Pˆ (Chinese| c) Pˆ (Japan| c) = Pˆ (Tokyo| c) ˆ ˆ P (Beijing| c) = P (Macao| c) = Pˆ (Shanghai| c) Pˆ (Chinese| c)
Pˆ (Japan| c) = Pˆ (Tokyo| c) Pˆ (Beijing| c) = Pˆ (Macao| c) = Pˆ (Shanghai| c)
The denominators are (3 + 2) and (1 + 2) because there are three documents in c and one document in c and because the constant B in Equation (13.7) is 2 – there are two cases to consider for each term, occurrence and nonoccurrence. The scores of the test document for the two classes are Pˆ (c| d5 )
Pˆ (c) · Pˆ (Chinese| c) · Pˆ (Japan| c) · Pˆ (Tokyo| c) · (1 − Pˆ (Beijing| c)) · (1 − Pˆ (Shanghai| c)) · (1 − Pˆ (Macao| c))
3/4 · 4/5 · 1/5 · 1/5 · (1 − 2/5) · (1 − 2/5) · (1 − 2/5) 0.005
and, analogously, Pˆ (c | d5 )
∝ ≈
Thus, the classifier assigns the test document to c = not-China. When looking only at binary occurrence and not at term frequency, Japan and Tokyo are indicators for c (2/3 > 1/5) and the conditional probabilities of Chinese for c and c are not different enough (4/5 vs. 2/3) to affect the classification decision.
13.4 Properties of Naive Bayes
Properties of Naive Bayes To gain a better understanding of the two models and the assumptions they make, let us go back and examine how we derived their classification rules in Chapters 11 and 12. We decide class membership of a document by assigning it to the class with the maximum a posteriori probability (cf. Section 11.3.2, page 226), which we compute as follows: cmap
= arg max P(c|d) c ∈C
= arg max c ∈C
P(d|c) P(c) P(d)
= arg max P(d|c) P(c), c ∈C
where Bayes' rule (Equation (11.4), page 220) is applied in (13.9) and we drop the denominator in the last step because P(d) is the same for all classes and does not affect the argmax. We can interpret Equation (13.10) as a description of the generative process we assume in Bayesian text classification. To generate a document, we first choose class c with probability P(c) (top nodes in Figures 13.4 and 13.5). The two models differ in the formalization of the second step, the generation of the document given the class, corresponding to the conditional distribution P ( d | c ): (13.11)
P(d|c)
P(ht1, . . . , tk , . . . , tnd i|c)
P(he1, . . . , ei , . . . , e M i|c),
where ht1 , . . . , tnd i is the sequence of terms as it occurs in d (minus terms that were excluded from the vocabulary) and he1 , . . . , ei , . . . , e M i is a binary vector of dimensionality M that indicates for each term whether it occurs in d or not. It should now be clearer why we introduced the document space X in Equation (13.1) when we defined the classification problem. A critical step in solving a text classification problem is to choose the document representation. ht1 , . . . , tnd i and he1 , . . . , e M i are two different document representations. In the first case, X is the set of all term sequences (or, more precisely, sequences of term tokens). In the second case, X is {0, 1} M . We cannot use Equations (13.11) and (13.12) for text classification directly. For the Bernoulli model, we would have to estimate 2 M |C | different parameters, one for each possible combination of M values ei and a class. The number of parameters in the multinomial case has the same order of magni-
C=China
X1 =Beijing
X2 =and
X3 =Taipei
X4 =join
X5 =WTO
◮ Figure 13.4 The multinomial NB model.
CONDITIONAL INDEPENDENCE ASSUMPTION
(13.13) (13.14)
RANDOM VARIABLE U
tude.3 This being a very large quantity, estimating these parameters reliably is infeasible. To reduce the number of parameters, we make the Naive Bayes conditional independence assumption. We assume that attribute values are independent of each other given the class: Multinomial Bernoulli
P(ht1, . . . , tnd i|c) = P(he1 , . . . , e M i|c) =
1≤i ≤ M
P ( Xk = t k | c ) P ( Ui = e i | c ) .
We have introduced two random variables here to make the two different generative models explicit. Xk is the random variable for position k in the document and takes as values terms from the vocabulary. P( Xk = t|c) is the probability that in a document of class c the term t will occur in position k. Ui is the random variable for vocabulary term i and takes as values 0 (absence) and 1 (presence). Pˆ (Ui = 1|c) is the probability that in a document of class c the term ti will occur – in any position and possibly multiple times. We illustrate the conditional independence assumption in Figures 13.4 and 13.5. The class China generates values for each of the five term attributes (multinomial) or six binary attributes (Bernoulli) with a certain probability, independent of the values of the other attributes. The fact that a document in the class China contains the term Taipei does not make it more likely or less likely that it also contains Beijing. In reality, the conditional independence assumption does not hold for text data. Terms are conditionally dependent on each other. But as we will discuss shortly, NB models perform well despite the conditional independence assumption. 3. In fact, if the length of documents is not bounded, the number of parameters in the multinomial case is infinite.
U Alaska =0
U Beijing =1
U India =0
U join =1
U Taipei =1
U WTO =1
◮ Figure 13.5 The Bernoulli NB model.
POSITIONAL INDEPENDENCE
Even when assuming conditional independence, we still have too many parameters for the multinomial model if we assume a different probability distribution for each position k in the document. The position of a term in a document by itself does not carry information about the class. Although there is a difference between China sues France and France sues China, the occurrence of China in position 1 versus position 3 of the document is not useful in NB classification because we look at each term separately. The conditional independence assumption commits us to this way of processing the evidence. Also, if we assumed different term distributions for each position k, we would have to estimate a different set of parameters for each k. The probability of bean appearing as the first term of a coffee document could be different from it appearing as the second term, and so on. This again causes problems in estimation owing to data sparseness. For these reasons, we make a second independence assumption for the multinomial model, positional independence: The conditional probabilities for a term are the same independent of position in the document. P ( Xk 1 = t | c ) = P ( Xk 2 = t | c ) for all positions k1 , k2 , terms t and classes c. Thus, we have a single distribution of terms that is valid for all positions k i and we can use X as its symbol.4 Positional independence is equivalent to adopting the bag of words model, which we introduced in the context of ad hoc retrieval in Chapter 6 (page 117). With conditional and positional independence assumptions, we only need to estimate Θ( M |C |) parameters P(tk |c) (multinomial model) or P(ei |c) (Bernoulli 4. Our terminology is nonstandard. The random variable X is a categorical variable, not a multinomial variable, and the corresponding NB model should perhaps be called a sequence model. We have chosen to present this sequence model and the multinomial model in Section 13.4.1 as the same model because they are computationally identical.
◮ Table 13.3 Multinomial versus Bernoulli model.
event model random variable(s) document representation
multinomial model generation of token X = t iff t occurs at given pos d = h t1 , . . . , t k , . . . , t n d i, t k ∈ V
parameter estimation decision rule: maximize multiple occurrences length of docs # features estimate for term the
Pˆ ( X = t|c) Pˆ (c) ∏1≤k≤nd Pˆ ( X = tk |c) taken into account can handle longer docs can handle more Pˆ ( X = the|c) ≈ 0.05
Bernoulli model generation of document Ut = 1 iff t occurs in doc d = h e1 , . . . , e i , . . . , e M i , ei ∈ {0, 1} Pˆ (Ui = e|c) Pˆ (c) ∏ti ∈V Pˆ (Ui = ei |c) ignored works best for short docs works best with fewer Pˆ (U the = 1|c) ≈ 1.0
model), one for each term–class combination, rather than a number that is at least exponential in M, the size of the vocabulary. The independence assumptions reduce the number of parameters to be estimated by several orders of magnitude. To summarize, we generate a document in the multinomial model (Figure 13.4) by first picking a class C = c with P(c) where C is a random variable taking values from C as values. Next we generate term tk in position k with P( Xk = tk |c) for each of the nd positions of the document. The Xk all have the same distribution over terms for a given c. In the example in Figure 13.4, we show the generation of ht1 , t2 , t3 , t4 , t5 i = hBeijing, and, Taipei, join, WTOi, corresponding to the one-sentence document Beijing and Taipei join WTO. For a completely specified document generation model, we would also have to define a distribution P(nd |c) over lengths. Without it, the multinomial model is a token generation model rather than a document generation model. We generate a document in the Bernoulli model (Figure 13.5) by first picking a class C = c with P(c) and then generating a binary indicator ei for each term ti of the vocabulary (1 ≤ i ≤ M). In the example in Figure 13.5, we show the generation of he1 , e2 , e3 , e4 , e5 , e6 i = h0, 1, 0, 1, 1, 1i, corresponding, again, to the one-sentence document Beijing and Taipei join WTO where we have assumed that and is a stop word. We compare the two models in Table 13.3, including estimation equations and decision rules. Naive Bayes is so called because the independence assumptions we have just made are indeed very naive for a model of natural language. The conditional independence assumption states that features are independent of each other given the class. This is hardly ever true for terms in documents. In many cases, the opposite is true. The pairs hong and kong or london and en-
◮ Table 13.4 Correct estimation implies accurate prediction, but accurate prediction does not imply correct estimation.
true probability P(c|d) Pˆ (c) ∏1≤k≤nd Pˆ (tk |c) (Equation (13.13)) NB estimate Pˆ (c|d)
c1 0.6 0.00099 0.99
class selected c1 c1
glish in Figure 13.7 are examples of highly dependent terms. In addition, the
multinomial model makes an assumption of positional independence. The Bernoulli model ignores positions in documents altogether because it only cares about absence or presence. This bag-of-words model discards all information that is communicated by the order of words in natural language sentences. How can NB be a good text classifier when its model of natural language is so oversimplified? The answer is that even though the probability estimates of NB are of low quality, its classification decisions are surprisingly good. Consider a document d with true probabilities P(c1 |d) = 0.6 and P(c2 |d) = 0.4 as shown in Table 13.4. Assume that d contains many terms that are positive indicators for c1 and many terms that are negative indicators for c2 . Thus, when using the multinomial model in Equation (13.13), Pˆ (c1 ) ∏1≤k≤nd Pˆ (tk |c1 ) will be much larger than Pˆ (c2 ) ∏1≤k≤nd Pˆ (tk |c2 ) (0.00099 vs. 0.00001 in the table). After division by 0.001 to get well-formed probabilities for P(c|d), we end up with one estimate that is close to 1.0 and one that is close to 0.0. This is common: The winning class in NB classification usually has a much larger probability than the other classes and the estimates diverge very significantly from the true probabilities. But the classification decision is based on which class gets the highest score. It does not matter how accurate the estimates are. Despite the bad estimates, NB estimates a higher probability for c1 and therefore assigns d to the correct class in Table 13.4. Correct estimation implies accurate prediction, but accurate prediction does not imply correct estimation. NB classifiers estimate badly, but often classify well. Even if it is not the method with the highest accuracy for text, NB has many virtues that make it a strong contender for text classification. It excels if there are many equally important features that jointly contribute to the classification decision. It is also somewhat robust to noise features (as defined in the next section) and concept drift – the gradual change over time of the concept underlying a class like US president from Bill Clinton to George W. Bush (see Section 13.7). Classifiers like kNN (Section 14.3, page 297) can be carefully tuned to idiosyncratic properties of a particular time period. This will then hurt them when documents in the following time period have slightly
◮ Table 13.5 problematic.
OPTIMAL CLASSIFIER
A set of documents for which the NB independence assumptions are
He moved from London, Ontario, to London, England. He moved from London, England, to London, Ontario. He moved from England to London, Ontario.
different properties. The Bernoulli model is particularly robust with respect to concept drift. We will see in Figure 13.8 that it can have decent performance when using fewer than a dozen terms. The most important indicators for a class are less likely to change. Thus, a model that only relies on these features is more likely to maintain a certain level of accuracy in concept drift. NB's main strength is its efficiency: Training and classification can be accomplished with one pass over the data. Because it combines efficiency with good accuracy it is often used as a baseline in text classification research. It is often the method of choice if (i) squeezing out a few extra percentage points of accuracy is not worth the trouble in a text classification application, (ii) a very large amount of training data is available and there is more to be gained from training on a lot of data than using a better classifier on a smaller training set, or (iii) if its robustness to concept drift can be exploited. In this book, we discuss NB as a classifier for text. The independence assumptions do not hold for text. However, it can be shown that NB is an optimal classifier (in the sense of minimal error rate on new data) for data where the independence assumptions do hold.
A variant of the multinomial model An alternative formalization of the multinomial model represents each document d as an M-dimensional vector of counts htft1 ,d , . . . , tft M ,d i where tfti ,d is the term frequency of ti in d. P(d|c) is then computed as follows (cf. Equation (12.8), page 243);
P(d|c) = P(htft1 ,d , . . . , tft M ,d i|c) ∝
∏ 1≤i ≤ M
P( X = ti |c)
i ,d
Note that we have omitted the multinomial factor. See Equation (12.8) (page 243). Equation (13.15) is equivalent to the sequence model in Equation (13.2) as tf P( X = ti |c) ti ,d = 1 for terms that do not occur in d (tfti ,d = 0) and a term that occurs tfti ,d ≥ 1 times will contribute tfti ,d factors both in Equation (13.2) and in Equation (13.15).
13.5 Feature selection
S ELECT F EATURES(D, c, k) 1 V ← E XTRACT V OCABULARY (D ) 2 L ← [] 3 for each t ∈ V 4 do A(t, c) ← C OMPUTE F EATURE U TILITY (D, t, c) 5 A PPEND ( L, h A(t, c), ti) 6 return F EATURES W ITH L ARGEST VALUES( L, k) ◮ Figure 13.6 Basic feature selection algorithm for selecting the k best features.
Which of the documents in Table 13.5 have identical and different bag of words representations for (i) the Bernoulli model (ii) the multinomial model? If there are differences, describe them. Exercise 13.3 The rationale for the positional independence assumption is that there is no useful information in the fact that a term occurs in position k of a document. Find exceptions. Consider formulaic documents with a fixed document structure. Exercise 13.4 Table 13.3 gives Bernoulli and multinomial estimates for the word the. Explain the difference.
NOISE FEATURE
Feature selection Feature selection is the process of selecting a subset of the terms occurring in the training set and using only this subset as features in text classification. Feature selection serves two main purposes. First, it makes training and applying a classifier more efficient by decreasing the size of the effective vocabulary. This is of particular importance for classifiers that, unlike NB, are expensive to train. Second, feature selection often increases classification accuracy by eliminating noise features. A noise feature is one that, when added to the document representation, increases the classification error on new data. Suppose a rare term, say arachnocentric, has no information about a class, say China, but all instances of arachnocentric happen to occur in China documents in our training set. Then the learning method might produce a classifier that misassigns test documents containing arachnocentric to China. Such an incorrect generalization from an accidental property of the training set is called overfitting. We can view feature selection as a method for replacing a complex classifier (using all features) with a simpler one (using a subset of the features).
It may appear counterintuitive at first that a seemingly weaker classifier is advantageous in statistical text classification, but when discussing the biasvariance tradeoff in Section 14.6 (page 308), we will see that weaker models are often preferable when limited training data are available. The basic feature selection algorithm is shown in Figure 13.6. For a given class c, we compute a utility measure A(t, c) for each term of the vocabulary and select the k terms that have the highest values of A(t, c). All other terms are discarded and not used in classification. We will introduce three different utility measures in this section: mutual information, A(t, c) = I (Ut ; Cc ); the χ2 test, A(t, c) = X 2 (t, c); and frequency, A(t, c) = N (t, c). Of the two NB models, the Bernoulli model is particularly sensitive to noise features. A Bernoulli NB classifier requires some form of feature selection or else its accuracy will be low. This section mainly addresses feature selection for two-class classification tasks like China versus not-China. Section 13.5.5 briefly discusses optimizations for systems with more than two classes.
13.5.1 MUTUAL INFORMATION
Mutual information A common feature selection method is to compute A(t, c) as the expected mutual information (MI) of term t and class c.5 MI measures how much information the presence/absence of a term contributes to making the correct classification decision on c. Formally: I (U; C )
e t ∈{1,0} e c ∈{1,0}
P(U = et , C = ec ) log2
P (U = et , C = ec ) , P (U = et ) P ( C = ec )
where U is a random variable that takes values et = 1 (the document contains term t) and et = 0 (the document does not contain t), as defined on page 266, and C is a random variable that takes values ec = 1 (the document is in class c) and ec = 0 (the document is not in class c). We write Ut and Cc if it is not clear from context which term t and class c we are referring to. ForMLEs of the probabilities, Equation (13.16) is equivalent to Equation (13.17): (13.17)
I (U; C )
N N11 N N N01 N11 + 01 log2 log2 N N1. N.1 N N0. N.1 N N10 N N N00 N + 00 log2 + 10 log2 N N1. N.0 N N0. N.0
where the Ns are counts of documents that have the values of et and ec that are indicated by the two subscripts. For example, N10 is the number of doc5. Take care not to confuse expected mutual information with pointwise mutual information, which is defined as log N11 /E11 where N11 and E11 are defined as in Equation (13.18). The two measures have different properties. See Section 13.7.
uments that contain t (et = 1) and are not in c (ec = 0). N1. = N10 + N11 is the number of documents that contain t (et = 1) and we count documents independent of class membership (ec ∈ {0, 1}). N = N00 + N01 + N10 + N11 is the total number of documents. An example of one of the MLE estimates that transform Equation (13.16) into Equation (13.17) is P(U = 1, C = 1) = N11 /N.
Example 13.3: Consider the class poultry and the term export in Reuters-RCV1. The counts of the number of documents with the four possible combinations of indicator values are as follows: ec = epoultry = 1 ec = epoultry = 0 et = eexport = 1 N11 = 49 N10 = 27,652 et = eexport = 0 N01 = 141 N00 = 774,106 After plugging these values into Equation (13.17) we get: I (U; C )
49 801,948 · 49 log2 801,948 (49 + 27,652)(49 + 141) 141 801,948 · 141 + log2 801,948 (141 + 774,106)(49 + 141) 27,652 801,948 · 27,652 + log2 801,948 (49 + 27,652)(27,652 + 774,106) 801,948 · 774,106 774,106 log2 + 801,948 (141 + 774,106)(27,652 + 774,106) 0.0001105
To select k terms t1 , . . . , tk for a given class, we use the feature selection algorithm in Figure 13.6: We compute the utility measure as A(t, c) = I (Ut , Cc ) and select the k terms with the largest values. Mutual information measures how much information – in the informationtheoretic sense – a term contains about the class. If a term's distribution is the same in the class as it is in the collection as a whole, then I (U; C ) = 0. MI reaches its maximum value if the term is a perfect indicator for class membership, that is, if the term is present in a document if and only if the document is in the class. Figure 13.7 shows terms with high mutual information scores for the six classes in Figure 13.1.6 The selected terms (e.g., london, uk, british for the class UK) are of obvious utility for making classification decisions for their respective classes. At the bottom of the list for UK we find terms like peripherals and tonight (not shown in the figure) that are clearly not helpful in deciding 6. Feature scores were computed on the first 100,000 documents, except for poultry, a rare class, for which 800,000 documents were used. We have omitted numbers and other special words from the top ten lists.
UK 0.1925 0.0755 0.0596 0.0555 0.0469 0.0357 0.0238 0.0212 0.0149 0.0126 coffee coffee 0.0111 bags 0.0042 growers 0.0025 kg 0.0019 colombia 0.0018 brazil 0.0016 export 0.0014 exporters 0.0013 exports 0.0013 crop 0.0012 london uk british stg britain plc england pence pounds english
China 0.0997 0.0523 0.0444 0.0344 0.0292 0.0198 0.0195 0.0155 0.0117 0.0108 elections election 0.0519 elections 0.0342 polls 0.0339 voters 0.0315 party 0.0303 vote 0.0299 poll 0.0225 candidate 0.0202 campaign 0.0202 democratic 0.0198 china chinese beijing yuan shanghai hong kong xinhua province taiwan
poultry 0.0013 0.0008 0.0006 0.0005 0.0004 0.0003 0.0003 0.0003 0.0003 0.0003 sports soccer 0.0681 cup 0.0515 match 0.0441 matches 0.0408 played 0.0388 league 0.0386 beat 0.0301 game 0.0299 games 0.0284 team 0.0264 poultry meat chicken agriculture avian broiler veterinary birds inspection pathogenic
◮ Figure 13.7 Features with high mutual information scores for six Reuters-RCV1 classes.
whether the document is in the class. As you might expect, keeping the informative terms and eliminating the non-informative ones tends to reduce noise and improve the classifier's accuracy. Such an accuracy increase can be observed in Figure 13.8, which shows F1 as a function of vocabulary size after feature selection for Reuters-RCV1.7 Comparing F1 at 132,776 features (corresponding to selection of all features) and at 10–100 features, we see that MI feature selection increases F1 by about 0.1 for the multinomial model and by more than 0.2 for the Bernoulli model. For the Bernoulli model, F1 peaks early, at ten features selected. At that point, the Bernoulli model is better than the multinomial model. When basing a classification decision on only a few features, it is more robust to consider binary occurrence only. For the multinomial model (MI feature selection), the peak occurs later, at 100 features, and its effectiveness recovers somewhat at 7. We trained the classifiers on the first 100,000 documents and computed F1 on the next 100,000. The graphs are averages over five classes.
# b
x b xx b
o b
o x x
x # o b
x b o #
x o #
o # #
F1 measure
#o o oo # #
o x # b
# o x b
multinomial, MI multinomial, chisquare multinomial, frequency binomial, MI
number of features selected
◮ Figure 13.8 Effect of feature set size on accuracy for multinomial and Bernoulli models.
the end when we use all features. The reason is that the multinomial takes the number of occurrences into account in parameter estimation and classification and therefore better exploits a larger number of features than the Bernoulli model. Regardless of the differences between the two methods, using a carefully selected subset of the features results in better effectiveness than using all features.
13.5.2 χ2
FEATURE SELECTION
χ2 Feature selection Another popular feature selection method is χ2 . In statistics, the χ2 test is applied to test the independence of two events, where two events A and B are defined to be independent if P( AB) = P( A) P( B) or, equivalently, P( A| B) = P( A) and P( B| A) = P( B). In feature selection, the two events are occurrence of the term and occurrence of the class. We then rank terms with respect to the following quantity: X 2 (D, t, c) =
∑ e t ∈{0,1} e c
( Net ec − Eet ec )2 Ee t e c ∈{0,1}
where et and ec are defined as in Equation (13.16). N is the observed frequency in D and E the expected frequency. For example, E11 is the expected frequency of t and c occurring together in a document assuming that term and class are independent.
We first compute E11 for the data in Example 13.3:
N11 + N10 N + N01 × 11 N N 49 + 27652 49 + 141 × ≈ 6.6 N× N N N × P (t ) × P (c) = N ×
where N is the total number of documents as before. We compute the other Ee t e c in the same way: eexport = 1 eexport = 0
epoultry = 1 N11 = 49 E11 ≈ 6.6 N01 = 141 E01 ≈ 183.4
epoultry = 0 N10 = 27,652 E10 ≈ 27,694.4 N00 = 774,106 E00 ≈ 774,063.6
Plugging these values into Equation (13.18), we get a X2 value of 284: X2 (D, t, c) =
STATISTICAL SIGNIFICANCE
( Ne t e c − Ee t e c )2 ≈ 284 Ee t e c ∈{0,1}
X 2 is a measure of how much expected counts E and observed counts N deviate from each other. A high value of X 2 indicates that the hypothesis of independence, which implies that expected and observed counts are similar, is incorrect. In our example, X 2 ≈ 284 > 10.83. Based on Table 13.6, we can reject the hypothesis that poultry and export are independent with only a 0.001 chance of being wrong.8 Equivalently, we say that the outcome X 2 ≈ 284 > 10.83 is statistically significant at the 0.001 level. If the two events are dependent, then the occurrence of the term makes the occurrence of the class more likely (or less likely), so it should be helpful as a feature. This is the rationale of χ2 feature selection. An arithmetically simpler way of computing X 2 is the following: X 2 (D, t, c) =
( N11 + N10 + N01 + N00 ) × ( N11 N00 − N10 N01 )2 ( N11 + N01 ) × ( N11 + N10 ) × ( N10 + N00 ) × ( N01 + N00 )
This is equivalent to Equation (13.18) (Exercise 13.14). 8. We can make this inference because, if the two events are independent, then X 2 ∼ χ2 , where χ2 is the χ2 distribution. See, for example, Rice (2006).
◮ Table 13.6 Critical values of the χ2 distribution with one degree of freedom. For example, if the two events are independent, then P ( X 2 > 6.63) < 0.01. So for X 2 > 6.63 the assumption of independence can be rejected with 99% confidence.
p 0.1 0.05 0.01 0.005 0.001
χ2 critical value 2.71 3.84 6.63 7.88 10.83
Assessing χ2 as a feature selection method From a statistical point of view, χ2 feature selection is problematic. For a test with one degree of freedom, the so-called Yates correction should be used (see Section 13.7), which makes it harder to reach statistical significance. Also, whenever a statistical test is used multiple times, then the probability of getting at least one error increases. If 1,000 hypotheses are rejected, each with 0.05 error probability, then 0.05 × 1000 = 50 calls of the test will be wrong on average. However, in text classification it rarely matters whether a few additional terms are added to the feature set or removed from it. Rather, the relative importance of features is important. As long as χ2 feature selection only ranks features with respect to their usefulness and is not used to make statements about statistical dependence or independence of variables, we need not be overly concerned that it does not adhere strictly to statistical theory.
Frequency-based feature selection A third feature selection method is frequency-based feature selection, that is, selecting the terms that are most common in the class. Frequency can be either defined as document frequency (the number of documents in the class c that contain the term t) or as collection frequency (the number of tokens of t that occur in documents in c). Document frequency is more appropriate for the Bernoulli model, collection frequency for the multinomial model. Frequency-based feature selection selects some frequent terms that have no specific information about the class, for example, the days of the week (Monday, Tuesday, . . . ), which are frequent across classes in newswire text. When many thousands of features are selected, then frequency-based feature selection often does well. Thus, if somewhat suboptimal accuracy is acceptable, then frequency-based feature selection can be a good alternative to more complex methods. However, Figure 13.8 is a case where frequency-
based feature selection performs a lot worse than MI and χ2 and should not be used.
Feature selection for multiple classifiers In an operational system with a large number of classifiers, it is desirable to select a single set of features instead of a different one for each classifier. One way of doing this is to compute the X 2 statistic for an n × 2 table where the columns are occurrence and nonoccurrence of the term and each row corresponds to one of the classes. We can then select the k terms with the highest X 2 statistic as before. More commonly, feature selection statistics are first computed separately for each class on the two-class classification task c versus c and then combined. One combination method computes a single figure of merit for each feature, for example, by averaging the values A(t, c) for feature t, and then selects the k features with highest figures of merit. Another frequently used combination method selects the top k/n features for each of n classifiers and then combines these n sets into one global feature set. Classification accuracy often decreases when selecting k common features for a system with n classifiers as opposed to n different sets of size k. But even if it does, the gain in efficiency owing to a common document representation may be worth the loss in accuracy.
Comparison of feature selection methods Mutual information and χ2 represent rather different feature selection methods. The independence of term t and class c can sometimes be rejected with high confidence even if t carries little information about membership of a document in c. This is particularly true for rare terms. If a term occurs once in a large collection and that one occurrence is in the poultry class, then this is statistically significant. But a single occurrence is not very informative according to the information-theoretic definition of information. Because its criterion is significance, χ2 selects more rare terms (which are often less reliable indicators) than mutual information. But the selection criterion of mutual information also does not necessarily select the terms that maximize classification accuracy. Despite the differences between the two methods, the classification accuracy of feature sets selected with χ2 and MI does not seem to differ systematically. In most text classification problems, there are a few strong indicators and many weak indicators. As long as all strong indicators and a large number of weak indicators are selected, accuracy is expected to be good. Both methods do this. Figure 13.8 compares MI and χ2 feature selection for the multinomial model.
13.6 Evaluation of text classification
GREEDY FEATURE SELECTION
Peak effectiveness is virtually the same for both methods. χ2 reaches this peak later, at 300 features, probably because the rare, but highly significant features it selects initially do not cover all documents in the class. However, features selected later (in the range of 100–300) are of better quality than those selected by MI. All three methods – MI, χ2 and frequency based – are greedy methods. They may select features that contribute no incremental information over previously selected features. In Figure 13.7, kong is selected as the seventh term even though it is highly correlated with previously selected hong and therefore redundant. Although such redundancy can negatively impact accuracy, non-greedy methods (see Section 13.7 for references) are rarely used in text classification due to their computational cost. Exercise 13.5 Consider the following frequencies for the class coffee for four terms in the first 100,000 documents of Reuters-RCV1: term brazil council producers roasted
N00 98,012 96,322 98,524 99,824
N01 102 133 119 143
N10 1835 3525 1118 23
N11 51 20 34 10
Select two of these four terms based on (i) χ2 , (ii) mutual information, (iii) frequency.
TWO - CLASS CLASSIFIER
M OD A PTE SPLIT
Evaluation of text classification ] Historically, the classic Reuters-21578 collection was the main benchmark for text classification evaluation. This is a collection of 21,578 newswire articles, originally collected and labeled by Carnegie Group, Inc. and Reuters, Ltd. in the course of developing the CONSTRUE text classification system. It is much smaller than and predates the Reuters-RCV1 collection discussed in Chapter 4 (page 69). The articles are assigned classes from a set of 118 topic categories. A document may be assigned several classes or none, but the commonest case is single assignment (documents with at least one class received an average of 1.24 classes). The standard approach to this any-of problem (Chapter 14, page 306) is to learn 118 two-class classifiers, one for each class, where the two-class classifier for class c is the classifier for the two classes c and its complement c. For each of these classifiers, we can measure recall, precision, and accuracy. In recent work, people almost invariably use the ModApte split, which includes only documents that were viewed and assessed by a human indexer,
◮ Table 13.7 The ten largest classes in the Reuters-21578 collection with number of documents in training and test sets.
class earn acquisitions money-fx grain crude
PERFORMANCE EFFICIENCY
MACROAVERAGING MICROAVERAGING
# train 2877 1650 538 433 389
# testclass 1087 trade 179 interest 179 ship 149 wheat 189 corn
# train 369 347 197 212 182
# test 119 131 89 71 56
and comprises 9,603 training documents and 3,299 test documents. The distribution of documents in classes is very uneven, and some work evaluates systems on only documents in the ten largest classes. They are listed in Table 13.7. A typical document with topics is shown in Figure 13.9. In Section 13.1, we stated as our goal in text classification the minimization of classification error on test data. Classification error is 1.0 minus classification accuracy, the proportion of correct decisions, a measure we introduced in Section 8.3 (page 155). This measure is appropriate if the percentage of documents in the class is high, perhaps 10% to 20% and higher. But as we discussed in Section 8.3, accuracy is not a good measure for "small" classes because always saying no, a strategy that defeats the purpose of building a classifier, will achieve high accuracy. The always-no classifier is 99% accurate for a class with relative frequency 1%. For small classes, precision, recall and F1 are better measures. We will use effectiveness as a generic term for measures that evaluate the quality of classification decisions, including precision, recall, F1 , and accuracy. Performance refers to the computational efficiency of classification and IR systems in this book. However, many researchers mean effectiveness, not efficiency of text classification when they use the term performance. When we process a collection with several two-class classifiers (such as Reuters-21578 with its 118 classes), we often want to compute a single aggregate measure that combines the measures for individual classifiers. There are two methods for doing this. Macroaveraging computes a simple average over classes. Microaveraging pools per-document decisions across classes, and then computes an effectiveness measure on the pooled contingency table. Table 13.8 gives an example. The differences between the two methods can be large. Macroaveraging gives equal weight to each class, whereas microaveraging gives equal weight to each per-document classification decision. Because the F1 measure ignores true negatives and its magnitude is mostly determined by the number of true positives, large classes dominate small classes in microaveraging. In the example, microaveraged precision (0.83) is much closer to the precision of
2-MAR-1987 16:51:43.42 livestockhog AMERICAN PORK CONGRESS KICKS OFF TOMORROW CHICAGO, March 2 - The American Pork Congress kicks off tomorrow, March 3, in Indianapolis with 160 of the nations pork producers from 44 member states determining industry positions on a number of issues, according to the National Pork Producers Council, NPPC. Delegates to the three day Congress will be considering 26 resolutions concerning various issues, including the future direction of farm policy and the tax law as it applies to the agriculture sector. The delegates will also debate whether to endorse concepts of a national PRV (pseudorabies virus) control and eradication program, the NPPC said. A large trade show, in conjunction with the congress, will feature the latest in technology in all areas of the industry, the NPPC added. Reuter \&\#3; ◮ Figure 13.9 A sample document from the Reuters-21578 collection.
c2 (0.9) than to the precision of c1 (0.5) because c2 is five times larger than c1 . Microaveraged results are therefore really a measure of effectiveness on the large classes in a test collection. To get a sense of effectiveness on small classes, you should compute macroaveraged results. In one-of classification (Section 14.5, page 306), microaveraged F1 is the same as accuracy (Exercise 13.6). Table 13.9 gives microaveraged and macroaveraged effectiveness of Naive Bayes for the ModApte split of Reuters-21578. To give a sense of the relative effectiveness of NB, we compare it with linear SVMs (rightmost column; see Chapter 15), one of the most effective classifiers, but also one that is more expensive to train than NB. NB has a microaveraged F1 of 80%, which is 9% less than the SVM (89%), a 10% relative decrease (row "micro-avg-L (90 classes)"). So there is a surprisingly small effectiveness penalty for its simplicity and efficiency. However, on small classes, some of which only have on the order of ten positive examples in the training set, NB does much worse. Its macroaveraged F1 is 13% below the SVM, a 22% relative decrease (row "macro-avg (90 classes)"). The table also compares NB with the other classifiers we cover in this book:
◮ Table 13.8 Macro- and microaveraging. "Truth" is the true class and "call" the decision of the classifier. In this example, macroaveraged precision is [10/(10 + 10) + 90/(10 + 90)] /2 = (0.5 + 0.9)/2 = 0.7. Microaveraged precision is 100/(100 + 20) ≈ 0.83.
call: yes call: no
class 1 truth: truth: yes no 10
pooled table truth: truth: yes no 100
◮ Table 13.9 Text classification effectiveness numbers on Reuters-21578 for F1 (in percent). Results from Li and Yang (2003) (a), Joachims (1998) (b: kNN) and Dumais et al. (1998) (b: NB, Rocchio, trees, SVM).
micro-avg-L (90 classes) macro-avg (90 classes)
NB 80 47
Rocchio 85 59
kNN 86 60
earn acq money-fx grain crude trade interest ship wheat corn micro-avg (top 10) micro-avg-D (118 classes)
NB 96 88 57 79 80 64 65 85 70 65 82 75
Rocchio 93 65 47 68 70 65 63 49 69 48 65 62
kNN 97 92 78 82 86 77 74 79 77 78 82 n/a
SVM 89 60
trees 98 90 66 85 85 73 67 74 93 92 88 n/a
SVM 98 94 75 95 89 76 78 86 92 90 92 87
Rocchio and kNN. In addition, we give numbers for decision trees, an important classification method we do not cover. The bottom part of the table shows that there is considerable variation from class to class. For instance, NB beats kNN on ship, but is much worse on money-fx. Comparing parts (a) and (b) of the table, one is struck by the degree to which the cited papers' results differ. This is partly due to the fact that the numbers in (b) are break-even scores (cf. page 161) averaged over 118 classes, whereas the numbers in (a) are true F1 scores (computed without any know-
DEVELOPMENT SET
HELD - OUT DATA
ledge of the test set) averaged over ninety classes. This is unfortunately typical of what happens when comparing different results in text classification: There are often differences in the experimental setup or the evaluation that complicate the interpretation of the results. These and other results have shown that the average effectiveness of NB is uncompetitive with classifiers like SVMs when trained and tested on independent and identically distributed (i.i.d.) data, that is, uniform data with all the good properties of statistical sampling. However, these differences may often be invisible or even reverse themselves when working in the real world where, usually, the training sample is drawn from a subset of the data to which the classifier will be applied, the nature of the data drifts over time rather than being stationary (the problem of concept drift we mentioned on page 269), and there may well be errors in the data (among other problems). Many practitioners have had the experience of being unable to build a fancy classifier for a certain problem that consistently performs better than NB. Our conclusion from the results in Table 13.9 is that, although most researchers believe that an SVM is better than kNN and kNN better than NB, the ranking of classifiers ultimately depends on the class, the document collection, and the experimental setup. In text classification, there is always more to know than simply which machine learning algorithm was used, as we further discuss in Section 15.3 (page 334). When performing evaluations like the one in Table 13.9, it is important to maintain a strict separation between the training set and the test set. We can easily make correct classification decisions on the test set by using information we have gleaned from the test set, such as the fact that a particular term is a good predictor in the test set (even though this is not the case in the training set). A more subtle example of using knowledge about the test set is to try a large number of values of a parameter (e.g., the number of selected features) and select the value that is best for the test set. As a rule, accuracy on new data – the type of data we will encounter when we use the classifier in an application – will be much lower than accuracy on a test set that the classifier has been tuned for. We discussed the same problem in ad hoc retrieval in Section 8.1 (page 153). In a clean statistical text classification experiment, you should never run any program on or even look at the test set while developing a text classification system. Instead, set aside a development set for testing while you develop your method. When such a set serves the primary purpose of finding a good value for a parameter, for example, the number of selected features, then it is also called held-out data. Train the classifier on the rest of the training set with different parameter values, and then select the value that gives best results on the held-out part of the training set. Ideally, at the very end, when all parameters have been set and the method is fully specified, you run one final experiment on the test set and publish the results. Because no informa-
◮ Table 13.10 Data for parameter estimation exercise.
words in document Taipei Taiwan Macao Taiwan Shanghai Japan Sapporo Sapporo Osaka Taiwan Taiwan Taiwan Sapporo
in c = China? yes yes no no ?
tion about the test set was used in developing the classifier, the results of this experiment should be indicative of actual performance in practice. This ideal often cannot be met; researchers tend to evaluate several systems on the same test set over a period of several years. But it is nevertheless highly important to not look at the test data and to run systems on it as sparingly as possible. Beginners often violate this rule, and their results lose validity because they have implicitly tuned their system to the test data simply by running many variant systems and keeping the tweaks to the system that worked best on the test set.
Assume a situation where every document in the test collection has been assigned exactly one class, and that a classifier also assigns exactly one class to each document. This setup is called one-of classification (Section 14.5, page 306). Show that in one-of classification (i) the total number of false positive decisions equals the total number of false negative decisions and (ii) microaveraged F1 and accuracy are identical. Exercise 13.7 The class priors in Figure 13.2 are computed as the fraction of documents in the class as opposed to the fraction of tokens in the class. Why? Exercise 13.8 The function A PPLY M ULTINOMIAL NB in Figure 13.2 has time complexity Θ ( La + |C | La ). How would you modify the function so that its time complexity is Θ( La + | C | M a )? Exercise 13.9 Based on the data in Table 13.10, (i) estimate a multinomial Naive Bayes classifier, (ii) apply the classifier to the test document, (iii) estimate a Bernoulli NB classifier, (iv) apply the classifier to the test document. You need not estimate parameters that you don't need for classifying the test document. Exercise 13.10 Your task is to classify words as English or not English. Words are generated by a source with the following distribution:
13.6 Evaluation of text classification event 1 2 3 4
word ozb uzu zoo bun
English? no no yes yes
probability 4/9 4/9 1/18 1/18
(i) Compute the parameters (priors and conditionals) of a multinomial NB classifier that uses the letters b, n, o, u, and z as features. Assume a training set that reflects the probability distribution of the source perfectly. Make the same independence assumptions that are usually made for a multinomial classifier that uses terms as features for text classification. Compute parameters using smoothing, in which computed-zero probabilities are smoothed into probability 0.01, and computed-nonzero probabilities are untouched. (This simplistic smoothing may cause P ( A) + P ( A) > 1. Solutions are not required to correct this.) (ii) How does the classifier classify the word zoo? (iii) Classify the word zoo using a multinomial classifier as in part (i), but do not make the assumption of positional independence. That is, estimate separate parameters for each position in a word. You only need to compute the parameters you need for classifying zoo. Exercise 13.11 What are the values of I (Ut ; Cc ) and X2 (D, t, c) if term and class are completely independent? What are the values if they are completely dependent? Exercise 13.12 The feature selection method in Equation (13.16) is most appropriate for the Bernoulli model. Why? How could one modify it for the multinomial model? INFORMATION GAIN
Exercise 13.13 Features can also be selected according toinformation gain (IG), which is defined as: IG(D, t, c) = H ( pD ) −
∑ x ∈{D t + ,D t − }
|x| H ( px ) |D |
where H is entropy, D is the training set, and D t+ , and D t− are the subset of D with term t, and the subset of D without term t, respectively. p A is the class distribution in (sub)collection A, e.g., p A (c) = 0.25, p A (c) = 0.75 if a quarter of the documents in A are in class c. Show that mutual information and information gain are equivalent. Exercise 13.14 Show that the two X2 formulas (Equations (13.18) and (13.19)) are equivalent. Exercise 13.15 In the χ2 example on page 276 we have | N11 − E11 | = | N10 − E10 | = | N01 − E01 | = | N00 − E00 |. Show that this holds in general. Exercise 13.16 χ2 and mutual information do not distinguish between positively and negatively correlated features. Because most good text classification features are positively correlated (i.e., they occur more often in c than in c), one may want to explicitly rule out the selection of negative indicators. How would you do this?
POINTWISE MUTUAL INFORMATION
UTILITY MEASURE
General introductions to statistical classification and machine learning can be found in (Hastie et al. 2001), (Mitchell 1997), and (Duda et al. 2000), including many important methods (e.g., decision trees and boosting) that we do not cover. A comprehensive review of text classification methods and results is (Sebastiani 2002). Manning and Schütze (1999, Chapter 16) give an accessible introduction to text classification with coverage of decision trees, perceptrons and maximum entropy models. More information on the superlinear time complexity of learning methods that are more accurate than Naive Bayes can be found in (Perkins et al. 2003) and (Joachims 2006a). Maron and Kuhns (1960) described one of the first NB text classifiers. Lewis (1998) focuses on the history of NB classification. Bernoulli and multinomial models and their accuracy for different collections are discussed by McCallum and Nigam (1998). Eyheramendy et al. (2003) present additional NB models. Domingos and Pazzani (1997), Friedman (1997), and Hand and Yu (2001) analyze why NB performs well although its probability estimates are poor. The first paper also discusses NB's optimality when the independence assumptions are true of the data. Pavlov et al. (2004) propose a modified document representation that partially addresses the inappropriateness of the independence assumptions. Bennett (2000) attributes the tendency of NB probability estimates to be close to either 0 or 1 to the effect of document length. Ng and Jordan (2001) show that NB is sometimes (although rarely) superior to discriminative methods because it more quickly reaches its optimal error rate. The basic NB model presented in this chapter can be tuned for better effectiveness (Rennie et al. 2003;Kołcz and Yih 2007). The problem of concept drift and other reasons why state-of-the-art classifiers do not always excel in practice are discussed by Forman (2006) and Hand (2006). Early uses of mutual information and χ2 for feature selection in text classification are Lewis and Ringuette (1994) and Schütze et al. (1995), respectively. Yang and Pedersen (1997) review feature selection methods and their impact on classification effectiveness. They find that pointwise mutual information is not competitive with other methods. Yang and Pedersen refer to expected mutual information (Equation (13.16)) as information gain (see Exercise 13.13, page 285). (Snedecor and Cochran 1989) is a good reference for the χ2 test in statistics, including the Yates' correction for continuity for 2 × 2 tables. Dunning (1993) discusses problems of the χ2 test when counts are small. Nongreedy feature selection techniques are described by Hastie et al. (2001). Cohen (1995) discusses the pitfalls of using multiple significance tests and methods to avoid them. Forman (2004) evaluates different methods for feature selection for multiple classifiers. David D. Lewis defines the ModApte split at www.daviddlewis.com/resources/testcollections/reuters215 based on Apté et al. (1994). Lewis (1995) describes utility measures for the
evaluation of text classification systems. Yang and Liu (1999) employ significance tests in the evaluation of text classification methods. Lewis et al. (2004) find that SVMs (Chapter 15) perform better on ReutersRCV1 than kNN and Rocchio (Chapter 14).
CONTIGUITY HYPOTHESIS
Vector space classification
The document representation in Naive Bayes is a sequence of terms or a binary vector he1 , . . . , e|V | i ∈ {0, 1}| V |. In this chapter we adopt a different representation for text classification, the vector space model, developed in Chapter 6. It represents each document as a vector with one real-valued component, usually a tf-idf weight, for each term. Thus, the document space X, the domain of the classification function γ, is R |V | . This chapter introduces a number of classification methods that operate on real-valued vectors. The basic hypothesis in using the vector space model for classification is the contiguity hypothesis. Contiguity hypothesis. Documents in the same class form a contiguous region and regions of different classes do not overlap. There are many classification tasks, in particular the type of text classification that we encountered in Chapter 13, where classes can be distinguished by word patterns. For example, documents in the class China tend to have high values on dimensions like Chinese, Beijing, and Mao whereas documents in the class UK tend to have high values for London, British and Queen. Documents of the two classes therefore form distinct contiguous regions as shown in Figure 14.1 and we can draw boundaries that separate them and classify new documents. How exactly this is done is the topic of this chapter. Whether or not a set of documents is mapped into a contiguous region depends on the particular choices we make for the document representation: type of weighting, stop list etc. To see that the document representation is crucial, consider the two classes written by a group vs. written by a single person. Frequent occurrence of the first person pronoun I is evidence for the single-person class. But that information is likely deleted from the document representation if we use a stop list. If the document representation chosen is unfavorable, the contiguity hypothesis will not hold and successful vector space classification is not possible. The same considerations that led us to prefer weighted representations, in particular length-normalized tf-idf representations, in Chapters 6 and 7 also
14 Vector space classification
⋄ ⋄
⋄
China x
◮ Figure 14.1 Vector space classification into three classes.
apply here. For example, a term with 5 occurrences in a document should get a higher weight than a term with one occurrence, but a weight 5 times larger would give too much emphasis to the term. Unweighted and unnormalized counts should not be used in vector space classification. We introduce two vector space classification methods in this chapter, Rocchio and kNN. Rocchio classification (Section 14.2) divides the vector space into regions centered on centroids or prototypes, one for each class, computed as the center of mass of all documents in the class. Rocchio classification is simple and efficient, but inaccurate if classes are not approximately spheres with similar radii. kNN or k nearest neighbor classification (Section 14.3) assigns the majority class of the k nearest neighbors to a test document. kNN requires no explicit training and can use the unprocessed training set directly in classification. It is less efficient than other classification methods in classifying documents. If the training set is large, then kNN can handle non-spherical and other complex classes better than Rocchio. A large number of text classifiers can be viewed as linear classifiers – classifiers that classify based on a simple linear combination of the features (Section 14.4). Such classifiers partition the space of features into regions separated by linear decision hyperplanes, in a manner to be detailed below. Because of the bias-variance tradeoff (Section 14.6) more complex nonlinear models
14.1 Document representations and measures of relatedness in vector spaces
x2 x3 x4 e
x1′
d tru
x2′ x3′ x4′ dprojected
◮ Figure 14.2 Projections of small areas of the unit sphere preserve distances. Left: A projection of the 2D semicircle to 1D. For the points x1 , x2 , x3 , x4 , x5 at x coordinates −0.9, −0.2, 0, 0.2, 0.9 the distance | x2 x3 | ≈ 0.201 only differs by 0.5% from | x2′ x3′ | = 0.2; but | x1 x3 | /| x1′ x3′ | = dtrue /dprojected ≈ 1.06/0.9 ≈ 1.18 is an example of a large distortion (18%) when projecting a large area. Right: The corresponding projection of the 3D hemisphere to 2D.
are not systematically better than linear models. Nonlinear models have more parameters to fit on a limited amount of training data and are more likely to make mistakes for small and noisy data sets. When applying two-class classifiers to problems with more than two classes, there are one-of tasks – a document must be assigned to exactly one of several mutually exclusive classes – and any-of tasks – a document can be assigned to any number of classes as we will explain in Section 14.5. Two-class classifiers solve any-of problems and can be combined to solve one-of problems.
Document representations and measures of relatedness in vector spaces As in Chapter 6, we represent documents as vectors in R |V | in this chapter. To illustrate properties of document vectors in vector classification, we will render these vectors as points in a plane as in the example in Figure 14.1. In reality, document vectors are length-normalized unit vectors that point to the surface of a hypersphere. We can view the 2D planes in our figures as projections onto a plane of the surface of a (hyper-)sphere as shown in Figure 14.2. Distances on the surface of the sphere and on the projection plane are approximately the same as long as we restrict ourselves to small areas of the surface and choose an appropriate projection (Exercise 14.1).
Decisions of many vector space classifiers are based on a notion of distance, e.g., when computing the nearest neighbors in kNN classification. We will use Euclidean distance in this chapter as the underlying distance measure. We observed earlier (Exercise 6.18, page 131) that there is a direct correspondence between cosine similarity and Euclidean distance for lengthnormalized vectors. In vector space classification, it rarely matters whether the relatedness of two documents is expressed in terms of similarity or distance. However, in addition to documents, centroids or averages of vectors also play an important role in vector space classification. Centroids are not lengthnormalized. For unnormalized vectors, dot product, cosine similarity and Euclidean distance all have different behavior in general (Exercise 14.6). We will be mostly concerned with small local regions when computing the similarity between a document and a centroid, and the smaller the region the more similar the behavior of the three measures is.
? 14.2
DECISION BOUNDARY
R OCCHIO CLASSIFICATION CENTROID
Exercise 14.1 For small areas, distances on the surface of the hypersphere are approximated well by distances on its projection (Figure 14.2) because α ≈ sin α for small angles. For what size angle is the distortion α/ sin(α) (i) 1.01, (ii) 1.05 and (iii) 1.1?
Rocchio classification Figure 14.1 shows three classes, China, UK and Kenya, in a two-dimensional (2D) space. Documents are shown as circles, diamonds and X's. The boundaries in the figure, which we call decision boundaries, are chosen to separate the three classes, but are otherwise arbitrary. To classify a new document, depicted as a star in the figure, we determine the region it occurs in and assign it the class of that region – China in this case. Our task in vector space classification is to devise algorithms that compute good boundaries where "good" means high classification accuracy on data unseen during training. Perhaps the best-known way of computing good class boundaries is Rocchio classification, which uses centroids to define the boundaries. The centroid of a class c is computed as the vector average or center of mass of its members: 1 ~µ(c) = ~v(d) | D c | d∑ ∈D c
where Dc is the set of documents in D whose class is c: Dc = {d : hd, ci ∈ D }. We denote the normalized vector of d by ~v(d) (Equation (6.11), page 122). Three example centroids are shown as solid circles in Figure 14.3. The boundary between two classes in Rocchio classification is the set of points with equal distance from the two centroids. For example, | a1 | = | a2 |,
14.2 Rocchio classification
◮ Figure 14.3 Rocchio classification.
|b1 | = |b2 |, and |c1 | = |c2 | in the figure. This set of points is always a line. The generalization of a line in M-dimensional space is a hyperplane, which we define as the set of points ~x that satisfy: (14.2) NORMAL VECTOR
w ~ T~x = b where w ~ is the M-dimensional normal vector1 of the hyperplane and b is a constant. This definition of hyperplanes includes lines (any line in 2D can be defined by w1 x1 + w2 x2 = b) and 2-dimensional planes (any plane in 3D can be defined by w1 x1 + w2 x2 + w3 x3 = b). A line divides a plane in two, a plane divides 3-dimensional space in two, and hyperplanes divide higherdimensional spaces in two. Thus, the boundaries of class regions in Rocchio classification are hyperplanes. The classification rule in Rocchio is to classify a point in accordance with the region it falls into. Equivalently, we determine the centroid ~µ (c) that the point is closest to and then assign it to c. As an example, consider the star in Figure 14.3. It is located in the China region of the space and Rocchio therefore assigns it to China. We show the Rocchio algorithm in pseudocode in Figure 14.4. 1. Recall from basic linear algebra that ~v · w ~ = ~v T w ~ , i.e., the dot product of ~v and w ~ equals the product by matrix multiplication of the transpose of ~v and w ~.
term weights vector d~1 d~2 d~3 d~4 d~5 ~µc ~µc
0 0 0 0.71 0.71 0 0.71
0 0 1.0 0 0 0.33 0
1.0 0 0 0 0 0.33 0
0 1.0 0 0 0 0.33 0
◮ Table 14.1 Vectors and class centroids for the data in Table 13.1.
Table 14.1 shows the tf-idf vector representations of the five documents in Table 13.1 (page 261), using the formula (1 + log10 tft,d ) log10 (4/dft ) if tft,d > 0 (Equation (6.14), page 127). The two class centroids are µ c = 1/3 · (d~1 + d~2 + d~3 ) and µ c = 1/1 · (d~4 ). The distances of the test document from the centroids are | µ c − d~5 | ≈ 1.15 and | µ c − d~5 | = 0.0. Thus, Rocchio assigns d5 to c. The separating hyperplane in this case has the following parameters:
w ~ b
≈ =
(0 − 0.71 − 0.71 1/3 1/3 1/3) T −1/3
See Exercise 14.15 for how to compute w ~ and b. We can easily verify that this hyperplane separates the documents as desired: w ~ T d~1 ≈ 0 · 0 + −0.71 · 0 + −0.71 · 0 + 1/3 · 0 + 1/3 · 1.0 + 1/3 · 0 = 1/3 > b (and, similarly, w ~ T ~di > b for i = 2 and i = 3) T and w ~ d~4 = −1 < b. Thus, documents in c are above the hyperplane (w ~ T d~ > b) and ~ T d~ < b). documents in c are below the hyperplane (w
The assignment criterion in Figure 14.4 is Euclidean distance (A PPLY R OC line 1). An alternative is cosine similarity:
CHIO,
Assign d to class c = arg max cos(~µ(c′ ), ~v(d)) c′
As discussed in Section 14.1, the two assignment criteria will sometimes make different classification decisions. We present the Euclidean distance variant of Rocchio classification here because it emphasizes Rocchio's close correspondence to K-means clustering (Section 16.4, page 360). Rocchio classification is a form of Rocchio relevance feedback (Section 9.1.1, page 178). The average of the relevant documents, corresponding to the most important component of the Rocchio vector in relevance feedback (Equation (9.3), page 182), is the centroid of the "class" of relevant documents. We omit the query component of the Rocchio formula in Rocchio classification since there is no query in text classification. Rocchio classification can be
T RAIN R OCCHIO (C, D ) 1 for each c j ∈ C 2 do D j ← {d : hd, c j i ∈ D } 3 ~µ j ← | D1 | ∑d∈ D j ~v(d) 4
return {~µ1 , . . . , ~µ J }
A PPLY R OCCHIO ({~µ1 , . . . , ~µ J }, d) 1 return arg min j |~µ j − ~v(d)| ◮ Figure 14.4 Rocchio classification: Training and testing.
aaaa Xa a a a
b b b b
X aa a
Bb b b b b b b b b b b b ◮ Figure 14.5 The multimodal class "a" consists of two different clusters (small upper circles centered on X's). Rocchio classification will misclassify "o" as "a" because it is closer to the centroid A of the "a" class than to the centroid B of the "b" class.
applied to J > 2 classes whereas Rocchio relevance feedback is designed to distinguish only two classes, relevant and nonrelevant. In addition to respecting contiguity, the classes in Rocchio classification must be approximate spheres with similar radii. In Figure 14.3, the solid square just below the boundary between UK and Kenya is a better fit for the class UK since UK is more scattered than Kenya. But Rocchio assigns it to Kenya because it ignores details of the distribution of points in a class and only uses distance from the centroid for classification. The assumption of sphericity also does not hold in Figure 14.5. We can-
◮ Table 14.2 Training and test times for Rocchio classification. Lave is the average number of tokens per document. La and Ma are the numbers of tokens and types, respectively, in the test document. Computing Euclidean distance between the class centroids and a document is Θ (|C | Ma ).
MULTIMODAL CLASS
not represent the "a" class well with a single prototype because it has two clusters. Rocchio often misclassifies this type of multimodal class. A text classification example for multimodality is a country like Burma, which changed its name to Myanmar in 1989. The two clusters before and after the name change need not be close to each other in space. We also encountered the problem of multimodality in relevance feedback (Section 9.1.2, page 184). Two-class classification is another case where classes are rarely distributed like spheres with similar radii. Most two-class classifiers distinguish between a class like China that occupies a small region of the space and its widely scattered complement. Assuming equal radii will result in a large number of false positives. Most two-class classification problems therefore require a modified decision rule of the form: Assign d to class c iff |~µ (c) − ~v(d)| < |~µ (c) − ~v(d)| − b for a positive constant b. As in Rocchio relevance feedback, the centroid of the negative documents is often not used at all, so that the decision criterion simplifies to |~µ (c) − ~v(d)| < b′ for a positive constant b′ . Table 14.2 gives the time complexity of Rocchio classification.2 Adding all documents to their respective (unnormalized) centroid is Θ(|D | Lave ) (as opposed to Θ(|D ||V |)) since we need only consider non-zero entries. Dividing each vector sum by the size of its class to compute the centroid is Θ(|V |). Overall, training time is linear in the size of the collection (cf. Exercise 13.1). Thus, Rocchio classification and Naive Bayes have the same linear training time complexity. In the next section, we will introduce another vector space classification method, kNN, that deals better with classes that have non-spherical, disconnected or other irregular shapes.
[⋆] Exercise 14.2 Show that Rocchio classification can assign a label to a document that is different from its training set label. 2. We write Θ(|D | Lave ) for Θ( T ) and assume that the length of test documents is bounded as we did on page 262.
k nearest neighbor
◮ Figure 14.6 Voronoi tessellation and decision boundaries (double lines) in 1NN classification. The three classes are: X, circle and diamond.
14.3 k NEAREST NEIGHBOR CLASSIFICATION
V ORONOI TESSELLATION
k nearest neighbor Unlike Rocchio, k nearest neighbor or kNN classification determines the decision boundary locally. For 1NN we assign each document to the class of its closest neighbor. For kNN we assign each document to the majority class of its k closest neighbors where k is a parameter. The rationale of kNN classification is that, based on the contiguity hypothesis, we expect a test document d to have the same label as the training documents located in the local region surrounding d. Decision boundaries in 1NN are concatenated segments of the Voronoi tessellation as shown in Figure 14.6. The Voronoi tessellation of a set of objects decomposes space into Voronoi cells, where each object's cell consists of all points that are closer to the object than to other objects. In our case, the objects are documents. The Voronoi tessellation then partitions the plane into |D | convex polygons, each containing its corresponding document (and no other) as shown in Figure 14.6, where a convex polygon is a convex region in 2-dimensional space bounded by lines. For general k ∈ N in kNN, consider the region in the space for which the set of k nearest neighbors is the same. This again is a convex polygon and the space is partitioned into convex polygons, within each of which the set of k
T RAIN - K NN (C, D ) 1 D ′ ← P REPROCESS (D ) 2 k ← S ELECT- K(C, D ′ ) 3 return D ′ , k A PPLY- K NN (C, D ′ , k, d) 1 Sk ← C OMPUTE N EAREST N EIGHBORS (D ′ , k, d) 2 for each c j ∈ C 3 do p j ← |Sk ∩ c j |/k 4 return arg maxj p j ◮ Figure 14.7 kNN training (with preprocessing) and testing. p j is an estimate for P (c j | Sk ) = P (c j | d). c j denotes the set of all documents in the class c j .
nearest neighbors is invariant (Exercise 14.11).3 1NN is not very robust. The classification decision of each test document relies on the class of a single training document, which may be incorrectly labeled or atypical. kNN for k > 1 is more robust. It assigns documents to the majority class of their k closest neighbors, with ties broken randomly. There is a probabilistic version of this kNN classification algorithm. We can estimate the probability of membership in class c as the proportion of the k nearest neighbors in c. Figure 14.6 gives an example for k = 3. Probability estimates for class membership of the star are Pˆ (circle class|star) = 1/3, Pˆ (X class|star) = 2/3, and Pˆ (diamond class|star) = 0. The 3nn estimate ( Pˆ1 (circle class|star) = 1/3) and the 1nn estimate ( Pˆ1 (circle class|star) = 1) differ with 3nn preferring the X class and 1nn preferring the circle class . The parameter k in kNN is often chosen based on experience or knowledge about the classification problem at hand. It is desirable for k to be odd to make ties less likely. k = 3 and k = 5 are common choices, but much larger values between 50 and 100 are also used. An alternative way of setting the parameter is to select the k that gives best results on a held-out portion of the training set. We can also weight the "votes" of the k nearest neighbors by their cosine 3. The generalization of a polygon to higher dimensions is a polytope. A polytope is a region in M-dimensional space bounded by ( M − 1)-dimensional hyperplanes. In M dimensions, the decision boundaries for kNN consist of segments of ( M − 1)-dimensional hyperplanes that form the Voronoi tessellation into convex polytopes for the training set of documents. The decision criterion of assigning a document to the majority class of its k nearest neighbors applies equally to M = 2 (tessellation into polygons) and M > 2 (tessellation into polytopes).
kNN with preprocessing of training set training Θ(|D | Lave ) testing Θ( La + |D | Mave Ma ) = Θ(|D | Mave Ma ) kNN without preprocessing of training set training Θ(1) testing Θ( La + |D | Lave Ma ) = Θ(|D | Lave Ma ) ◮ Table 14.3 Training and test times for kNN classification. Mave is the average size of the vocabulary of documents in the collection.
similarity. In this scheme, a class's score is computed as: score(c, d) =
Ic (d′ ) cos(~v(d′ ), ~v(d))
d ′ ∈ Sk ( d )
where Sk (d) is the set of d's k nearest neighbors and Ic (d′ ) = 1 iff d′ is in class c and 0 otherwise. We then assign the document to the class with the highest score. Weighting by similarities is often more accurate than simple voting. For example, if two classes have the same number of neighbors in the top k, the class with the more similar neighbors wins. Figure 14.7 summarizes the kNN algorithm.
Time complexity and optimality of kNN
The distances of the test document from the four training documents in Table 14.1 are | d~1 − d~5 | = | d~2 − d~5 | = | d~3 − d~5 | ≈ 1.41 and | d~4 − d~5 | = 0.0. d5 's nearest neighbor is therefore d4 and 1NN assigns d5 to d4 's class, c.
Table 14.3 gives the time complexity of kNN. kNN has properties that are quite different from most other classification algorithms. Training a kNN classifier simply consists of determining k and preprocessing documents. In fact, if we preselect a value for k and do not preprocess, then kNN requires no training at all. In practice, we have to perform preprocessing steps like tokenization. It makes more sense to preprocess training documents once as part of the training phase rather than repeatedly every time we classify a new test document. Test time is Θ(|D | Mave Ma ) for kNN. It is linear in the size of the training set as we need to compute the distance of each training document from the test document. Test time is independent of the number of classes J. kNN therefore has a potential advantage for problems with large J. In kNN classification, we do not perform any estimation of parameters as we do in Rocchio classification (centroids) or in Naive Bayes (priors and conditional probabilities). kNN simply memorizes all examples in the training
MEMORY- BASED LEARNING
B AYES ERROR RATE
set and then compares the test document to them. For this reason, kNN is also called memory-based learning or instance-based learning. It is usually desirable to have as much training data as possible in machine learning. But in kNN large training sets come with a severe efficiency penalty in classification. Can kNN testing be made more efficient than Θ(|D | Mave Ma ) or, ignoring the length of documents, more efficient than Θ(|D |)? There are fast kNN algorithms for small dimensionality M (Exercise 14.12). There are also approximations for large M that give error bounds for specific efficiency gains (see Section 14.7). These approximations have not been extensively tested for text classification applications, so it is not clear whether they can achieve much better efficiency than Θ(|D |) without a significant loss of accuracy. The reader may have noticed the similarity between the problem of finding nearest neighbors of a test document and ad hoc retrieval, where we search for the documents with the highest similarity to the query (Section 6.3.2, page 123). In fact, the two problems are both k nearest neighbor problems and only differ in the relative density of (the vector of) the test document in kNN (10s or 100s of non-zero entries) versus the sparseness of (the vector of) the query in ad hoc retrieval (usually fewer than 10 non-zero entries). We introduced the inverted index for efficient ad hoc retrieval in Section 1.1 (page 6). Is the inverted index also the solution for efficient kNN? An inverted index restricts a search to those documents that have at least one term in common with the query. Thus in the context of kNN, the inverted index will be efficient if the test document has no term overlap with a large number of training documents. Whether this is the case depends on the classification problem. If documents are long and no stop list is used, then less time will be saved. But with short documents and a large stop list, an inverted index may well cut the average test time by a factor of 10 or more. The search time in an inverted index is a function of the length of the postings lists of the terms in the query. Postings lists grow sublinearly with the length of the collection since the vocabulary increases according to Heaps' law – if the probability of occurrence of some terms increases, then the probability of occurrence of others must decrease. However, most new terms are infrequent. We therefore take the complexity of inverted index search to be Θ( T ) (as discussed in Section 2.4.2, page 41) and, assuming average document length does not change over time, Θ( T ) = Θ(|D |). As we will see in the next chapter, kNN's effectiveness is close to that of the most accurate learning methods in text classification (Table 15.2, page 334). A measure of the quality of a learning method is its Bayes error rate, the average error rate of classifiers learned by it for a particular problem. kNN is not optimal for problems with a non-zero Bayes error rate – that is, for problems where even the best possible classifier has a non-zero classification error. The error of 1NN is asymptotically (as the training set increases) bounded by
14.4 Linear versus nonlinear classifiers
◮ Figure 14.8 There are an infinite number of hyperplanes that separate two linearly separable classes.
twice the Bayes error rate. That is, if the optimal classifier has an error rate of x, then 1NN has an asymptotic error rate of less than 2x. This is due to the effect of noise – we already saw one example of noise in the form of noisy features in Section 13.5 (page 271), but noise can also take other forms as we will discuss in the next section. Noise affects two components of kNN: the test document and the closest training document. The two sources of noise are additive, so the overall error of 1NN is twice the optimal error rate. For problems with Bayes error rate 0, the error rate of 1NN will approach 0 as the size of the training set increases.
LINEAR CLASSIFIER
Exercise 14.3 Explain why kNN handles multimodal classes better than Rocchio.
Linear versus nonlinear classifiers In this section, we show that the two learning methods Naive Bayes and Rocchio are instances of linear classifiers, the perhaps most important group of text classifiers, and contrast them with nonlinear classifiers. To simplify the discussion, we will only consider two-class classifiers in this section and define a linear classifier as a two-class classifier that decides class membership by comparing a linear combination of the features to a threshold. In two dimensions, a linear classifier is a line. Five examples are shown in Figure 14.8. These lines have the functional form w1 x1 + w2 x2 = b. The classification rule of a linear classifier is to assign a document to c if w1 x1 + w2 x2 > b and to c if w1 x1 + w2 x2 ≤ b. Here, ( x1 , x2 ) T is the two-dimensional vector representation of the document and (w1 , w2 ) T is the parameter vector
A PPLY L INEAR C LASSIFIER (~ w, b, ~x ) 1 score ← ∑iM =1 w i x i 2 if score > b 3 then return 1 4 else return 0 ◮ Figure 14.9 Linear classification algorithm.
that defines (together with b) the decision boundary. An alternative geometric interpretation of a linear classifier is provided in Figure 15.7 (page 343). We can generalize this 2D linear classifier to higher dimensions by defining a hyperplane as we did in Equation (14.2), repeated here as Equation (14.3): w ~ T~x = b
(14.3) DECISION HYPERPLANE
The assignment criterion then is: assign to c if w ~ T~x > b and to c if w ~ T~x ≤ b. We call a hyperplane that we use as a linear classifier a decision hyperplane. The corresponding algorithm for linear classification in M dimensions is shown in Figure 14.9. Linear classification at first seems trivial given the simplicity of this algorithm. However, the difficulty is in training the linear classifier, that is, in determining the parameters w ~ and b based on the training set. In general, some learning methods compute much better parameters than others where our criterion for evaluating the quality of a learning method is the effectiveness of the learned linear classifier on new data. We now show that Rocchio and Naive Bayes are linear classifiers. To see this for Rocchio, observe that a vector ~x is on the decision boundary if it has equal distance to the two class centroids:
|~µ(c1 ) − ~x| = |~µ (c2 ) − ~x | Some basic arithmetic shows that this corresponds to a linear classifier with normal vector w ~ = ~µ(c1 ) − ~µ(c2 ) and b = 0.5 ∗ (|~µ(c1 )|2 − |~µ (c2 )|2 ) (Exercise 14.15). We can derive the linearity of Naive Bayes from its decision rule, which chooses the category c with the largest Pˆ (c|d) (Figure 13.2, page 260) where: Pˆ (c|d) ∝ Pˆ (c)
Pˆ (tk |c)
and nd is the number of tokens in the document that are part of the vocabu¯ we obtain for the log odds: lary. Denoting the complement category as c, (14.5)
Pˆ (c) Pˆ (tk |c) Pˆ (c|d) = log + ∑ log Pˆ (c¯|d) Pˆ (c¯) 1≤k≤nd Pˆ (tk |c¯)
ti prime rate interest rates discount bundesbank
wi 0.70 0.67 0.63 0.60 0.46 0.43
d1i 0 1 0 0 1 0
ti dlrs world sees year group dlr
wi -0.71 -0.35 -0.33 -0.25 -0.24 -0.24
◮ Table 14.4 A linear classifier. The dimensions ti and parameters wi of a linear classifier for the class interest (as in interest rate) in Reuters-21578. The threshold is b = 0. Terms like dlr and world have negative weights because they are indicators for the competing class currency.
We choose class c if the odds are greater than 1 or, equivalently, if the log odds are greater than 0. It is easy to see that Equation (14.5) is an instance of Equation (14.3) for wi = log[ Pˆ (ti |c)/ Pˆ (ti |c¯)], x i = number of occurrences of ti in d, and b = − log[ Pˆ (c)/ Pˆ (c¯)]. Here, the index i, 1 ≤ i ≤ M, refers to terms of the vocabulary (not to positions in d as k does; cf. Section 13.4.1, page 270) and ~x and w ~ are M-dimensional vectors. So in log space, Naive Bayes is a linear classifier.
CLASS BOUNDARY
NOISE DOCUMENT
Table 14.4 defines a linear classifier for the category interest in Reuters-21578 (see Section 13.6, page 279). We assign document d~1 "rate discount dlrs world" to interest since w ~ T d~1 = 0.67 · 1 + 0.46 · 1 + (−0.71) · 1 + (−0.35) · 1 = 0.07 > 0 = b. We assign d~2 "prime dlrs" to the complement class (not in interest) since w ~ T d~2 = −0.01 ≤ b. For simplicity, we assume a simple binary vector representation in this example: 1 for occurring terms, 0 for non-occurring terms.
Figure 14.10 is a graphical example of a linear problem, which we define to mean that the underlying distributions P(d|c) and P(d|c) of the two classes are separated by a line. We call this separating line the class boundary. It is the "true" boundary of the two classes and we distinguish it from the decision boundary that the learning method computes to approximate the class boundary. As is typical in text classification, there are some noise documents in Figure 14.10 (marked with arrows) that do not fit well into the overall distribution of the classes. In Section 13.5 (page 271), we defined a noise feature as a misleading feature that, when included in the document representation, on average increases the classification error. Analogously, a noise document is a document that, when included in the training set, misleads the learning method and increases classification error. Intuitively, the underlying distribution partitions the representation space into areas with mostly ho-
◮ Figure 14.10 A linear problem with noise. In this hypothetical web page classification scenario, Chinese-only web pages are solid circles and mixed Chinese-English web pages are squares. The two classes are separated by a linear class boundary (dashed line, short dashes), except for three noise documents (marked with arrows).
LINEAR SEPARABILITY
mogeneous class assignments. A document that does not conform with the dominant class in its area is a noise document. Noise documents are one reason why training a linear classifier is hard. If we pay too much attention to noise documents when choosing the decision hyperplane of the classifier, then it will be inaccurate on new data. More fundamentally, it is usually difficult to determine which documents are noise documents and therefore potentially misleading. If there exists a hyperplane that perfectly separates the two classes, then we call the two classes linearly separable. In fact, if linear separability holds, then there is an infinite number of linear separators (Exercise 14.4) as illustrated by Figure 14.8, where the number of possible separating hyperplanes is infinite. Figure 14.8 illustrates another challenge in training a linear classifier. If we are dealing with a linearly separable problem, then we need a criterion for selecting among all decision hyperplanes that perfectly separate the training data. In general, some of these hyperplanes will do well on new data, some
◮ Figure 14.11 A nonlinear problem.
NONLINEAR CLASSIFIER
will not. An example of a nonlinear classifier is kNN. The nonlinearity of kNN is intuitively clear when looking at examples like Figure 14.6. The decision boundaries of kNN (the double lines in Figure 14.6) are locally linear segments, but in general have a complex shape that is not equivalent to a line in 2D or a hyperplane in higher dimensions. Figure 14.11 is another example of a nonlinear problem: there is no good linear separator between the distributions P(d|c) and P(d|c) because of the circular "enclave" in the upper left part of the graph. Linear classifiers misclassify the enclave, whereas a nonlinear classifier like kNN will be highly accurate for this type of problem if the training set is large enough. If a problem is nonlinear and its class boundaries cannot be approximated well with linear hyperplanes, then nonlinear classifiers are often more accurate than linear classifiers. If a problem is linear, it is best to use a simpler linear classifier.
Exercise 14.4 Prove that the number of linear separators of two classes is either infinite or zero.
ANY- OF CLASSIFICATION
Classification with more than two classes We can extend two-class linear classifiers to J > 2 classes. The method to use depends on whether the classes are mutually exclusive or not. Classification for classes that are not mutually exclusive is called any-of , multilabel, or multivalue classification. In this case, a document can belong to several classes simultaneously, or to a single class, or to none of the classes. A decision on one class leaves all options open for the others. It is sometimes said that the classes are independent of each other, but this is misleading since the classes are rarely statistically independent in the sense defined on page 275. In terms of the formal definition of the classification problem in Equation (13.1) (page 256), we learn J different classifiers γ j in any-of classification, each returning either c j or c j : γ j (d) ∈ {c j , c j }. Solving an any-of classification task with linear classifiers is straightforward: 1. Build a classifier for each class, where the training set consists of the set of documents in the class (positive labels) and its complement (negative labels). 2. Given the test document, apply each classifier separately. The decision of one classifier has no influence on the decisions of the other classifiers.
ONE - OF CLASSIFICATION
The second type of classification with more than two classes is one-of classification. Here, the classes are mutually exclusive. Each document must belong to exactly one of the classes. One-of classification is also called multinomial, polytomous4 , multiclass, or single-label classification. Formally, there is a single classification function γ in one-of classification whose range is C, i.e., γ(d) ∈ {c1 , . . . , c J }. kNN is a (nonlinear) one-of classifier. True one-of problems are less common in text classification than any-of problems. With classes like UK, China, poultry, or coffee, a document can be relevant to many topics simultaneously – as when the prime minister of the UK visits China to talk about the coffee and poultry trade. Nevertheless, we will often make a one-of assumption, as we did in Figure 14.1, even if classes are not really mutually exclusive. For the classification problem of identifying the language of a document, the one-of assumption is a good approximation as most text is written in only one language. In such cases, imposing a one-of constraint can increase the classifier's effectiveness because errors that are due to the fact that the any-of classifiers assigned a document to either no class or more than one class are eliminated. J hyperplanes do not divide R |V | into J distinct regions as illustrated in Figure 14.12. Thus, we must use a combination method when using twoclass linear classifiers for one-of classification. The simplest method is to 4. A synonym of polytomous is polychotomous.
14.5 Classification with more than two classes
J hyperplanes do not divide space into J disjoint regions.
rank classes and then select the top-ranked class. Geometrically, the ranking can be with respect to the distances from the J linear separators. Documents close to a class's separator are more likely to be misclassified, so the greater the distance from the separator, the more plausible it is that a positive classification decision is correct. Alternatively, we can use a direct measure of confidence to rank classes, e.g., probability of class membership. We can state this algorithm for one-of classification with linear classifiers as follows: 1. Build a classifier for each class, where the training set consists of the set of documents in the class (positive labels) and its complement (negative labels). 2. Given the test document, apply each classifier separately. 3. Assign the document to the class with • the maximum score, • the maximum confidence value, • or the maximum probability. CONFUSION MATRIX
An important tool for analyzing the performance of a classifier for J > 2 classes is the confusion matrix. The confusion matrix shows for each pair of classes hc1 , c2 i, how many documents from c1 were incorrectly assigned to c2 . In Table 14.5, the classifier manages to distinguish the three financial classes money-fx, trade, and interest from the three agricultural classes wheat, corn, and grain, but makes many errors within these two groups. The confusion matrix can help pinpoint opportunities for improving the accuracy of the
assigned class true class money-fx trade interest wheat corn grain
money-fx
95 1 13 0 1 0
◮ Table 14.5 A confusion matrix for Reuters-21578. For example, 14 documents from grain were incorrectly assigned to wheat. Adapted from Picca et al. (2006).
system. For example, to address the second largest error in Table 14.5 (14 in the row grain), one could attempt to introduce features that distinguish wheat documents from grain documents.
? ✄
Exercise 14.5 Create a training set of 300 documents, 100 each from three different languages (e.g., English, French, Spanish). Create a test set by the same procedure, but also add 100 documents from a fourth language. Train (i) a one-of classifier (ii) an any-of classifier on this training set and evaluate it on the test set. (iii) Are there any interesting differences in how the two classifiers behave on this task?
The bias-variance tradeoff Nonlinear classifiers are more powerful than linear classifiers. For some problems, there exists a nonlinear classifier with zero classification error, but no such linear classifier. Does that mean that we should always use nonlinear classifiers for optimal effectiveness in statistical text classification? To answer this question, we introduce the bias-variance tradeoff in this section, one of the most important concepts in machine learning. The tradeoff helps explain why there is no universally optimal learning method. Selecting an appropriate learning method is therefore an unavoidable part of solving a text classification problem. Throughout this section, we use linear and nonlinear classifiers as prototypical examples of "less powerful" and "more powerful" learning, respectively. This is a simplification for a number of reasons. First, many nonlinear models subsume linear models as a special case. For instance, a nonlinear learning method like kNN will in some cases produce a linear classifier. Second, there are nonlinear models that are less complex than linear models. For instance, a quadratic polynomial with two parameters is less powerful than a 10,000-dimensional linear classifier. Third, the complexity of learning is not really a property of the classifier because there are many aspects
14.6 The bias-variance tradeoff
of learning (such as feature selection, cf. (Section 13.5, page 271), regularization, and constraints such as margin maximization in Chapter 15) that make a learning method either more powerful or less powerful without affecting the type of classifier that is the final result of learning – regardless of whether that classifier is linear or nonlinear. We refer the reader to the publications listed in Section 14.7 for a treatment of the bias-variance tradeoff that takes into account these complexities. In this section, linear and nonlinear classifiers will simply serve as proxies for weaker and stronger learning methods in text classification. We first need to state our objective in text classification more precisely. In Section 13.1 (page 256), we said that we want to minimize classification error on the test set. The implicit assumption was that training documents and test documents are generated according to the same underlying distribution. We will denote this distribution P(hd, ci) where d is the document and c its label or class. Figures 13.4 and 13.5 were examples of generative models that decompose P(hd, ci) into the product of P(c) and P(d|c). Figures 14.10 and 14.11 depict generative models for hd, ci with d ∈ R2 and c ∈ {square, solid circle}. In this section, instead of using the number of correctly classified test documents (or, equivalently, the error rate on test documents) as evaluation measure, we adopt an evaluation measure that addresses the inherent uncertainty of labeling. In many text classification problems, a given document representation can arise from documents belonging to different classes. This is because documents from different classes can be mapped to the same document representation. For example, the one-sentence documents China sues France and France sues China are mapped to the same document representation d′ = {China, France, sues} in a bag of words model. But only the latter document is relevant to the class c′ = legal actions brought by France (which might be defined, for example, as a standing query by an international trade lawyer). To simplify the calculations in this section, we do not count the number of errors on the test set when evaluating a classifier, but instead look at how well the classifier estimates the conditional probability P(c|d) of a document being in a class. In the above example, we might have P(c′ |d′ ) = 0.5. Our goal in text classification then is to find a classifier γ such that, averaged over documents d, γ(d) is as close as possible to the true probability P(c|d). We measure this using mean squared error: (14.6)
MSE(γ) = Ed [γ(d) − P(c|d)]2 where Ed is the expectation with respect to P(d). The mean squared error term gives partial credit for decisions by γ that are close if not completely right.
E [ x − α ]2
ED Ed [ΓD (d) − P(c|d)]2
= Ex2 − 2Exα + α2 = ( Ex )2 − 2Exα + α2 + Ex2 − 2( Ex )2 + ( Ex )2 = [ Ex − α]2 + Ex2 − E2x ( Ex ) + E( Ex )2 = [ Ex − α]2 + E[ x − Ex ]2 = =
Ed ED [ΓD (d) − P(c|d)]2 Ed [ [ ED ΓD (d) − P(c|d)]2
+ ED [ΓD (d) − ED ΓD (d)]2 ] ◮ Figure 14.13 Arithmetic transformations for the bias-variance decomposition. For the derivation of Equation (14.9), we set α = P (c| d) and x = ΓD (d) in Equation (14.8).
LEARNING ERROR
OPTIMAL LEARNING METHOD
We define a classifier γ to be optimal for a distribution P(hd, ci) if it minimizes MSE(γ). Minimizing MSE is a desideratum for classifiers. We also need a criterion for learning methods. Recall that we defined a learning method Γ as a function that takes a labeled training set D as input and returns a classifier γ. For learning methods, we adopt as our goal to find a Γ that, averaged over training sets, learns classifiers γ with minimal MSE. We can formalize this as minimizing learning error: learning-error(Γ) = ED [MSE(Γ(D ))] where ED is the expectation over labeled training sets. To keep things simple, we can assume that training sets have a fixed size – the distribution P(hd, ci) then defines a distribution P(D ) over training sets. We can use learning error as a criterion for selecting a learning method in statistical text classification. A learning method Γ is optimal for a distribution P(D ) if it minimizes the learning error. Writing ΓD for Γ(D ) for better readability, we can transform Equation (14.7) as follows: learning-error(Γ)
ED [MSE(ΓD )]
Ed [bias(Γ, d) + variance(Γ, d)]
bias(Γ, d) variance(Γ, d)
= [ P(c|d) − ED ΓD (d)]2 = ED [ΓD (d) − ED ΓD (d)]2
where the equivalence between Equations (14.10) and (14.11) is shown in Equation (14.9) in Figure 14.13. Note that d and D are independent of each other. In general, for a random document d and a random training set D, D does not contain a labeled instance of d. Bias is the squared difference between P(c|d), the true conditional probability of d being in c, and ΓD (d), the prediction of the learned classifier, averaged over training sets. Bias is large if the learning method produces classifiers that are consistently wrong. Bias is small if (i) the classifiers are consistently right or (ii) different training sets cause errors on different documents or (iii) different training sets cause positive and negative errors on the same documents, but that average out to close to 0. If one of these three conditions holds, then ED ΓD (d), the expectation over all training sets, is close to P ( c | d ). Linear methods like Rocchio and Naive Bayes have a high bias for nonlinear problems because they can only model one type of class boundary, a linear hyperplane. If the generative model P(hd, ci) has a complex nonlinear class boundary, the bias term in Equation (14.11) will be high because a large number of points will be consistently misclassified. For example, the circular enclave in Figure 14.11 does not fit a linear model and will be misclassified consistently by linear classifiers. We can think of bias as resulting from our domain knowledge (or lack thereof) that we build into the classifier. If we know that the true boundary between the two classes is linear, then a learning method that produces linear classifiers is more likely to succeed than a nonlinear method. But if the true class boundary is not linear and we incorrectly bias the classifier to be linear, then classification accuracy will be low on average. Nonlinear methods like kNN have low bias. We can see in Figure 14.6 that the decision boundaries of kNN are variable – depending on the distribution of documents in the training set, learned decision boundaries can vary greatly. As a result, each document has a chance of being classified correctly for some training sets. The average prediction ED ΓD (d) is therefore closer to P(c|d) and bias is smaller than for a linear learning method. Variance is the variation of the prediction of learned classifiers: the average squared difference between ΓD (d) and its average ED ΓD (d). Variance is large if different training sets D give rise to very different classifiers ΓD . It is small if the training set has a minor effect on the classification decisions ΓD makes, be they correct or incorrect. Variance measures how inconsistent the decisions are, not whether they are correct or incorrect. Linear learning methods have low variance because most randomly drawn training sets produce similar decision hyperplanes. The decision lines pro-
BIAS - VARIANCE TRADEOFF
duced by linear learning methods in Figures 14.10 and 14.11 will deviate slightly from the main class boundaries, depending on the training set, but the class assignment for the vast majority of documents (with the exception of those close to the main boundary) will not be affected. The circular enclave in Figure 14.11 will be consistently misclassified. Nonlinear methods like kNN have high variance. It is apparent from Figure 14.6 that kNN can model very complex boundaries between two classes. It is therefore sensitive to noise documents of the sort depicted in Figure 14.10. As a result the variance term in Equation (14.11) is large for kNN: Test documents are sometimes misclassified – if they happen to be close to a noise document in the training set – and sometimes correctly classified – if there are no noise documents in the training set near them. This results in high variation from training set to training set. High-variance learning methods are prone to overfitting the training data. The goal in classification is to fit the training data to the extent that we capture true properties of the underlying distribution P(hd, ci). In overfitting, the learning method also learns from noise. Overfitting increases MSE and frequently is a problem for high-variance learning methods. We can also think of variance as the model complexity or, equivalently, memory capacity of the learning method – how detailed a characterization of the training set it can remember and then apply to new data. This capacity corresponds to the number of independent parameters available to fit the training set. Each kNN neighborhood Sk makes an independent classification decision. The parameter in this case is the estimate Pˆ (c|Sk ) from Figure 14.7. Thus, kNN's capacity is only limited by the size of the training set. It can memorize arbitrarily large training sets. In contrast, the number of parameters of Rocchio is fixed – J parameters per dimension, one for each centroid – and independent of the size of the training set. The Rocchio classifier (in form of the centroids defining it) cannot "remember" fine-grained details of the distribution of the documents in the training set. According to Equation (14.7), our goal in selecting a learning method is to minimize learning error. The fundamental insight captured by Equation (14.11), which we can succinctly state as: learning-error = bias + variance, is that the learning error has two components, bias and variance, which in general cannot be minimized simultaneously. When comparing two learning methods Γ1 and Γ2 , in most cases the comparison comes down to one method having higher bias and lower variance and the other lower bias and higher variance. The decision for one learning method vs. another is then not simply a matter of selecting the one that reliably produces good classifiers across training sets (small variance) or the one that can learn classification problems with very difficult decision boundaries (small bias). Instead, we have to weigh the respective merits of bias and variance in our application and choose accordingly. This tradeoff is called the bias-variance tradeoff .
Figure 14.10 provides an illustration, which is somewhat contrived, but will be useful as an example for the tradeoff. Some Chinese text contains English words written in the Roman alphabet like CPU, ONLINE, and GPS. Consider the task of distinguishing Chinese-only web pages from mixed Chinese-English web pages. A search engine might offer Chinese users without knowledge of English (but who understand loanwords like CPU) the option of filtering out mixed pages. We use two features for this classification task: number of Roman alphabet characters and number of Chinese characters on the web page. As stated earlier, the distribution P(hd, ci) of the generative model generates most mixed (respectively, Chinese) documents above (respectively, below) the short-dashed line, but there are a few noise documents. In Figure 14.10, we see three classifiers: • One-feature classifier. Shown as a dotted horizontal line. This classifier uses only one feature, the number of Roman alphabet characters. Assuming a learning method that minimizes the number of misclassifications in the training set, the position of the horizontal decision boundary is not greatly affected by differences in the training set (e.g., noise documents). So a learning method producing this type of classifier has low variance. But its bias is high since it will consistently misclassify squares in the lower left corner and "solid circle" documents with more than 50 Roman characters. • Linear classifier. Shown as a dashed line with long dashes. Learning linear classifiers has less bias since only noise documents and possibly a few documents close to the boundary between the two classes are misclassified. The variance is higher than for the one-feature classifiers, but still small: The dashed line with long dashes deviates only slightly from the true boundary between the two classes, and so will almost all linear decision boundaries learned from training sets. Thus, very few documents (documents close to the class boundary) will be inconsistently classified. • "Fit-training-set-perfectly" classifier. Shown as a solid line. Here, the learning method constructs a decision boundary that perfectly separates the classes in the training set. This method has the lowest bias because there is no document that is consistently misclassified – the classifiers sometimes even get noise documents in the test set right. But the variance of this learning method is high. Because noise documents can move the decision boundary arbitrarily, test documents close to noise documents in the training set will be misclassified – something that a linear learning method is unlikely to do. It is perhaps surprising that so many of the best-known text classification algorithms are linear. Some of these methods, in particular linear SVMs, reg-
ularized logistic regression and regularized linear regression, are among the most effective known methods. The bias-variance tradeoff provides insight into their success. Typical classes in text classification are complex and seem unlikely to be modeled well linearly. However, this intuition is misleading for the high-dimensional spaces that we typically encounter in text applications. With increased dimensionality, the likelihood of linear separability increases rapidly (Exercise 14.17). Thus, linear models in high-dimensional spaces are quite powerful despite their linearity. Even more powerful nonlinear learning methods can model decision boundaries that are more complex than a hyperplane, but they are also more sensitive to noise in the training data. Nonlinear learning methods sometimes perform better if the training set is large, but by no means in all cases.
PUSH MODEL PULL MODEL
CENTROID - BASED CLASSIFICATION
References and further reading As discussed in Chapter 9, Rocchio relevance feedback is due to Rocchio (1971). Joachims (1997) presents a probabilistic analysis of the method. Rocchio classification was widely used as a classification method in TREC in the 1990s (Buckley et al. 1994a;b, Voorhees and Harman 2005). Initially, it was used as a form of routing. Routing merely ranks documents according to relevance to a class without assigning them. Early work on filtering, a true classification approach that makes an assignment decision on each document, was published by Ittner et al. (1995) and Schapire et al. (1998). The definition of routing we use here should not be confused with another sense. Routing can also refer to the electronic distribution of documents to subscribers, the so-called push model of document distribution. In a pull model, each transfer of a document to the user is initiated by the user – for example, by means of search or by selecting it from a list of documents on a news aggregation website. Some authors restrict the name Roccchio classification to two-class problems and use the terms cluster-based (Iwayama and Tokunaga 1995) and centroidbased classification (Han and Karypis 2000, Tan and Cheng 2007) for Rocchio classification with J > 2. A more detailed treatment of kNN can be found in (Hastie et al. 2001), including methods for tuning the parameter k. An example of an approximate fast kNN algorithm is locality-based hashing (Andoni et al. 2006). Kleinberg (1997) presents an approximate Θ(( M log2 M )( M + log N )) kNN algorithm (where M is the dimensionality of the space and N the number of data points), but at the cost of exponential storage requirements: Θ(( N log M )2M ). Indyk (2004) surveys nearest neighbor methods in high-dimensional spaces. Early work on kNN in text classification was motivated by the availability of massively parallel hardware architectures (Creecy et al. 1992). Yang (1994)
uses an inverted index to speed up kNN classification. The optimality result for 1NN (twice the Bayes error rate asymptotically) is due to Cover and Hart (1967). The effectiveness of Rocchio classification and kNN is highly dependent on careful parameter tuning (in particular, the parameters b′ for Rocchio on page 296 and k for kNN), feature engineering (Section 15.3, page 334) and feature selection (Section 13.5, page 271). Buckley and Salton (1995), Schapire et al. (1998), Yang and Kisiel (2003) and Moschitti (2003) address these issues for Rocchio and Yang (2001) and Ault and Yang (2002) for kNN. Zavrel et al. (2000) compare feature selection methods for kNN. The bias-variance tradeoff was introduced by Geman et al. (1992). The derivation in Section 14.6 is for MSE(γ), but the tradeoff applies to many loss functions (cf. Friedman (1997), Domingos (2000)). Schütze et al. (1995) and Lewis et al. (1996) discuss linear classifiers for text and Hastie et al. (2001) linear classifiers in general. Readers interested in the algorithms mentioned, but not described in this chapter may wish to consult Bishop (2006) for neural networks, Hastie et al. (2001) for linear and logistic regression, and Minsky and Papert (1988) for the perceptron algorithm. Anagnostopoulos et al. (2006) show that an inverted index can be used for highly efficient document classification with any linear classifier, provided that the classifier is still effective when trained on a modest number of features via feature selection. We have only presented the simplest method for combining two-class classifiers into a one-of classifier. Another important method is the use of errorcorrecting codes, where a vector of decisions of different two-class classifiers is constructed for each document. A test document's decision vector is then "corrected" based on the distribution of decision vectors in the training set, a procedure that incorporates information from all two-class classifiers and their correlations into the final classification decision (Dietterich and Bakiri 1995). Ghamrawi and McCallum (2005) also exploit dependencies between classes in any-of classification. Allwein et al. (2000) propose a general framework for combining two-class classifiers.
Exercises Exercise 14.6 In Figure 14.14, which of the three vectors ~a, ~b, and ~c is (i) most similar to ~x according to dot product similarity, (ii) most similar to ~x according to cosine similarity, (iii) closest to ~x according to Euclidean distance? Exercise 14.7 Download Reuters-21578 and train and test Rocchio and kNN classifiers for the classes acquisitions, corn, crude, earn, grain, interest, money-fx, ship, trade, and wheat. Use the ModApte split. You may want to use one of a number of software packages that im-
c b a
0 1 2 3 4 5 6 7 8 ◮ Figure 14.14 Example for differences between Euclidean distance, dot product similarity and cosine similarity. The vectors are ~a = (0.5 1.5) T , ~x = (2 2) T , ~b = (4 4) T , and ~c = (8 6) T .
plement Rocchio classification and kNN classification, for example, the Bow toolkit (McCallum 1996). Exercise 14.8 Download 20 Newgroups (page 154) and train and test Rocchio and kNN classifiers for its 20 classes. Exercise 14.9 Show that the decision boundaries in Rocchio classification are, as in kNN, given by the Voronoi tessellation. Exercise 14.10
Computing the distance between a dense centroid and a sparse vector is Θ ( M ) for a naive implementation that iterates over all M dimensions. Based on the equality ∑( xi − µ i )2 = 1.0 + ∑ µ2i − 2 ∑ xi µ i and assuming that ∑ µ2i has been precomputed, write down an algorithm that is Θ ( Ma ) instead, where Ma is the number of distinct terms in the test document. Exercise 14.11
[⋆ ⋆ ⋆]
Prove that the region of the plane consisting of all points with the same k nearest neighbors is a convex polygon. Exercise 14.12 Design an algorithm that performs an efficient 1NN search in 1 dimension (where efficiency is with respect to the number of documents N). What is the time complexity of the algorithm? Exercise 14.13
Design an algorithm that performs an efficient 1NN search in 2 dimensions with at most polynomial (in N) preprocessing time.
b b ◮ Figure 14.15 A simple non-separable set of points.
Exercise 14.14 [ ⋆ ⋆ ⋆] Can one design an exact efficient algorithm for 1NN for very large M along the ideas you used to solve the last exercise? Exercise 14.15 Show that Equation (14.4) defines a hyperplane with w ~ = ~µ(c1 ) − ~µ (c2 ) and b = 0.5 ∗ (|~µ(c1 )|2 − |~µ(c2 )|2 ). Exercise 14.16 We can easily construct non-separable data sets in high dimensions by embedding a non-separable set like the one shown in Figure 14.15. Consider embedding Figure 14.15 in 3D and then perturbing the 4 points slightly (i.e., moving them a small distance in a random direction). Why would you expect the resulting configuration to be linearly separable? How likely is then a non-separable set of m ≪ M points in M-dimensional space? Exercise 14.17 Assuming two classes, show that the percentage of non-separable assignments of the vertices of a hypercube decreases with dimensionality M for M > 1. For example, for M = 1 the proportion of non-separable assignments is 0, for M = 2, it is 2/16. One of the two non-separable cases for M = 2 is shown in Figure 14.15, the other is its mirror image. Solve the exercise either analytically or by simulation. Exercise 14.18 Although we point out the similarities of Naive Bayes with linear vector space classifiers, it does not make sense to represent count vectors (the document representations in NB) in a continuous vector space. There is however a formalization of NB that is analogous to Rocchio. Show that NB assigns a document to the class (represented as a parameter vector) whose Kullback-Leibler (KL) divergence (Section 12.4, page 251) to the document (represented as a count vector as in Section 13.4.1 (page 270), normalized to sum to 1) is smallest.
Support vector machines and machine learning on documents
Improving classifier effectiveness has been an area of intensive machinelearning research over the last two decades, and this work has led to a new generation of state-of-the-art classifiers, such as support vector machines, boosted decision trees, regularized logistic regression, neural networks, and random forests. Many of these methods, including support vector machines (SVMs), the main topic of this chapter, have been applied with success to information retrieval problems, particularly text classification. An SVM is a kind of large-margin classifier: it is a vector space based machine learning method where the goal is to find a decision boundary between two classes that is maximally far from any point in the training data (possibly discounting some points as outliers or noise). We will initially motivate and develop SVMs for the case of two-class data sets that are separable by a linear classifier (Section 15.1), and then extend the model in Section 15.2 to non-separable data, multi-class problems, and nonlinear models, and also present some additional discussion of SVM performance. The chapter then moves to consider the practical deployment of text classifiers in Section 15.3: what sorts of classifiers are appropriate when, and how can you exploit domain-specific text features in classification? Finally, we will consider how the machine learning technology that we have been building for text classification can be applied back to the problem of learning how to rank documents in ad hoc retrieval (Section 15.4). While several machine learning methods have been applied to this task, use of SVMs has been prominent. Support vector machines are not necessarily better than other machine learning methods (except perhaps in situations with little training data), but they perform at the state-of-the-art level and have much current theoretical and empirical appeal.
15 Support vector machines and machine learning on documents
Maximum margin decision hyperplane
Support vectors
t u
t u t u
Margin is maximized
◮ Figure 15.1 The support vectors are the 5 points right up against the margin of the classifier.
SUPPORT VECTOR
Support vector machines: The linearly separable case For two-class, separable training data sets, such as the one in Figure 14.8 (page 301), there are lots of possible linear separators. Intuitively, a decision boundary drawn in the middle of the void between data items of the two classes seems better than one which approaches very close to examples of one or both classes. While some learning methods such as the perceptron algorithm (see references in Section 14.7, page 314) find just any linear separator, others, like Naive Bayes, search for the best linear separator according to some criterion. The SVM in particular defines the criterion to be looking for a decision surface that is maximally far away from any data point. This distance from the decision surface to the closest data point determines the margin of the classifier. This method of construction necessarily means that the decision function for an SVM is fully specified by a (usually small) subset of the data which defines the position of the separator. These points are referred to as the support vectors (in a vector space, a point can be thought of as a vector between the origin and that point). Figure 15.1 shows the margin and support vectors for a sample problem. Other data points play no part in determining the decision surface that is chosen.
15.1 Support vector machines: The linearly separable case
◮ Figure 15.2 An intuition for large-margin classification. Insisting on a large margin reduces the capacity of the model: the range of angles at which the fat decision surface can be placed is smaller than for a decision hyperplane (cf. Figure 14.8, page 301).
Maximizing the margin seems good because points near the decision surface represent very uncertain classification decisions: there is almost a 50% chance of the classifier deciding either way. A classifier with a large margin makes no low certainty classification decisions. This gives you a classification safety margin: a slight error in measurement or a slight document variation will not cause a misclassification. Another intuition motivating SVMs is shown in Figure 15.2. By construction, an SVM classifier insists on a large margin around the decision boundary. Compared to a decision hyperplane, if you have to place a fat separator between classes, you have fewer choices of where it can be put. As a result of this, the memory capacity of the model has been decreased, and hence we expect that its ability to correctly generalize to test data is increased (cf. the discussion of the bias-variance tradeoff in Chapter 14, page 312). Let us formalize an SVM with algebra. A decision hyperplane (page 302) can be defined by an intercept term b and a decision hyperplane normal vector w ~ which is perpendicular to the hyperplane. This vector is commonly
WEIGHT VECTOR
FUNCTIONAL MARGIN
referred to in the machine learning literature as the weight vector. To choose among all the hyperplanes that are perpendicular to the normal vector, we specify the intercept term b. Because the hyperplane is perpendicular to the normal vector, all points ~x on the hyperplane satisfy w ~ T~x = −b. Now suppose that we have a set of training data points D = {(~xi , yi )}, where each member is a pair of a point ~x i and a class label y i corresponding to it.1 For SVMs, the two data classes are always named +1 and −1 (rather than 1 and 0), and the intercept term is always explicitly represented as b (rather than being folded into the weight vector w ~ by adding an extra always-on feature). The math works out much more cleanly if you do things this way, as we will see almost immediately in the definition of functional margin. The linear classifier is then: f (~x ) = sign(~ wT~x + b) A value of −1 indicates one class, and a value of +1 the other class. We are confident in the classification of a point if it is far away from the decision boundary. For a given data set and decision hyperplane, we define the functional margin of the ith example ~x i with respect to a hyperplane h~ w, b i as the quantity y i (~ wT~xi + b). The functional margin of a data set with respect to a decision surface is then twice the functional margin of any of the points in the data set with minimal functional margin (the factor of 2 comes from measuring across the whole width of the margin, as in Figure 15.3). However, there is a problem with using this definition as is: the value is underconstrained, because we can always make the functional margin as big as we wish by simply scaling up w ~ and b. For example, if we replace w ~ by 5w ~ and b by 5b then the functional margin yi (5w ~ T~xi + 5b) is five times as large. This suggests that we need to place some constraint on the size of the w ~ vector. To get a sense of how to do that, let us look at the actual geometry. What is the Euclidean distance from a point ~x to the decision boundary? In Figure 15.3, we denote by r this distance. We know that the shortest distance between a point and a hyperplane is perpendicular to the plane, and hence, parallel to w ~ . A unit vector in this direction is w ~ /|~ w|. The dotted line in the diagram is then a translation of the vector r w ~ /|~ w|. Let us label the point on the hyperplane closest to ~x as ~x ′ . Then:
~x ′ = ~x − yr
w ~ |~ w|
where multiplying by y just changes the sign for the two cases of ~x being on either side of the decision surface. Moreover, ~x ′ lies on the decision boundary 1. As discussed in Section 14.1 (page 291), we present the general case of points in a vector space, but if the points are length normalized document vectors, then all the action is taking place on the surface of a unit sphere, and the decision surface intersects the sphere's surface.
u~x t
+ ′
2 w ~
◮ Figure 15.3 The geometric margin of a point (r) and a decision boundary (ρ).
and so satisfies w ~ T~x ′ + b = 0. Hence: w ~ T ~x − yr
(15.3) Solving for r gives:2 (15.4)
GEOMETRIC MARGIN
r=y
w ~ +b = 0 |~ w| w ~ T~x + b |~ w|
Again, the points closest to the separating hyperplane are support vectors. The geometric margin of the classifier is the maximum width of the band that can be drawn separating the support vectors of the two classes. That is, it is twice the minimum value over data points for r given in Equation (15.4), or, equivalently, the maximal width of one of the fat separators shown in Figure 15.2. The geometric margin is clearly invariant to scaling of parameters: if we replace w ~ by 5w ~ and b by 5b, then the geometric margin is the same, because it is inherently normalized by the length of w ~ . This means that we can impose any scaling constraint we wish on w ~ without affecting the geometric margin. Among other choices, we could use unit vectors, as in Chapter 6, by 2. Recall that |~ w| =
w ~ Tw ~.
requiring that |~ w| = 1. This would have the effect of making the geometric margin the same as the functional margin. Since we can scale the functional margin as we please, for convenience in solving large SVMs, let us choose to require that the functional margin of all data points is at least 1 and that it is equal to 1 for at least one data vector. That is, for all items in the data: yi (~ wT~xi + b) ≥ 1
and there exist support vectors for which the inequality is an equality. Since each example's distance from the hyperplane is ri = yi (~ wT~x i + b)/|~ w |, the geometric margin is ρ = 2/|~ w|. Our desire is still to maximize this geometric margin. That is, we want to find w ~ and b such that: • ρ = 2/|~ w| is maximized
• For all (~x i , yi ) ∈ D, y i (~ wT~xi + b) ≥ 1 Maximizing 2/|~ w| is the same as minimizing |~ w|/2. This gives the final standard formulation of an SVM as a minimization problem: (15.6)
Find w ~ and b such that: •
1 T ~ w ~ 2w
is minimized, and
• for all {(~x i , yi )}, yi (~ wT~xi + b) ≥ 1 Q UADRATIC
P ROGRAMMING
We are now optimizing a quadratic function subject to linear constraints. Quadratic optimization problems are a standard, well-known class of mathematical optimization problems, and many algorithms exist for solving them. We could in principle build our SVM using standard quadratic programming (QP) libraries, but there has been much recent research in this area aiming to exploit the structure of the kind of QP that emerges from an SVM. As a result, there are more intricate but much faster and more scalable libraries available especially for building SVMs, which almost everyone uses to build models. We will not present the details of such algorithms here. However, it will be helpful to what follows to understand the shape of the solution of such an optimization problem. The solution involves constructing a dual problem where a Lagrange multiplier αi is associated with each constraint yi (~ wT ~x i + b) ≥ 1 in the primal problem: Find α1 , . . . α N such that ∑ αi −
∑i ∑ j αi α j yi y j~xi T~x j is maximized, and
• ∑i αi yi = 0 • αi ≥ 0 for all 1 ≤ i ≤ N The solution is then of the form:
◮ Figure 15.4 A tiny 3 data point training set for an SVM.
w ~ = ∑ αi yi~xi b = yk − w ~ T~xk for any ~xk such that αk 6= 0
In the solution, most of the αi are zero. Each non-zero αi indicates that the corresponding ~xi is a support vector. The classification function is then: f (~x ) = sign(∑i αi yi~xi T~x + b)
Both the term to be maximized in the dual problem and the classifying function involve a dot product between pairs of points (~x and ~x i or ~xi and ~x j ), and that is the only way the data are used – we will return to the significance of this later. To recap, we start with a training data set. The data set uniquely defines the best separating hyperplane, and we feed the data through a quadratic optimization procedure to find this plane. Given a new point ~x to classify, the classification function f (~x ) in either Equation (15.1) or Equation (15.9) is computing the projection of the point onto the hyperplane normal. The sign of this function determines the class to assign to the point. If the point is within the margin of the classifier (or another confidence threshold t that we might have determined to minimize classification mistakes) then the classifier can return "don't know" rather than one of the two classes. The value of f (~x ) may also be transformed into a probability of classification; fitting a sigmoid to transform the values is standard (Platt 2000). Also, since the margin is constant, if the model includes dimensions from various sources, careful rescaling of some dimensions may be required. However, this is not a problem if our documents (points) are on the unit hypersphere.
Consider building an SVM over the (very little) data set shown in Figure 15.4. Working geometrically, for an example like this, the maximum margin weight vector will be parallel to the shortest line connecting points of the two classes, that is, the line between (1, 1) and (2, 3), giving a weight vector of (1, 2). The optimal decision surface is orthogonal to that line and intersects it at the halfway point.
15 Support vector machines and machine learning on documents Therefore, it passes through (1.5, 2). So, the SVM decision boundary is: y = x1 + 2x2 − 5.5 Working algebraically, with the standard constraint that sign(yi (~ wT~xi + b )) ≥ 1, we seek to minimize |~ w|. This happens when this constraint is satisfied with equality by the two support vectors. Further we know that the solution is w ~ = ( a, 2a) for some a. So we have that: a + 2a + b 2a + 6a + b
−1 1
Therefore, a = 2/5 and b = −11/5. So the optimal hyperplane is given by w ~ = (2/5, 4/5) and b = −11/5. √ √ √ The margin ρ is 2/|~ w| = 2/ 4/25 + 16/25 = 2/(2 5/5) = 5. This answer can be confirmed geometrically by examining Figure 15.4.
What is the minimum number of support vectors that there can be for a data set (which contains instances of each class)? Exercise 15.2
The basis of being able to use kernels in SVMs (see Section 15.2.3) is that the classification function can be written in the form of Equation (15.9) (where, for large problems, most αi are 0). Show explicitly how the classification function could be written in this form for the data set from Example 15.1. That is, write f as a function where the data points appear and the only variable is ~x. Exercise 15.3
Install an SVM package such as SVMlight (http://svmlight.joachims.org/), and build an SVM for the data set discussed in Example 15.1. Confirm that the program gives the same solution as the text. For SVMlight, or another package that accepts the same training data format, the training file would be:
+1 1:2 2:3 −1 1:2 2:0 −1 1:1 2:1 The training command for SVMlight is then: svm_learn -c 1 -a alphas.dat train.dat model.dat
The -c 1 option is needed to turn off use of the slack variables that we discuss in Section 15.2.1. Check that the norm of the weight vector agrees with what we found in Example 15.1. Examine the file alphas.dat which contains the αi values, and check that they agree with your answers in Exercise 15.2.
15.2 Extensions to the SVM model
ξi
t t u u
~xi b
ξj
~x j ◮ Figure 15.5 Large margin classification with slack variables.
SLACK VARIABLES
Extensions to the SVM model Soft margin classification For the very high dimensional problems common in text classification, sometimes the data are linearly separable. But in the general case they are not, and even if they are, we might prefer a solution that better separates the bulk of the data while ignoring a few weird noise documents. If the training set D is not linearly separable, the standard approach is to allow the fat decision margin to make a few mistakes (some points – outliers or noisy examples – are inside or on the wrong side of the margin). We then pay a cost for each misclassified example, which depends on how far it is from meeting the margin requirement given in Equation (15.5). To implement this, we introduce slack variables ξ i . A non-zero value for ξ i allows ~x i to not meet the margin requirement at a cost proportional to the value of ξ i . See Figure 15.5. The formulation of the SVM optimization problem with slack variables is: Find w ~ , b, and ξ i ≥ 0 such that: •
1 T ~ w ~ + C ∑i ξ i 2w
is minimized
• and for all {(~xi , yi )}, y i (~ wT~x i + b) ≥ 1 − ξ i
REGULARIZATION
The optimization problem is then trading off how fat it can make the margin versus how many points have to be moved around to allow this margin. The margin can be less than 1 for a point ~x i by setting ξ i > 0, but then one pays a penalty of Cξ i in the minimization for having done that. The sum of the ξ i gives an upper bound on the number of training errors. Soft-margin SVMs minimize training error traded off against margin. The parameter C is a regularization term, which provides a way to control overfitting: as C becomes large, it is unattractive to not respect the data at the cost of reducing the geometric margin; when it is small, it is easy to account for some data points with the use of slack variables and to have a fat margin placed so it models the bulk of the data. The dual problem for soft margin classification becomes: Find α1 , . . . α N such that ∑ αi −
• ∑i αi yi = 0 • 0 ≤ αi ≤ C for all 1 ≤ i ≤ N Neither the slack variables ξ i nor Lagrange multipliers for them appear in the dual problem. All we are left with is the constant C bounding the possible size of the Lagrange multipliers for the support vector data points. As before, the ~xi with non-zero αi will be the support vectors. The solution of the dual problem is of the form: (15.12)
w ~ = ∑ αyi~xi b = y k (1 − ξ k ) − w ~ T~xk for k = arg maxk αk Again w ~ is not needed explicitly for classification, which can be done in terms of dot products with data points, as in Equation (15.9). Typically, the support vectors will be a small proportion of the training data. However, if the problem is non-separable or with small margin, then every data point which is misclassified or within the margin will have a nonzero αi . If this set of points becomes large, then, for the nonlinear case which we turn to in Section 15.2.3, this can be a major slowdown for using SVMs at test time. The complexity of training and testing with linear SVMs is shown in Table 15.1.3 The time for training an SVM is dominated by the time for solving the underlying QP, and so the theoretical and empirical complexity varies depending on the method used to solve it. The standard result for solving QPs is that it takes time cubic in the size of the data set (Kozlov et al. 1979). All the recent work on SVM training has worked to reduce that complexity, often by 3. We write Θ(|D | Lave ) for Θ( T ) (page 262) and assume that the length of test documents is bounded as we did on page 262.
Classifier NB NB
Time complexity Θ(|D | Lave + |C ||V |) Θ(|C | Ma )
Rocchio Rocchio
training testing
kNN kNN
preprocessing preprocessing
Θ(|D | Lave ) Θ(|D | Mave Ma )
no preprocessing no preprocessing
Θ(1) Θ(|D | Lave Ma )
SVM SVM
cutting planes
O(|C ||D |3 Mave ); ≈ O(|C ||D |1.7 Mave ), empirically O(|C ||D | Mave ) O(|C | Ma )
Θ(|D | Lave + |C ||V |) Θ(|C | Ma )
◮ Table 15.1 Training and testing complexity of various classifiers including SVMs. Training is the time the learning method takes to learn a classifier over D, while testing is the time it takes a classifier to classify one document. For SVMs, multiclass classification is assumed to be done by a set of |C | one-versus-rest classifiers. Lave is the average number of tokens per document, while Mave is the average vocabulary (number of non-zero features) of a document. La and Ma are the numbers of tokens and types, respectively, in the test document.
being satisfied with approximate solutions. Standardly, empirical complexity is about O(|D |1.7 ) (Joachims 2006a). Nevertheless, the super-linear training time of traditional SVM algorithms makes them difficult or impossible to use on very large training data sets. Alternative traditional SVM solution algorithms which are linear in the number of training examples scale badly with a large number of features, which is another standard attribute of text problems. However, a new training algorithm based on cutting plane techniques gives a promising answer to this issue by having running time linear in the number of training examples and the number of non-zero features in examples (Joachims 2006a). Nevertheless, the actual speed of doing quadratic optimization remains much slower than simply counting terms as is done in a Naive Bayes model. Extending SVM algorithms to nonlinear SVMs, as in the next section, standardly increases training complexity by a factor of |D | (since dot products between examples need to be calculated), making them impractical. In practice it can often be cheaper to materialize
the higher-order features and to train a linear SVM.4
STRUCTURAL SVM S
Multiclass SVMs SVMs are inherently two-class classifiers. The traditional way to do multiclass classification with SVMs is to use one of the methods discussed in Section 14.5 (page 306). In particular, the most common technique in practice has been to build |C | one-versus-rest classifiers (commonly referred to as "one-versus-all" or OVA classification), and to choose the class which classifies the test datum with greatest margin. Another strategy is to build a set of one-versus-one classifiers, and to choose the class that is selected by the most classifiers. While this involves building |C |(|C | − 1)/2 classifiers, the time for training classifiers may actually decrease, since the training data set for each classifier is much smaller. However, these are not very elegant approaches to solving multiclass problems. A better alternative is provided by the construction of multiclass SVMs, where we build a two-class classifier over a feature vector Φ(~x, y) derived from the pair consisting of the input features and the class of the datum. At ~ T Φ(~x, y′ ). The martest time, the classifier chooses the class y = arg maxy′ w gin during training is the gap between this value for the correct class and for the nearest other class, and so the quadratic program formulation will require that ∀i ∀y 6= y i w ~ T Φ(~xi , yi ) − w ~ T Φ(~xi , y) ≥ 1 − ξ i . This general method can be extended to give a multiclass formulation of various kinds of linear classifiers. It is also a simple instance of a generalization of classification where the classes are not just a set of independent, categorical labels, but may be arbitrary structured objects with relationships defined between them. In the SVM world, such work comes under the label of structural SVMs. We mention them again in Section 15.4.2.
Nonlinear SVMs With what we have presented so far, data sets that are linearly separable (perhaps with a few exceptions or some noise) are well-handled. But what are we going to do if the data set just doesn't allow classification by a linear classifier? Let us look at a one-dimensional case. The top data set in Figure 15.6 is straightforwardly classified by a linear classifier but the middle data set is not. We instead need to be able to pick out an interval. One way to solve this problem is to map the data on to a higher dimensional space and then to use a linear classifier in the higher dimensional space. For example, the bottom part of the figure shows that a linear separator can easily classify the data 4. Materializing the features refers to directly calculating higher order and interaction terms and then putting them into a linear model.
◮ Figure 15.6 Projecting data that is not linearly separable into a higher dimensional space can make it linearly separable.
KERNEL TRICK
if we use a quadratic function to map the data into two dimensions (a polar coordinates projection would be another possibility). The general idea is to map the original feature space to some higher-dimensional feature space where the training set is separable. Of course, we would want to do so in ways that preserve relevant dimensions of relatedness between data points, so that the resultant classifier should still generalize well. SVMs, and also a number of other linear classifiers, provide an easy and efficient way of doing this mapping to a higher dimensional space, which is referred to as "the kernel trick". It's not really a trick: it just exploits the math that we have seen. The SVM linear classifier relies on a dot product between data point vectors. Let K (~xi , ~x j ) = ~xi T~x j . Then the classifier we have seen so
far is: (15.13)
f (~x) = sign(∑ αi yi K (~xi , ~x) + b) i
KERNEL FUNCTION
✎ (15.14)
M ERCER KERNEL
Now suppose we decide to map every data point into a higher dimensional space via some transformation Φ: ~x 7→ φ(~x ). Then the dot product becomes φ(~x i )T φ(~x j ). If it turned out that this dot product (which is just a real number) could be computed simply and efficiently in terms of the original data points, then we wouldn't have to actually map from ~x 7→ φ(~x ). Rather, we could simply compute the quantity K (~x i , ~x j ) = φ(~xi )T φ(~x j ), and then use the function's value in Equation (15.13). A kernel function K is such a function that corresponds to a dot product in some expanded feature space. Example 15.2: The quadratic kernel in two dimensions. For 2-dimensional
vectors ~u = (u1 u2 ), ~v = (v1 v2 ), consider K (~u, ~v) = (1 + ~uT~v)2 . We wish to T show that i.e., √ this is a2 kernel, √ √ that K (~u, ~v ) = φ(~u ) φ(~v ) for some φ. Consider φ(~u ) = 2 (1 u1 2u1 u2 u2 2u1 2u2 ). Then: K (~u, ~v)
(1 + ~uT~v)2
1 + u21 v21 + 2u1 v1 u2 v2 + u22 v22 + 2u1 v1 + 2u2 v2 √ √ √ √ √ √ (1 u21 2u1 u2 u22 2u1 2u2 )T (1 v21 2v1 v2 v22 2v1 2v2 )
φ(~u)T φ(~v)
In the language of functional analysis, what kinds of functions are valid kernel functions? Kernel functions are sometimes more precisely referred to as Mercer Rkernels, because they must satisfy Mercer's condition: for any g(~x) such that g(~x)2 d~x is finite, we must have that: Z
K (~x, ~z) g(~x) g(~z)d~xd~z ≥ 0 .
A kernel function K must be continuous, symmetric, and have a positive definite gram matrix. Such a K means that there exists a mapping to a reproducing kernel Hilbert space (a Hilbert space is a vector space closed under dot products) such that the dot product there gives the same value as the function K. If a kernel does not satisfy Mercer's condition, then the corresponding QP may have no solution. If you would like to better understand these issues, you should consult the books on SVMs mentioned in Section 15.5. Otherwise, you can content yourself with knowing that 90% of work with kernels uses one of two straightforward families of functions of two vectors, which we define below, and which define valid kernels. The two commonly used families of kernels are polynomial kernels and radial basis functions. Polynomial kernels are of the form K (~x, ~z) = (1 +
~xT~z)d . The case of d = 1 is a linear kernel, which is what we had before the start of this section (the constant 1 just changing the threshold). The case of d = 2 gives a quadratic kernel, and is very commonly used. We illustrated the quadratic kernel in Example 15.2. The most common form of radial basis function is a Gaussian distribution, calculated as: K (~x, ~z) = e−(~x−~z)
2 / (2σ2 )
A radial basis function (rbf) is equivalent to mapping the data into an infinite dimensional Hilbert space, and so we cannot illustrate the radial basis function concretely, as we did a quadratic kernel. Beyond these two families, there has been interesting work developing other kernels, some of which is promising for text applications. In particular, there has been investigation of string kernels (see Section 15.5). The world of SVMs comes with its own language, which is rather different from the language otherwise used in machine learning. The terminology does have deep roots in mathematics, but it's important not to be too awed by that terminology. Really, we are talking about some quite simple things. A polynomial kernel allows us to model feature conjunctions (up to the order of the polynomial). That is, if we want to be able to model occurrences of pairs of words, which give distinctive information about topic classification, not given by the individual words alone, like perhaps operating AND system or ethnic AND cleansing, then we need to use a quadratic kernel. If occurrences of triples of words give distinctive information, then we need to use a cubic kernel. Simultaneously you also get the powers of the basic features – for most text applications, that probably isn't useful, but just comes along with the math and hopefully doesn't do harm. A radial basis function allows you to have features that pick out circles (hyperspheres) – although the decision boundaries become much more complex as multiple such features interact. A string kernel lets you have features that are character subsequences of terms. All of these are straightforward notions which have also been used in many other places under different names.
Experimental results We presented results in Section 13.6 showing that an SVM is a very effective text classifier. The results of Dumais et al. (1998) given in Table 13.9 show SVMs clearly performing the best. This was one of several pieces of work from this time that established the strong reputation of SVMs for text classification. Another pioneering work on scaling and evaluating SVMs for text classification was (Joachims 1998). We present some of his results
earn acq money-fx grain crude trade interest ship wheat corn microavg.
NB 96.0 90.7 59.6 69.8 81.2 52.2 57.6 80.9 63.4 45.2 72.3
Rocchio 96.1 92.1 67.6 79.5 81.5 77.4 72.5 83.1 79.4 62.2 79.9
Dec. Trees 96.1 85.3 69.4 89.1 75.5 59.2 49.1 80.9 85.5 87.7 79.4
kNN 97.8 91.8 75.4 82.6 85.8 77.9 76.7 79.8 72.9 71.4 82.6
linear SVM C = 0.5 C = 1.0 98.0 98.2 95.5 95.6 78.8 78.5 91.9 93.1 89.4 89.4 79.2 79.2 75.6 74.8 87.4 86.5 86.6 86.8 87.5 87.8 86.7 87.5
rbf-SVM σ≈7 98.1 94.7 74.3 93.4 88.7 76.6 69.1 85.8 82.4 84.6 86.4
◮ Table 15.2 SVM classifier break-even F1 from (Joachims 2002a, p. 114). Results are shown for the 10 largest categories and for microaveraged performance over all 90 categories on the Reuters-21578 data set.
from (Joachims 2002a) in Table 15.2.5 Joachims used a large number of term features in contrast to Dumais et al. (1998), who used MI feature selection (Section 13.5.1, page 272) to build classifiers with a much more limited number of features. The success of the linear SVM mirrors the results discussed in Section 14.6 (page 308) on other linear approaches like Naive Bayes. It seems that working with simple term features can get one a long way. It is again noticeable the extent to which different papers' results for the same machine learning methods differ. In particular, based on replications by other researchers, the Naive Bayes results of (Joachims 1998) appear too weak, and the results in Table 13.9 should be taken as representative.
Issues in the classification of text documents There are lots of applications of text classification in the commercial world; email spam filtering is perhaps now the most ubiquitous. Jackson and Moulinier (2002) write: "There is no question concerning the commercial value of being able to classify documents automatically by content. There are myriad 5. These results are in terms of the break-even F1 (see Section 8.4). Many researchers disprefer this measure for text classification evaluation, since its calculation may involve interpolation rather than an actual parameter setting of the system and it is not clear why this value should be reported rather than maximal F1 or another point on the precision/recall curve motivated by the task at hand. While earlier results in (Joachims 1998) suggested notable gains on this task from the use of higher order polynomial or rbf kernels, this was with hard-margin SVMs. With soft-margin SVMs, a simple linear SVM with the default C = 1 performs best.
15.3 Issues in the classification of text documents
potential applications of such a capability for corporate Intranets, government departments, and Internet publishers." Most of our discussion of classification has focused on introducing various machine learning methods rather than discussing particular features of text documents relevant to classification. This bias is appropriate for a textbook, but is misplaced for an application developer. It is frequently the case that greater performance gains can be achieved from exploiting domain-specific text features than from changing from one machine learning method to another. Jackson and Moulinier (2002) suggest that "Understanding the data is one of the keys to successful categorization, yet this is an area in which most categorization tool vendors are extremely weak. Many of the 'one size fits all' tools on the market have not been tested on a wide range of content types." In this section we wish to step back a little and consider the applications of text classification, the space of possible solutions, and the utility of application-specific heuristics.
Choosing what kind of classifier to use When confronted with a need to build a text classifier, the first question to ask is how much training data is there currently available? None? Very little? Quite a lot? Or a huge amount, growing every day? Often one of the biggest practical challenges in fielding a machine learning classifier in real applications is creating or obtaining enough training data. For many problems and algorithms, hundreds or thousands of examples from each class are required to produce a high performance classifier and many real world contexts involve large sets of categories. We will initially assume that the classifier is needed as soon as possible; if a lot of time is available for implementation, much of it might be spent on assembling data resources. If you have no labeled training data, and especially if there are existing staff knowledgeable about the domain of the data, then you should never forget the solution of using hand-written rules. That is, you write standing queries, as we touched on at the beginning of Chapter 13. For example: IF
(wheat
grain)
(whole
bread)
c = grain
In practice, rules get a lot bigger than this, and can be phrased using more sophisticated query languages than just Boolean expressions, including the use of numeric scores. With careful crafting (that is, by humans tuning the rules on development data), the accuracy of such rules can become very high. Jacobs and Rau (1990) report identifying articles about takeovers with 92% precision and 88.5% recall, and Hayes and Weinstein (1990) report 94% recall and 84% precision over 675 categories on Reuters newswire documents. Nevertheless the amount of work to create such well-tuned rules is very large. A reasonable estimate is 2 days per class, and extra time has to go
SEMI - SUPERVISED LEARNING
TRANSDUCTIVE SVM S
into maintenance of rules, as the content of documents in classes drifts over time (cf. page 269). If you have fairly little data and you are going to train a supervised classifier, then machine learning theory says you should stick to a classifier with high bias, as we discussed in Section 14.6 (page 308). For example, there are theoretical and empirical results that Naive Bayes does well in such circumstances (Ng and Jordan 2001, Forman and Cohen 2004), although this effect is not necessarily observed in practice with regularized models over textual data (Klein and Manning 2002). At any rate, a very low bias model like a nearest neighbor model is probably counterindicated. Regardless, the quality of the model will be adversely affected by the limited training data. Here, the theoretically interesting answer is to try to apply semi-supervised training methods. This includes methods such as bootstrapping or the EM algorithm, which we will introduce in Section 16.5 (page 368). In these methods, the system gets some labeled documents, and a further large supply of unlabeled documents over which it can attempt to learn. One of the big advantages of Naive Bayes is that it can be straightforwardly extended to be a semi-supervised learning algorithm, but for SVMs, there is also semisupervised learning work which goes under the title of transductive SVMs. See the references for pointers. Often, the practical answer is to work out how to get more labeled data as quickly as you can. The best way to do this is to insert yourself into a process where humans will be willing to label data for you as part of their natural tasks. For example, in many cases humans will sort or route email for their own purposes, and these actions give information about classes. The alternative of getting human labelers expressly for the task of training classifiers is often difficult to organize, and the labeling is often of lower quality, because the labels are not embedded in a realistic task context. Rather than getting people to label all or a random sample of documents, there has also been considerable research on active learning, where a system is built which decides which documents a human should label. Usually these are the ones on which a classifier is uncertain of the correct classification. This can be effective in reducing annotation costs by a factor of 2–4, but has the problem that the good documents to label to train one type of classifier often are not the good documents to label to train a different type of classifier. If there is a reasonable amount of labeled data, then you are in the perfect position to use everything that we have presented about text classification. For instance, you may wish to use an SVM. However, if you are deploying a linear classifier such as an SVM, you should probably design an application that overlays a Boolean rule-based classifier over the machine learning classifier. Users frequently like to adjust things that do not come out quite right, and if management gets on the phone and wants the classification of a particular document fixed right now, then this is much easier to
do by hand-writing a rule than by working out how to adjust the weights of an SVM without destroying the overall classification accuracy. This is one reason why machine learning models like decision trees which produce userinterpretable Boolean-like models retain considerable popularity. If a huge amount of data are available, then the choice of classifier probably has little effect on your results and the best choice may be unclear (cf. Banko and Brill 2001). It may be best to choose a classifier based on the scalability of training or even runtime efficiency. To get to this point, you need to have huge amounts of data. The general rule of thumb is that each doubling of the training data size produces a linear increase in classifier performance, but with very large amounts of data, the improvement becomes sub-linear.
Improving classifier performance For any particular application, there is usually significant room for improving classifier effectiveness through exploiting features specific to the domain or document collection. Often documents will contain zones which are especially useful for classification. Often there will be particular subvocabularies which demand special treatment for optimal classification effectiveness. Large and difficult category taxonomies
HIERARCHICAL CLASSIFICATION
If a text classification problem consists of a small number of well-separated categories, then many classification algorithms are likely to work well. But many real classification problems consist of a very large number of often very similar categories. The reader might think of examples like web directories (the Yahoo! Directory or the Open Directory Project), library classification schemes (Dewey Decimal or Library of Congress) or the classification schemes used in legal or medical applications. For instance, the Yahoo! Directory consists of over 200,000 categories in a deep hierarchy. Accurate classification over large sets of closely related classes is inherently difficult. Most large sets of categories have a hierarchical structure, and attempting to exploit the hierarchy by doing hierarchical classification is a promising approach. However, at present the effectiveness gains from doing this rather than just working with the classes that are the leaves of the hierarchy remain modest.6 But the technique can be very useful simply to improve the scalability of building classifiers over large hierarchies. Another simple way to improve the scalability of classifiers over large hierarchies is the use of aggressive feature selection. We provide references to some work on hierarchical classification in Section 15.5. 6. Using the small hierarchy in Figure 13.1 (page 257) as an example, the leaf classes are ones like poultry and coffee, as opposed to higher-up classes like industries.
A general result in machine learning is that you can always get a small boost in classification accuracy by combining multiple classifiers, provided only that the mistakes that they make are at least somewhat independent. There is now a large literature on techniques such as voting, bagging, and boosting multiple classifiers. Again, there are some pointers in the references. Nevertheless, ultimately a hybrid automatic/manual solution may be needed to achieve sufficient classification accuracy. A common approach in such situations is to run a classifier first, and to accept all its high confidence decisions, but to put low confidence decisions in a queue for manual review. Such a process also automatically leads to the production of new training data which can be used in future versions of the machine learning classifier. However, note that this is a case in point where the resulting training data is clearly not randomly sampled from the space of documents. Features for text
The default in both ad hoc retrieval and text classification is to use terms as features. However, for text classification, a great deal of mileage can be achieved by designing additional features which are suited to a specific problem. Unlike the case of IR query languages, since these features are internal to the classifier, there is no problem of communicating these features to an end user. This process is generally referred to as feature engineering. At present, feature engineering remains a human craft, rather than something done by machine learning. Good feature engineering can often markedly improve the performance of a text classifier. It is especially beneficial in some of the most important applications of text classification, like spam and porn filtering. Classification problems will often contain large numbers of terms which can be conveniently grouped, and which have a similar vote in text classification problems. Typical examples might be year mentions or strings of exclamation marks. Or they may be more specialized tokens like ISBNs or chemical formulas. Often, using them directly in a classifier would greatly increase the vocabulary without providing classificatory power beyond knowing that, say, a chemical formula is present. In such cases, the number of features and feature sparseness can be reduced by matching such items with regular expressions and converting them into distinguished tokens. Consequently, effectiveness and classifier speed are normally enhanced. Sometimes all numbers are converted into a single feature, but often some value can be had by distinguishing different kinds of numbers, such as four digit numbers (which are usually years) versus other cardinal numbers versus real numbers with a decimal point. Similar techniques can be applied to dates, ISBN numbers, sports game scores, and so on. Going in the other direction, it is often useful to increase the number of fea-
tures by matching parts of words, and by matching selected multiword patterns that are particularly discriminative. Parts of words are often matched by character k-gram features. Such features can be particularly good at providing classification clues for otherwise unknown words when the classifier is deployed. For instance, an unknown word ending in -rase is likely to be an enzyme, even if it wasn't seen in the training data. Good multiword patterns are often found by looking for distinctively common word pairs (perhaps using a mutual information criterion between words, in a similar way to its use in Section 13.5.1 (page 272) for feature selection) and then using feature selection methods evaluated against classes. They are useful when the components of a compound would themselves be misleading as classification cues. For instance, this would be the case if the keyword ethnic was most indicative of the categories food and arts, the keyword cleansing was most indicative of the category home, but the collocation ethnic cleansing instead indicates the category world news. Some text classifiers also make use of features from named entity recognizers (cf. page 195). Do techniques like stemming and lowercasing (Section 2.2, page 22) help for text classification? As always, the ultimate test is empirical evaluations conducted on an appropriate test collection. But it is nevertheless useful to note that such techniques have a more restricted chance of being useful for classification. For IR, you often need to collapse forms of a word like oxygenate and oxygenation, because the appearance of either in a document is a good clue that the document will be relevant to a query about oxygenation. Given copious training data, stemming necessarily delivers no value for text classification. If several forms that stem together have a similar signal, the parameters estimated for all of them will have similar weights. Techniques like stemming help only in compensating for data sparseness. This can be a useful role (as noted at the start of this section), but often different forms of a word can convey significantly different cues about the correct document classification. Overly aggressive stemming can easily degrade classification performance. Document zones in text classification As already discussed in Section 6.1, documents usually have zones, such as mail message headers like the subject and author, or the title and keywords of a research article. Text classifiers can usually gain from making use of these zones during training and classification. Upweighting document zones. In text classification problems, you can frequently get a nice boost to effectiveness by differentially weighting contributions from different document zones. Often, upweighting title words is particularly effective (Cohen and Singer 1999, p. 163). As a rule of thumb,
it is often effective to double the weight of title words in text classification problems. You can also get value from upweighting words from pieces of text that are not so much clearly defined zones, but where nevertheless evidence from document structure or content suggests that they are important. Murata et al. (2000) suggest that you can also get value (in an ad hoc retrieval context) from upweighting the first sentence of a (newswire) document.
PARAMETER TYING
Separate feature spaces for document zones. There are two strategies that can be used for document zones. Above we upweighted words that appear in certain zones. This means that we are using the same features (that is, parameters are "tied" across different zones), but we pay more attention to the occurrence of terms in particular zones. An alternative strategy is to have a completely separate set of features and corresponding parameters for words occurring in different zones. This is in principle more powerful: a word could usually indicate the topic Middle East when in the title but Commodities when in the body of a document. But, in practice, tying parameters is usually more successful. Having separate feature sets means having two or more times as many parameters, many of which will be much more sparsely seen in the training data, and hence with worse estimates, whereas upweighting has no bad effects of this sort. Moreover, it is quite uncommon for words to have different preferences when appearing in different zones; it is mainly the strength of their vote that should be adjusted. Nevertheless, ultimately this is a contingent result, depending on the nature and quantity of the training data. Connections to text summarization. In Section 8.7, we mentioned the field of text summarization, and how most work in that field has adopted the limited goal of extracting and assembling pieces of the original text that are judged to be central based on features of sentences that consider the sentence's position and content. Much of this work can be used to suggest zones that may be distinctively useful for text classification. For example Kołcz et al. (2000) consider a form of feature selection where you classify documents based only on words in certain zones. Based on text summarization research, they consider using (i) only the title, (ii) only the first paragraph, (iii) only the paragraph with the most title words or keywords, (iv) the first two paragraphs or the first and last paragraph, or (v) all sentences with a minimum number of title words or keywords. In general, these positional feature selection methods produced as good results as mutual information (Section 13.5.1), and resulted in quite competitive classifiers. Ko et al. (2004) also took inspiration from text summarization research to upweight sentences with either words from the title or words that are central to the document's content, leading to classification accuracy gains of almost 1%. This
15.4 Machine learning methods in ad hoc information retrieval
presumably works because most such sentences are somehow more central to the concerns of the document.
Exercise 15.4 [⋆⋆] Spam email often makes use of various cloaking techniques to try to get through. One method is to pad or substitute characters so as to defeat word-based text classifiers. For example, you see terms like the following in spam email: Rep1icaRolex PHARlbdMACY
bonmus [LEV]i[IT]l[RA]
Viiiaaaagra se∧xual
pi11z ClAfLlS
Discuss how you could engineer features that would largely defeat this strategy. Exercise 15.5 [⋆⋆] Another strategy often used by purveyors of email spam is to follow the message they wish to send (such as buying a cheap stock or whatever) with a paragraph of text from another innocuous source (such as a news article). Why might this strategy be effective? How might it be addressed by a text classifier? Exercise 15.6 [ ⋆] What other kinds of features appear as if they would be useful in an email spam classifier?
Machine learning methods in ad hoc information retrieval Rather than coming up with term and document weighting functions by hand, as we primarily did in Chapter 6, we can view different sources of relevance signal (cosine score, title match, etc.) as features in a learning problem. A classifier that has been fed examples of relevant and nonrelevant documents for each of a set of queries can then figure out the relative weights of these signals. If we configure the problem so that there are pairs of a document and a query which are assigned a relevance judgment of relevant or nonrelevant, then we can think of this problem too as a text classification problem. Taking such a classification approach is not necessarily best, and we present an alternative in Section 15.4.2. Nevertheless, given the material we have covered, the simplest place to start is to approach this problem as a classification problem, by ordering the documents according to the confidence of a two-class classifier in its relevance decision. And this move is not purely pedagogical; exactly this approach is sometimes used in practice.
A simple example of machine-learned scoring In this section we generalize the methodology of Section 6.1.2 (page 113) to machine learning of the scoring function. In Section 6.1.2 we considered a case where we had to combine Boolean indicators of relevance; here we consider more general factors to further develop the notion of machine-learned
Example Φ1 Φ2 Φ3 Φ4 Φ5 Φ6 Φ7 ···
DocID 37 37 238 238 1741 2094 3191 ···
Query linux operating system penguin logo operating system runtime environment kernel layer device driver device driver
Cosine score 0.032 0.02 0.043 0.004 0.022 0.03 0.027 ···
ω 3 4 2 2 3 2 5 ···
Judgment relevant nonrelevant relevant nonrelevant relevant relevant nonrelevant ···
◮ Table 15.3 Training examples for machine-learned scoring.
relevance. In particular, the factors we now consider go beyond Boolean functions of query term presence in document zones, as in Section 6.1.2. We develop the ideas in a setting where the scoring function is a linear combination of two factors: (1) the vector space cosine similarity between query and document and (2) the minimum window width ω within which the query terms lie. As we noted in Section 7.2.2 (page 144), query term proximity is often very indicative of a document being on topic, especially with longer documents and on the web. Among other things, this quantity gives us an implementation of implicit phrases. Thus we have one factor that depends on the statistics of query terms in the document as a bag of words, and another that depends on proximity weighting. We consider only two features in the development of the ideas because a two-feature exposition remains simple enough to visualize. The technique can be generalized to many more features. As in Section 6.1.2, we are provided with a set of training examples, each of which is a pair consisting of a query and a document, together with a relevance judgment for that document on that query that is either relevant or nonrelevant. For each such example we can compute the vector space cosine similarity, as well as the window width ω. The result is a training set as shown in Table 15.3, which resembles Figure 6.5 (page 115) from Section 6.1.2. Here, the two features (cosine score denoted α and window width ω) are real-valued predictors. If we once again quantify the judgment relevant as 1 and nonrelevant as 0, we seek a scoring function that combines the values of the features to generate a value that is (close to) 0 or 1. We wish this function to be in agreement with our set of training examples as far as possible. Without loss of generality, a linear classifier will use a linear combination of features of the form (15.17)
Score(d, q) = Score(α, ω ) = aα + bω + c, with the coefficients a, b, c to be learned from the training data. While it is
◮ Figure 15.7 A collection of training examples. Each R denotes a training example labeled relevant, while each N is a training example labeled nonrelevant.
possible to formulate this as an error minimization problem as we did in Section 6.1.2, it is instructive to visualize the geometry of Equation (15.17). The examples in Table 15.3 can be plotted on a two-dimensional plane with axes corresponding to the cosine score α and the window width ω. This is depicted in Figure 15.7. In this setting, the function Score(α, ω ) from Equation (15.17) represents a plane "hanging above" Figure 15.7. Ideally this plane (in the direction perpendicular to the page containing Figure 15.7) assumes values close to 1 above the points marked R, and values close to 0 above the points marked N. Since a plane is unlikely to assume only values close to 0 or 1 above the training sample points, we make use of thresholding: given any query and document for which we wish to determine relevance, we pick a value θ and if Score(α, ω ) > θ we declare the document to be relevant, else we declare the document to be nonrelevant. As we know from Figure 14.8 (page 301), all points that satisfy Score(α, ω ) = θ form a line (shown as a dashed line in Figure 15.7) and we thus have a linear classifier that separates relevant
from nonrelevant instances. Geometrically, we can find the separating line as follows. Consider the line passing through the plane Score(α, ω ) whose height is θ above the page containing Figure 15.7. Project this line down onto Figure 15.7; this will be the dashed line in Figure 15.7. Then, any subsequent query/document pair that falls below the dashed line in Figure 15.7 is deemed nonrelevant; above the dashed line, relevant. Thus, the problem of making a binary relevant/nonrelevant judgment given training examples as above turns into one of learning the dashed line in Figure 15.7 separating relevant training examples from the nonrelevant ones. Being in the α-ω plane, this line can be written as a linear equation involving α and ω, with two parameters (slope and intercept). The methods of linear classification that we have already looked at in Chapters 13–15 provide methods for choosing this line. Provided we can build a sufficiently rich collection of training samples, we can thus altogether avoid hand-tuning score functions as in Section 7.2.3 (page 145). The bottleneck of course is the ability to maintain a suitably representative set of training examples, whose relevance assessments must be made by experts.
REGRESSION ORDINAL REGRESSION
Result ranking by machine learning The above ideas can be readily generalized to functions of many more than two variables. There are lots of other scores that are indicative of the relevance of a document to a query, including static quality (PageRank-style measures, discussed in Chapter 21), document age, zone contributions, document length, and so on. Providing that these measures can be calculated for a training document collection with relevance judgments, any number of such measures can be used to train a machine learning classifier. For instance, we could train an SVM over binary relevance judgments, and order documents based on their probability of relevance, which is monotonic with the documents' signed distance from the decision boundary. However, approaching IR result ranking like this is not necessarily the right way to think about the problem. Statisticians normally first divide problems into classification problems (where a categorical variable is predicted) versus regression problems (where a real number is predicted). In between is the specialized field of ordinal regression where a ranking is predicted. Machine learning for ad hoc retrieval is most properly thought of as an ordinal regression problem, where the goal is to rank a set of documents for a query, given training data of the same sort. This formulation gives some additional power, since documents can be evaluated relative to other candidate documents for the same query, rather than having to be mapped to a global scale of goodness, while also weakening the problem space, since just a ranking is required rather than an absolute measure of relevance. Issues of ranking are especially germane in web search, where the ranking at
the very top of the results list is exceedingly important, whereas decisions of relevance of a document to a query may be much less important. Such work can and has been pursued using the structural SVM framework which we mentioned in Section 15.2.2, where the class being predicted is a ranking of results for a query, but here we will present the slightly simpler ranking SVM. The construction of a ranking SVM proceeds as follows. We begin with a set of judged queries. For each training query q, we have a set of documents returned in response to the query, which have been totally ordered by a person for relevance to the query. We construct a vector of features ψj = ψ(d j , q) for each document/query pair, using features such as those discussed in Section 15.4.1, and many more. For two documents d i and d j , we then form the vector of feature differences: Φ( d i , d j , q) = ψ ( d i , q) − ψ ( d j , q)
By hypothesis, one of di and d j has been judged more relevant. If d i is judged more relevant than d j , denoted di ≺ d j (di should precede d j in the results ordering), then we will assign the vector Φ(d i , d j , q) the class y ijq = +1; otherwise −1. The goal then is to build a classifier which will return w ~ T Φ(di , d j , q) > 0 iff
di ≺ d j
This SVM learning task is formalized in a manner much like the other examples that we saw before: (15.20)
Find w ~ , and ξ i,j ≥ 0 such that: •
1 T ~ w ~ + C ∑i,j ξ i,j 2w
• and for all {Φ(d i , d j , q) : di ≺ d j }, w ~ T Φ(di , d j , q) ≥ 1 − ξ i,j We can leave out yijq in the statement of the constraint, since we only need to consider the constraint for document pairs ordered in one direction, since ≺ is antisymmetric. These constraints are then solved, as before, to give a linear classifier which can rank pairs of documents. This approach has been used to build ranking functions which outperform standard hand-built ranking functions in IR evaluations on standard data sets; see the references for papers that present such results. Both of the methods that we have just looked at use a linear weighting of document features that are indicators of relevance, as has most work in this area. It is therefore perhaps interesting to note that much of traditional IR weighting involves nonlinear scaling of basic measurements (such as logweighting of term frequency, or idf). At the present time, machine learning is very good at producing optimal weights for features in a linear combination
(or other similar restricted model classes), but it is not good at coming up with good nonlinear scalings of basic measurements. This area remains the domain of human feature engineering. The idea of learning ranking functions has been around for a number of years, but it is only very recently that sufficient machine learning knowledge, training document collections, and computational power have come together to make this method practical and exciting. It is thus too early to write something definitive on machine learning approaches to ranking in information retrieval, but there is every reason to expect the use and importance of machine learned ranking approaches to grow over time. While skilled humans can do a very good job at defining ranking functions by hand, hand tuning is difficult, and it has to be done again for each new document collection and class of users.
Exercise 15.7 Plot the first 7 rows of Table 15.3 in the α-ω plane to produce a figure like that in Figure 15.7. Exercise 15.8 Write down the equation of a line in the α-ω plane separating the Rs from the Ns. Exercise 15.9 Give a training example (consisting of values for α, ω and the relevance judgment) that when added to the training set makes it impossible to separate the R's from the N's using a line in the α-ω plane.
References and further reading The somewhat quirky name support vector machine originates in the neural networks literature, where learning algorithms were thought of as architectures, and often referred to as "machines". The distinctive element of this model is that the decision boundary to use is completely decided ("supported") by a few training data points, the support vectors. For a more detailed presentation of SVMs, a good, well-known articlelength introduction is (Burges 1998). Chen et al. (2005) introduce the more recent ν-SVM, which provides an alternative parameterization for dealing with inseparable problems, whereby rather than specifying a penalty C, you specify a parameter ν which bounds the number of examples which can appear on the wrong side of the decision surface. There are now also several books dedicated to SVMs, large margin learning, and kernels: (Cristianini and Shawe-Taylor 2000) and (Schölkopf and Smola 2001) are more mathematically oriented, while (Shawe-Taylor and Cristianini 2004) aims to be more practical. For the foundations by their originator, see (Vapnik 1998).
Some recent, more general books on statistical learning, such as (Hastie et al. 2001) also give thorough coverage of SVMs. The construction of multiclass SVMs is discussed in (Weston and Watkins 1999), (Crammer and Singer 2001), and (Tsochantaridis et al. 2005). The last reference provides an introduction to the general framework of structural SVMs. The kernel trick was first presented in (Aizerman et al. 1964). For more about string kernels and other kernels for structured data, see (Lodhi et al. 2002) and (Gaertner et al. 2002). The Advances in Neural Information Processing (NIPS) conferences have become the premier venue for theoretical machine learning work, such as on SVMs. Other venues such as SIGIR are much stronger on experimental methodology and using text-specific features to improve classifier effectiveness. A recent comparison of most current machine learning classifiers (though on problems rather different from typical text problems) can be found in (Caruana and Niculescu-Mizil 2006). (Li and Yang 2003), discussed in Section 13.6, is the most recent comparative evaluation of machine learning classifiers on text classification. Older examinations of classifiers on text problems can be found in (Yang 1999, Yang and Liu 1999, Dumais et al. 1998). Joachims (2002a) presents his work on SVMs applied to text problems in detail. Zhang and Oles (2001) present an insightful comparison of Naive Bayes, regularized logistic regression and SVM classifiers. Joachims (1999) discusses methods of making SVM learning practical over large text data sets. Joachims (2006a) improves on this work. A number of approaches to hierarchical classification have been developed in order to deal with the common situation where the classes to be assigned have a natural hierarchical organization (Koller and Sahami 1997, McCallum et al. 1998, Weigend et al. 1999, Dumais and Chen 2000). In a recent large study on scaling SVMs to the entire Yahoo! directory, Liu et al. (2005) conclude that hierarchical classification noticeably if still modestly outperforms flat classification. Classifier effectiveness remains limited by the very small number of training documents for many classes. For a more general approach that can be applied to modeling relations between classes, which may be arbitrary rather than simply the case of a hierarchy, see Tsochantaridis et al. (2005). Moschitti and Basili (2004) investigate the use of complex nominals, proper nouns and word senses as features in text classification. Dietterich (2002) overviews ensemble methods for classifier combination, while Schapire (2003) focuses particularly on boosting, which is applied to text classification in (Schapire and Singer 2000). Chapelle et al. (2006) present an introduction to work in semi-supervised methods, including in particular chapters on using EM for semi-supervised text classification (Nigam et al. 2006) and on transductive SVMs (Joachims
2006b). Sindhwani and Keerthi (2006) present a more efficient implementation of a transductive SVM for large data sets. Tong and Koller (2001) explore active learning with SVMs for text classification; Baldridge and Osborne (2004) point out that examples selected for annotation with one classifier in an active learning context may be no better than random examples when used with another classifier. Machine learning approaches to ranking for ad hoc retrieval were pioneered in (Wong et al. 1988), (Fuhr 1992), and (Gey 1994). But limited training data and poor machine learning techniques meant that these pieces of work achieved only middling results, and hence they only had limited impact at the time. Taylor et al. (2006) study using machine learning to tune the parameters of the BM25 family of ranking functions (Section 11.4.3, page 232) so as to maximize NDCG (Section 8.4, page 163). Machine learning approaches to ordinal regression appear in (Herbrich et al. 2000) and (Burges et al. 2005), and are applied to clickstream data in (Joachims 2002b). Cao et al. (2006) study how to make this approach effective in IR, and Qin et al. (2007) suggest an extension involving using multiple hyperplanes. Yue et al. (2007) study how to do ranking with a structural SVM approach, and in particular show how this construction can be effectively used to directly optimize for MAP (Section 8.4, page 158), rather than using surrogate measures like accuracy or area under the ROC curve. Geng et al. (2007) study feature selection for the ranking problem. Other approaches to learning to rank have also been shown to be effective for web search, such as (Burges et al. 2005, Richardson et al. 2006).
Flat clustering
Clustering algorithms group a set of documents into subsets or clusters. The algorithms' goal is to create clusters that are coherent internally, but clearly different from each other. In other words, documents within a cluster should be as similar as possible; and documents in one cluster should be as dissimilar as possible from documents in other clusters.
◮ Figure 16.1 An example of a data set with a clear cluster structure. UNSUPERVISED LEARNING
Clustering is the most common form of unsupervised learning. No supervision means that there is no human expert who has assigned documents to classes. In clustering, it is the distribution and makeup of the data that will determine cluster membership. A simple example is Figure 16.1. It is visually clear that there are three distinct clusters of points. This chapter and Chapter 17 introduce algorithms that find such clusters in an unsupervised fashion. The difference between clustering and classification may not seem great at first. After all, in both cases we have a partition of a set of documents into groups. But as we will see the two problems are fundamentally different. Classification is a form of supervised learning (Chapter 13, page 256): our goal is to replicate a categorical distinction that a human supervisor im-
HARD CLUSTERING SOFT CLUSTERING
16.1 CLUSTER HYPOTHESIS
poses on the data. In unsupervised learning, of which clustering is the most important example, we have no such teacher to guide us. The key input to a clustering algorithm is the distance measure. In Figure 16.1, the distance measure is distance in the 2D plane. This measure suggests three different clusters in the figure. In document clustering, the distance measure is often also Euclidean distance. Different distance measures give rise to different clusterings. Thus, the distance measure is an important means by which we can influence the outcome of clustering. Flat clustering creates a flat set of clusters without any explicit structure that would relate clusters to each other. Hierarchical clustering creates a hierarchy of clusters and will be covered in Chapter 17. Chapter 17 also addresses the difficult problem of labeling clusters automatically. A second important distinction can be made between hard and soft clustering algorithms. Hard clustering computes a hard assignment – each document is a member of exactly one cluster. The assignment of soft clustering algorithms is soft – a document's assignment is a distribution over all clusters. In a soft assignment, a document has fractional membership in several clusters. Latent semantic indexing, a form of dimensionality reduction, is a soft clustering algorithm (Chapter 18, page 417). This chapter motivates the use of clustering in information retrieval by introducing a number of applications (Section 16.1), defines the problem we are trying to solve in clustering (Section 16.2) and discusses measures for evaluating cluster quality (Section 16.3). It then describes two flat clustering algorithms, K-means (Section 16.4), a hard clustering algorithm, and the Expectation-Maximization (or EM) algorithm (Section 16.5), a soft clustering algorithm. K-means is perhaps the most widely used flat clustering algorithm due to its simplicity and efficiency. The EM algorithm is a generalization of K-means and can be applied to a large variety of document representations and distributions.
Clustering in information retrieval The cluster hypothesis states the fundamental assumption we make when using clustering in information retrieval. Cluster hypothesis. Documents in the same cluster behave similarly with respect to relevance to information needs. The hypothesis states that if there is a document from a cluster that is relevant to a search request, then it is likely that other documents from the same cluster are also relevant. This is because clustering puts together documents that share many terms. The cluster hypothesis essentially is the contiguity
16.1 Clustering in information retrieval
Application Search result clustering Scatter-Gather Collection clustering
What is clustered? search results (subsets of) collection collection
Language modeling
Cluster-based retrieval
◮ Table 16.1
SEARCH RESULT CLUSTERING
S CATTER -G ATHER
more effective information presentation to user alternative user interface: "search without typing" effective information presentation for exploratory browsing increased precision and/or recall higher efficiency: faster search
Figure 16.2 Figure 16.3 McKeown et al. (2002), http://news.google.com
Liu and Croft (2004) Salton (1971a)
Some applications of clustering in information retrieval.
hypothesis in Chapter 14 (page 289). In both cases, we posit that similar documents behave similarly with respect to relevance. Table 16.1 shows some of the main applications of clustering in information retrieval. They differ in the set of documents that they cluster – search results, collection or subsets of the collection – and the aspect of an information retrieval system they try to improve – user experience, user interface, effectiveness or efficiency of the search system. But they are all based on the basic assumption stated by the cluster hypothesis. The first application mentioned in Table 16.1 is search result clustering where by search results we mean the documents that were returned in response to a query. The default presentation of search results in information retrieval is a simple list. Users scan the list from top to bottom until they have found the information they are looking for. Instead, search result clustering clusters the search results, so that similar documents appear together. It is often easier to scan a few coherent groups than many individual documents. This is particularly useful if a search term has different word senses. The example in Figure 16.2 is jaguar. Three frequent senses on the web refer to the car, the animal and an Apple operating system. The Clustered Results panel returned by the Vivísimo search engine (http://vivisimo.com) can be a more effective user interface for understanding what is in the search results than a simple list of documents. A better user interface is also the goal of Scatter-Gather, the second application in Table 16.1. Scatter-Gather clusters the whole collection to get groups of documents that the user can select or gather. The selected groups are merged and the resulting set is again clustered. This process is repeated until a cluster of interest is found. An example is shown in Figure 16.3.
◮ Figure 16.2 Clustering of search results to improve recall. None of the top hits cover the animal sense of jaguar, but users can easily access it by clicking on the cat cluster in the Clustered Results panel on the left (third arrow from the top).
Automatically generated clusters like those in Figure 16.3 are not as neatly organized as a manually constructed hierarchical tree like the Open Directory at http://dmoz.org. Also, finding descriptive labels for clusters automatically is a difficult problem (Section 17.7, page 396). But cluster-based navigation is an interesting alternative to keyword searching, the standard information retrieval paradigm. This is especially true in scenarios where users prefer browsing over searching because they are unsure about which search terms to use. As an alternative to the user-mediated iterative clustering in Scatter-Gather, we can also compute a static hierarchical clustering of a collection that is not influenced by user interactions ("Collection clustering" in Table 16.1). Google News and its precursor, the Columbia NewsBlaster system, are examples of this approach. In the case of news, we need to frequently recompute the clustering to make sure that users can access the latest breaking stories. Clustering is well suited for access to a collection of news stories since news reading is not really search, but rather a process of selecting a subset of stories about recent events.
◮ Figure 16.3 An example of a user session in Scatter-Gather. A collection of New York Times news stories is clustered ("scattered") into eight clusters (top row). The user manually gathers three of these into a smaller collection International Stories and performs another scattering operation. This process repeats until a small cluster with relevant documents is found (e.g., Trinidad).
The fourth application of clustering exploits the cluster hypothesis directly for improving search results, based on a clustering of the entire collection. We use a standard inverted index to identify an initial set of documents that match the query, but we then add other documents from the same clusters even if they have low similarity to the query. For example, if the query is car and several car documents are taken from a cluster of automobile documents, then we can add documents from this cluster that use terms other than car (automobile, vehicle etc). This can increase recall since a group of documents with high mutual similarity is often relevant as a whole. More recently this idea has been used for language modeling. Equation (12.10), page 245, showed that to avoid sparse data problems in the language modeling approach to IR, the model of document d can be interpolated with a
collection model. But the collection contains many documents with terms untypical of d. By replacing the collection model with a model derived from d's cluster, we get more accurate estimates of the occurrence probabilities of terms in d. Clustering can also speed up search. As we saw in Section 6.3.2 (page 123) search in the vector space model amounts to finding the nearest neighbors to the query. The inverted index supports fast nearest-neighbor search for the standard IR setting. However, sometimes we may not be able to use an inverted index efficiently, e.g., in latent semantic indexing (Chapter 18). In such cases, we could compute the similarity of the query to every document, but this is slow. The cluster hypothesis offers an alternative: Find the clusters that are closest to the query and only consider documents from these clusters. Within this much smaller set, we can compute similarities exhaustively and rank documents in the usual way. Since there are many fewer clusters than documents, finding the closest cluster is fast; and since the documents matching a query are all similar to each other, they tend to be in the same clusters. While this algorithm is inexact, the expected decrease in search quality is small. This is essentially the application of clustering that was covered in Section 7.1.6 (page 141).
Exercise 16.1 Define two documents as similar if they have at least two proper names like Clinton or Sarkozy in common. Give an example of an information need and two documents, for which the cluster hypothesis does not hold for this notion of similarity. Exercise 16.2 Make up a simple one-dimensional example (i.e. points on a line) with two clusters where the inexactness of cluster-based retrieval shows up. In your example, retrieving clusters close to the query should do worse than direct nearest neighbor search.
Problem statement We can define the goal in hard flat clustering as follows. Given (i) a set of documents D = {d1 , . . . , d N }, (ii) a desired number of clusters K, and (iii) an objective function that evaluates the quality of a clustering, we want to compute an assignment γ : D → {1, . . . , K } that minimizes (or, in other cases, maximizes) the objective function. In most cases, we also demand that γ is surjective, i.e., that none of the K clusters is empty. The objective function is often defined in terms of similarity or distance between documents. Below, we will see that the objective in K-means clustering is to minimize the average distance between documents and their centroids or, equivalently, to maximize the similarity between documents and their centroids. The discussion of similarity measures and distance metrics
16.2 Problem statement
in Chapter 14 (page 291) also applies to this chapter. As in Chapter 14, we use both similarity and distance to talk about relatedness between documents. For documents, the type of similarity we want is usually topic similarity or high values on the same dimensions in the vector space model. For example, documents about China have high values on dimensions like Chinese, Beijing, and Mao whereas documents about the UK tend to have high values for London, Britain and Queen. We approximate topic similarity with cosine similarity or Euclidean distance in vector space (Chapter 6). If we intend to capture similarity of a type other than topic, for example, similarity of language, then a different representation may be appropriate. When computing topic similarity, stop words can be safely ignored, but they are important cues for separating clusters of English (in which the occurs frequently and la infrequently) and French documents (in which the occurs infrequently and la frequently).
PARTITIONAL CLUSTERING
EXHAUSTIVE
16.2.1 CARDINALITY
A note on terminology. An alternative definition of hard clustering is that a document can be a full member of more than one cluster. Partitional clustering always refers to a clustering where each document belongs to exactly one cluster. (But in a partitional hierarchical clustering (Chapter 17) all members of a cluster are of course also members of its parent.) On the definition of hard clustering that permits multiple membership, the difference between soft clustering and hard clustering is that membership values in hard clustering are either 0 or 1, whereas they can take on any non-negative value in soft clustering. Some researchers distinguish between exhaustive clusterings that assign each document to a cluster and non-exhaustive clusterings, in which some documents will be assigned to no cluster. Non-exhaustive clusterings in which each document is a member of either no cluster or one cluster are called exclusive. We define clustering to be exhaustive in this book.
Cardinality – the number of clusters A difficult issue in clustering is determining the number of clusters or cardinality of a clustering, which we denote by K. Often K is nothing more than a good guess based on experience or domain knowledge. But for K-means, we will also introduce a heuristic method for choosing K and an attempt to incorporate the selection of K into the objective function. Sometimes the application puts constraints on the range of K. For example, the Scatter-Gather interface in Figure 16.3 could not display more than about K = 10 clusters per layer because of the size and resolution of computer monitors in the early 1990s. Since our goal is to optimize an objective function, clustering is essentially
a search problem. The brute force solution would be to enumerate all possible clusterings and pick the best. However, there are exponentially many partitions, so this approach is not feasible.1 For this reason, most flat clustering algorithms refine an initial partitioning iteratively. If the search starts at an unfavorable initial point, we may miss the global optimum. Finding a good starting point is therefore another important problem we have to solve in flat clustering.
INTERNAL CRITERION OF QUALITY
EXTERNAL CRITERION OF QUALITY
Evaluation of clustering Typical objective functions in clustering formalize the goal of attaining high intra-cluster similarity (documents within a cluster are similar) and low intercluster similarity (documents from different clusters are dissimilar). This is an internal criterion for the quality of a clustering. But good scores on an internal criterion do not necessarily translate into good effectiveness in an application. An alternative to internal criteria is direct evaluation in the application of interest. For search result clustering, we may want to measure the time it takes users to find an answer with different clustering algorithms. This is the most direct evaluation, but it is expensive, especially if large user studies are necessary. As a surrogate for user judgments, we can use a set of classes in an evaluation benchmark or gold standard (see Section 8.5, page 164, and Section 13.6, page 279). The gold standard is ideally produced by human judges with a good level of inter-judge agreement (see Chapter 8, page 152). We can then compute an external criterion that evaluates how well the clustering matches the gold standard classes. For example, we may want to say that the optimal clustering of the search results for jaguar in Figure 16.2 consists of three classes corresponding to the three senses car, animal, and operating system. In this type of evaluation, we only use the partition provided by the gold standard, not the class labels. This section introduces four external criteria of clustering quality. Purity is a simple and transparent evaluation measure. Normalized mutual information can be information-theoretically interpreted. The Rand index penalizes both false positive and false negative decisions during clustering. The F measure in addition supports differential weighting of these two types of errors. To compute purity, each cluster is assigned to the class which is most frequent in the cluster, and then the accuracy of this assignment is measured by counting the number of correctly assigned documents and dividing by N. 1. An upper bound on the number of clusterings is K N /K!. The exact number of different partitions of N documents into K clusters is the Stirling number of the second kind. See http://mathworld.wolfram.com/StirlingNumberoftheSecondKind.html or Comtet (1974).
16.3 Evaluation of clustering
x o
o o ⋄
◮ Figure 16.4 Purity as an external evaluation criterion for cluster quality. Majority class and number of members of the majority class for the three clusters are: x, 5 (cluster 1); o, 4 (cluster 2); and ⋄, 3 (cluster 3). Purity is (1/17) × (5 + 4 + 3) ≈ 0.71.
lower bound maximum value for Figure 16.4
purity 0.0 1 0.71
NMI 0.0 1 0.36
RI 0.0 1 0.68
F5 0.0 1 0.46
◮ Table 16.2 The four external evaluation measures applied to the clustering in Figure 16.4.
Formally: (16.1)
NORMALIZED MUTUAL INFORMATION
purity(Ω, C ) =
|ωk ∩ c j | ∑ max j k
where Ω = {ω1 , ω2 , . . . , ωK } is the set of clusters and C = {c1 , c2 , . . . , c J } is the set of classes. We interpret ωk as the set of documents in ωk and c j as the set of documents in c j in Equation (16.1). We present an example of how to compute purity in Figure 16.4.2 Bad clusterings have purity values close to 0, a perfect clustering has a purity of 1. Purity is compared with the other three measures discussed in this chapter in Table 16.2. High purity is easy to achieve when the number of clusters is large – in particular, purity is 1 if each document gets its own cluster. Thus, we cannot use purity to trade off the quality of the clustering against the number of clusters. A measure that allows us to make this tradeoff is normalized mutual infor2. Recall our note of caution from Figure 14.2 (page 291) when looking at this and other 2D figures in this and the following chapter: these illustrations can be misleading because 2D projections of length-normalized vectors distort similarities and distances between points.
mation or NMI: (16.2)
I (Ω; C ) [ H (Ω) + H (C )]/2 I is mutual information (cf. Chapter 13, page 272): NMI(Ω, C ) =
I (Ω; C )
P(ωk ∩ c j )
∑ ∑ P(ωk ∩ c j ) log P(ωk ) P(c j ) k
∑∑ k
|ωk ∩ c j | N |ωk ∩ c j | log N |ωk ||c j |
where P(ωk ), P(c j ), and P(ωk ∩ c j ) are the probabilities of a document being in cluster ωk , class c j , and in the intersection of ωk and c j , respectively. Equation (16.4) is equivalent to Equation (16.3) for maximum likelihood estimates of the probabilities (i.e., the estimate of each probability is the corresponding relative frequency). H is entropy as defined in Chapter 5 (page 99): (16.5)
H (Ω)
= − ∑ P(ωk ) log P(ωk ) k
= −∑ k
|ωk | |ω | log k N N
where, again, the second equation is based on maximum likelihood estimates of the probabilities. I (Ω; C ) in Equation (16.3) measures the amount of information by which our knowledge about the classes increases when we are told what the clusters are. The minimum of I (Ω; C ) is 0 if the clustering is random with respect to class membership. In that case, knowing that a document is in a particular cluster does not give us any new information about what its class might be. Maximum mutual information is reached for a clustering Ωexact that perfectly recreates the classes – but also if clusters in Ωexact are further subdivided into smaller clusters (Exercise 16.7). In particular, a clustering with K = N onedocument clusters has maximum MI. So MI has the same problem as purity: it does not penalize large cardinalities and thus does not formalize our bias that, other things being equal, fewer clusters are better. The normalization by the denominator [ H (Ω) + H (C )]/2 in Equation (16.2) fixes this problem since entropy tends to increase with the number of clusters. For example, H (Ω) reaches its maximum log N for K = N, which ensures that NMI is low for K = N. Because NMI is normalized, we can use it to compare clusterings with different numbers of clusters. The particular form of the denominator is chosen because [ H (Ω) + H (C )]/2 is a tight upper bound on I (Ω; C ) (Exercise 16.8). Thus, NMI is always a number between 0 and 1.
R AND INDEX RI
An alternative to this information-theoretic interpretation of clustering is to view it as a series of decisions, one for each of the N ( N − 1)/2 pairs of documents in the collection. We want to assign two documents to the same cluster if and only if they are similar. A true positive (TP) decision assigns two similar documents to the same cluster, a true negative (TN) decision assigns two dissimilar documents to different clusters. There are two types of errors we can commit. A false positive (FP) decision assigns two dissimilar documents to the same cluster. A false negative (FN) decision assigns two similar documents to different clusters. The Rand index (RI) measures the percentage of decisions that are correct. That is, it is simply accuracy (Section 8.3, page 155). RI =
TP + TN TP + FP + FN + TN
As an example, we compute RI for Figure 16.4. We first compute TP + FP. The three clusters contain 6, 6, and 5 points, respectively, so the total number of "positives" or pairs of documents that are in the same cluster is: 6 6 5 TP + FP = + + = 40 2 2 2 Of these, the x pairs in cluster 1, the o pairs in cluster 2, the ⋄ pairs in cluster 3, and the x pair in cluster 3 are true positives: 2 3 4 5 = 20 + + + TP = 2 2 2 2 Thus, FP = 40 − 20 = 20. FN and TN are computed similarly, resulting in the following contingency table: Same class Different classes
Same cluster TP = 20 FP = 20
Different clusters FN = 24 TN = 72
RI is then (20 + 72)/(20 + 20 + 24 + 72) ≈ 0.68. The Rand index gives equal weight to false positives and false negatives. Separating similar documents is sometimes worse than putting pairs of dissimilar documents in the same cluster. We can use the F measure (Section 8.3, page 154) to penalize false negatives more strongly than false positives by selecting a value β > 1, thus giving more weight to recall. P=
TP TP + FP
R=
TP TP + FN
Fβ =
( β2 + 1) PR β2 P + R
Based on the numbers in the contingency table, P = 20/40 = 0.5 and R = 20/44 ≈ 0.455. This gives us F1 ≈ 0.48 for β = 1 and F5 ≈ 0.456 for β = 5. In information retrieval, evaluating clustering with F has the advantage that the measure is already familiar to the research community.
Exercise 16.3 Replace every point d in Figure 16.4 with two identical copies of d in the same class. (i) Is it less difficult, equally difficult or more difficult to cluster this set of 34 points as opposed to the 17 points in Figure 16.4? (ii) Compute purity, NMI, RI, and F5 for the clustering with 34 points. Which measures increase and which stay the same after doubling the number of points? (iii) Given your assessment in (i) and the results in (ii), which measures are best suited to compare the quality of the two clusterings?
K-means K-means is the most important flat clustering algorithm. Its objective is to minimize the average squared Euclidean distance (Chapter 6, page 131) of documents from their cluster centers where a cluster center is defined as the mean or centroid ~µ of the documents in a cluster ω:
~µ (ω ) =
RESIDUAL SUM OF SQUARES
The definition assumes that documents are represented as length-normalized vectors in a real-valued space in the familiar way. We used centroids for Rocchio classification in Chapter 14 (page 292). They play a similar role here. The ideal cluster in K-means is a sphere with the centroid as its center of gravity. Ideally, the clusters should not overlap. Our desiderata for classes in Rocchio classification were the same. The difference is that we have no labeled training set in clustering for which we know which documents should be in the same cluster. A measure of how well the centroids represent the members of their clusters is the residual sum of squares or RSS, the squared distance of each vector from its centroid summed over all vectors: RSSk =
∑ ~x ∈ ω k
1 ~x |ω | ~x∑ ∈ω
RSS =
|~x − ~µ(ωk )|2 K
∑ RSSk
RSS is the objective function in K-means and our goal is to minimize it. Since N is fixed, minimizing RSS is equivalent to minimizing the average squared distance, a measure of how well centroids represent their documents.
K-means
K- MEANS ({~x1 , . . . , ~x N }, K ) 1 (~s1 ,~s2 , . . . ,~sK ) ← S ELECT R ANDOM S EEDS({~x1 , . . . , ~x N }, K ) 2 for k ← 1 to K 3 do ~µk ← ~sk 4 while stopping criterion has not been met 5 do for k ← 1 to K 6 do ωk ← {} 7 for n ← 1 to N 8 do j ← arg minj′ |~µ j′ − ~xn | 9 ω j ← ω j ∪ {~xn } (reassignment of vectors) 10 for k ← 1 to K 11 do ~µk ← |ω1 | ∑~x∈ωk ~x (recomputation of centroids) k 12 return {~µ1 , . . . , ~µK } ◮ Figure 16.5 The K-means algorithm. For most IR applications, the vectors ~xn ∈ R M should be length-normalized. Alternative methods of seed selection and initialization are discussed on page 364.
The first step of K-means is to select as initial cluster centers K randomly selected documents, the seeds. The algorithm then moves the cluster centers around in space in order to minimize RSS. As shown in Figure 16.5, this is done iteratively by repeating two steps until a stopping criterion is met: reassigning documents to the cluster with the closest centroid; and recomputing each centroid based on the current members of its cluster. Figure 16.6 shows snapshots from nine iterations of the K-means algorithm for a set of points. The "centroid" column of Table 17.2 (page 397) shows examples of centroids. We can apply one of the following termination conditions. • A fixed number of iterations I has been completed. This condition limits the runtime of the clustering algorithm, but in some cases the quality of the clustering will be poor because of an insufficient number of iterations. • Assignment of documents to clusters (the partitioning function γ) does not change between iterations. Except for cases with a bad local minimum, this produces a good clustering, but runtimes may be unacceptably long. • Centroids ~µk do not change between iterations. This is equivalent to γ not changing (Exercise 16.5). • Terminate when RSS falls below a threshold. This criterion ensures that the clustering is of a desired quality after termination. In practice, we
b b b b b b b b b b b bb b b b b b b b bb bb bb bb b b b b b b b b
× × b
oo o o ++ +× × o ++ + + o ++ o oo + + o + + ++ + + + ++ + + + + + + + + 0 1 2 3 +4 5 6
recomputation/movement of ~µ's (iter. 1) 4 3 2 1 0
assignment of documents (iter. 1)
selection of seeds 4
b b b b b b b b b b b bb b b b b b b b bb bb bb bb b b b b b b b b b
++ + ++ ++ o ++ + + o ++ o oo + + + + o o oo + + ++ o + + o o + + o 0 1 2 3 o4 5 6
~µ's after convergence (iter. 9)
. . . .. .. . ... . . . .. . . . . . . . . . .. . .. . . . . . . . . 0 1 2 3 .4 5 6 movement of ~µ's in 9 iterations
◮ Figure 16.6 A K-means example for K = 2 in R 2 . The position of the two centroids (~µ's shown as X's in the top four panels) converges after nine iterations.
need to combine it with a bound on the number of iterations to guarantee termination. • Terminate when the decrease in RSS falls below a threshold θ. For small θ, this indicates that we are close to convergence. Again, we need to combine it with a bound on the number of iterations to prevent very long runtimes. We now show that K-means converges by proving that RSS monotonically decreases in each iteration. We will use decrease in the meaning decrease or does not change in this section. First, RSS decreases in the reassignment step since each vector is assigned to the closest centroid, so the distance it contributes to RSS decreases. Second, it decreases in the recomputation step because the new centroid is the vector ~v for which RSSk reaches its minimum. (16.8) (16.9)
RSSk (~v) ∂RSSk (~v) ∂vm
|~v − ~x |2 =
∑ ∑ ( v m − x m )2
~x ∈ ω k m=1
2( v m − x m )
where xm and vm are the mth components of their respective vectors. Setting the partial derivative to zero, we get: (16.10)
1 xm |ωk | ~x∑ ∈ω k
SINGLETON CLUSTER
which is the componentwise definition of the centroid. Thus, we minimize RSSk when the old centroid is replaced with the new centroid. RSS, the sum of the RSSk , must then also decrease during recomputation. Since there is only a finite set of possible clusterings, a monotonically decreasing algorithm will eventually arrive at a (local) minimum. Take care, however, to break ties consistently, e.g., by assigning a document to the cluster with the lowest index if there are several equidistant centroids. Otherwise, the algorithm can cycle forever in a loop of clusterings that have the same cost. While this proves the convergence of K-means, there is unfortunately no guarantee that a global minimum in the objective function will be reached. This is a particular problem if a document set contains many outliers, documents that are far from any other documents and therefore do not fit well into any cluster. Frequently, if an outlier is chosen as an initial seed, then no other vector is assigned to it during subsequent iterations. Thus, we end up with a singleton cluster (a cluster with only one document) even though there is probably a clustering with lower RSS. Figure 16.7 shows an example of a suboptimal clustering resulting from a bad choice of initial seeds.
d1 ×
◮ Figure 16.7 The outcome of clustering in K-means depends on the initial seeds. For seeds d2 and d5 , K-means converges to {{d1 , d2 , d3 }, {d4 , d5 , d6 }}, a suboptimal clustering. For seeds d2 and d3 , it converges to {{d1 , d2 , d4 , d5 }, {d3 , d6 }}, the global optimum for K = 2.
Another type of suboptimal clustering that frequently occurs is one with empty clusters (Exercise 16.11). Effective heuristics for seed selection include (i) excluding outliers from the seed set; (ii) trying out multiple starting points and choosing the clustering with lowest cost; and (iii) obtaining seeds from another method such as hierarchical clustering. Since deterministic hierarchical clustering methods are more predictable than K-means, a hierarchical clustering of a small random sample of size iK (e.g., for i = 5 or i = 10) often provides good seeds (see the description of the Buckshot algorithm, Chapter 17, page 399). Other initialization methods compute seeds that are not selected from the vectors to be clustered. A robust method that works well for a large variety of document distributions is to select i (e.g., i = 10) random vectors for each cluster and use their centroid as the seed for this cluster. See Section 16.6 for more sophisticated initializations. What is the time complexity of K-means? Most of the time is spent on computing vector distances. One such operation costs Θ( M ). The reassignment step computes KN distances, so its overall complexity is Θ(KN M ). In the recomputation step, each vector gets added to a centroid once, so the complexity of this step is Θ( N M ). For a fixed number of iterations I, the overall complexity is therefore Θ( IKN M ). Thus, K-means is linear in all relevant factors: iterations, number of clusters, number of vectors and dimensionality of the space. This means that K-means is more efficient than the hierarchical algorithms in Chapter 17. We had to fix the number of iterations I, which can be tricky in practice. But in most cases, K-means quickly reaches either complete convergence or a clustering that is close to convergence. In the latter case, a few documents would switch membership if further iterations were computed, but this has a small effect on the overall quality of the clustering.
K- MEDOIDS MEDOID
There is one subtlety in the preceding argument. Even a linear algorithm can be quite slow if one of the arguments of Θ(. . .) is large, and M usually is large. High dimensionality is not a problem for computing the distance between two documents. Their vectors are sparse, so that only a small fraction of the theoretically possible M componentwise differences need to be computed. Centroids, however, are dense since they pool all terms that occur in any of the documents of their clusters. As a result, distance computations are time consuming in a naive implementation of K-means. However, there are simple and effective heuristics for making centroid-document similarities as fast to compute as document-document similarities. Truncating centroids to the most significant k terms (e.g., k = 1000) hardly decreases cluster quality while achieving a significant speedup of the reassignment step (see references in Section 16.6). The same efficiency problem is addressed by K-medoids, a variant of Kmeans that computes medoids instead of centroids as cluster centers. We define the medoid of a cluster as the document vector that is closest to the centroid. Since medoids are sparse document vectors, distance computations are fast.
Cluster cardinality in K-means We stated in Section 16.2 that the number of clusters K is an input to most flat clustering algorithms. What do we do if we cannot come up with a plausible guess for K? A naive approach would be to select the optimal value of K according to the objective function, namely the value of K that minimizes RSS. Defining RSSmin (K ) as the minimal RSS of all clusterings with K clusters, we observe that RSSmin (K ) is a monotonically decreasing function in K (Exercise 16.13), which reaches its minimum 0 for K = N where N is the number of documents. We would end up with each document being in its own cluster. Clearly, this is not an optimal clustering. A heuristic method that gets around this problem is to estimate RSSmin (K ) as follows. We first perform i (e.g., i = 10) clusterings with K clusters (each with a different initialization) and compute the RSS of each. Then we take the d min (K ). Now minimum of the i RSS values. We denote this minimum by RSS d min (K ) as K increases and find the "knee" in the we can inspect the values RSS d min become noticeably curve – the point where successive decreases in RSS smaller. There are two such points in Figure 16.8, one at K = 4, where the gradient flattens slightly, and a clearer flattening at K = 9. This is typical: there is seldom a single best number of clusters. We still need to employ an external constraint to choose from a number of possible values of K (4 and 9 in this case).
number of clusters
◮ Figure 16.8 Estimated minimal residual sum of squares as a function of the number of clusters in K-means. In this clustering of 1203 Reuters-RCV1 documents, there d min curve flattens: at 4 clusters and at 9 clusters. The are two points where the RSS documents were selected from the categories China, Germany, Russia and Sports, so the K = 4 clustering is closest to the Reuters classification.
DISTORTION MODEL COMPLEXITY
A second type of criterion for cluster cardinality imposes a penalty for each new cluster – where conceptually we start with a single cluster containing all documents and then search for the optimal number of clusters K by successively incrementing K by one. To determine the cluster cardinality in this way, we create a generalized objective function that combines two elements: distortion, a measure of how much documents deviate from the prototype of their clusters (e.g., RSS for K-means); and a measure of model complexity. We interpret a clustering here as a model of the data. Model complexity in clustering is usually the number of clusters or a function thereof. For K-means, we then get this selection criterion for K: K = arg min[RSSmin (K ) + λK ] K
where λ is a weighting factor. A large value of λ favors solutions with few clusters. For λ = 0, there is no penalty for more clusters and K = N is the best solution.
A KAIKE I NFORMATION C RITERION
The obvious difficulty with Equation (16.11) is that we need to determine λ. Unless this is easier than determining K directly, then we are back to square one. In some cases, we can choose values of λ that have worked well for similar data sets in the past. For example, if we periodically cluster news stories from a newswire, there is likely to be a fixed value of λ that gives us the right K in each successive clustering. In this application, we would not be able to determine K based on past experience since K changes. A theoretical justification for Equation (16.11) is the Akaike Information Criterion or AIC, an information-theoretic measure that trades off distortion against model complexity. The general form of AIC is: AIC:
K = arg min[−2L(K ) + 2q(K )] K
where − L(K ), the negative maximum log-likelihood of the data for K clusters, is a measure of distortion and q(K ), the number of parameters of a model with K clusters, is a measure of model complexity. We will not attempt to derive the AIC here, but it is easy to understand intuitively. The first property of a good model of the data is that each data point is modeled well by the model. This is the goal of low distortion. But models should also be small (i.e., have low model complexity) since a model that merely describes the data (and therefore has zero distortion) is worthless. AIC provides a theoretical justification for one particular way of weighting these two factors, distortion and model complexity, when selecting a model. For K-means, the AIC can be stated as follows: (16.13)
AIC:
K = arg min[RSSmin (K ) + 2MK ] K
Equation (16.13) is a special case of Equation (16.11) for λ = 2M. To derive Equation (16.13) from Equation (16.12) observe that q(K ) = KM in K-means since each element of the K centroids is a parameter that can be varied independently; and that L(K ) = −(1/2)RSSmin (K ) (modulo a constant) if we view the model underlying K-means as a Gaussian mixture with hard assignment, uniform cluster priors and identical spherical covariance matrices (see Exercise 16.19). The derivation of AIC is based on a number of assumptions, e.g., that the data are independent and identically distributed. These assumptions are only approximately true for data sets in information retrieval. As a consequence, the AIC can rarely be applied without modification in text clustering. In Figure 16.8, the dimensionality of the vector space is M ≈ 50,000. Thus, d min (1) < 5000, 2MK > 50,000 dominates the smaller RSS-based term (RSS not shown in the figure) and the minimum of the expression is reached for K = 1. But as we know, K = 4 (corresponding to the four classes China,
Germany, Russia and Sports) is a better choice than K = 1. In practice, Equation (16.11) is often more useful than Equation (16.13) – with the caveat that we need to come up with an estimate for λ.
Exercise 16.4 Why are documents that do not use the same term for the concept car likely to end up in the same cluster in K-means clustering? Exercise 16.5 Two of the possible termination conditions for K-means were (1) assignment does not change, (2) centroids do not change (page 361). Do these two conditions imply each other?
MODEL - BASED CLUSTERING
Model-based clustering In this section, we describe a generalization of K-means, the EM algorithm. It can be applied to a larger variety of document representations and distributions than K-means. In K-means, we attempt to find centroids that are good representatives. We can view the set of K centroids as a model that generates the data. Generating a document in this model consists of first picking a centroid at random and then adding some noise. If the noise is normally distributed, this procedure will result in clusters of spherical shape. Model-based clustering assumes that the data were generated by a model and tries to recover the original model from the data. The model that we recover from the data then defines clusters and an assignment of documents to clusters. A commonly used criterion for estimating the model parameters is maximum likelihood. In K-means, the quantity exp(−RSS) is proportional to the likelihood that a particular model (i.e., a set of centroids) generated the data. For K-means, maximum likelihood and minimal RSS are equivalent criteria. We denote the model parameters by Θ. In K-means, Θ = {~µ1 , . . . , ~µK }. More generally, the maximum likelihood criterion is to select the parameters Θ that maximize the log-likelihood of generating the data D: N
Θ = arg max L( D |Θ) = arg max log ∏ P(dn |Θ) = arg max Θ
n =1
∑ log P(dn |Θ)
L( D |Θ) is the objective function that measures the goodness of the clustering. Given two clusterings with the same number of clusters, we prefer the one with higher L( D |Θ). This is the same approach we took in Chapter 12 (page 237) for language modeling and in Section 13.1 (page 265) for text classification. In text classification, we chose the class that maximizes the likelihood of generating a particular document. Here, we choose the clustering Θ that maximizes the
16.5 Model-based clustering
E XPECTATION M AXIMIZATION ALGORITHM
likelihood of generating a given set of documents. Once we have Θ, we can compute an assignment probability P(d|ωk ; Θ) for each document-cluster pair. This set of assignment probabilities defines a soft clustering. An example of a soft assignment is that a document about Chinese cars may have a fractional membership of 0.5 in each of the two clusters China and automobiles, reflecting the fact that both topics are pertinent. A hard clustering like K-means cannot model this simultaneous relevance to two topics. Model-based clustering provides a framework for incorporating our knowledge about a domain. K-means and the hierarchical algorithms in Chapter 17 make fairly rigid assumptions about the data. For example, clusters in K-means are assumed to be spheres. Model-based clustering offers more flexibility. The clustering model can be adapted to what we know about the underlying distribution of the data, be it Bernoulli (as in the example in Table 16.3), Gaussian with non-spherical variance (another model that is important in document clustering) or a member of a different family. A commonly used algorithm for model-based clustering is the ExpectationMaximization algorithm or EM algorithm. EM clustering is an iterative algorithm that maximizes L( D |Θ). EM can be applied to many different types of probabilistic modeling. We will work with a mixture of multivariate Bernoulli distributions here, the distribution we know from Section 11.3 (page 222) and Section 13.3 (page 263): ! ! P ( d |ωk ; Θ) =
qmk
tm ∈d
∏ (1 − qmk )
tm ∈ /d
where Θ = {Θ1 , . . . , ΘK }, Θk = (αk , q1k , . . . , q Mk ), and qmk = P(Um = 1|ωk ) are the parameters of the model.3 P(Um = 1|ωk ) is the probability that a document from cluster ωk contains term tm . The probability αk is the prior of cluster ωk : the probability that a document d is in ωk if we have no information about d. The mixture model then is: ! ! K
P ( d |Θ) =
∑ αk ∏
In this model, we generate a document by first picking a cluster k with probability αk and then generating the terms of the document according to the parameters qmk . Recall that the document representation of the multivariate Bernoulli is a vector of M Boolean values (and not a real-valued vector). 3. Um is the random variable we defined in Section 13.3 (page 266) for the Bernoulli Naive Bayes model. It takes the values 1 (term tm is present in the document) and 0 (term tm is absent in the document).
EXPECTATION STEP MAXIMIZATION STEP
How do we use EM to infer the parameters of the clustering from the data? That is, how do we choose parameters Θ that maximize L( D |Θ)? EM is similar to K-means in that it alternates between an expectation step, corresponding to reassignment, and a maximization step, corresponding to recomputation of the parameters of the model. The parameters of K-means are the centroids, the parameters of the instance of EM in this section are the αk and qmk . The maximization step recomputes the conditional parameters qmk and the priors αk as follows:
Maximization step: qmk =
∑nN=1 rnk I (tm ∈ dn ) ∑nN=1 rnk
αk =
∑nN=1 rnk N
where I (tm ∈ dn ) = 1 if tm ∈ dn and 0 otherwise and rnk is the soft assignment of document d n to cluster k as computed in the preceding iteration. (We'll address the issue of initialization in a moment.) These are the maximum likelihood estimates for the parameters of the multivariate Bernoulli from Table 13.3 (page 268) except that documents are assigned fractionally to clusters here. These maximum likelihood estimates maximize the likelihood of the data given the model. The expectation step computes the soft assignment of documents to clusters given the current parameters qmk and αk : (16.17)
Expectation step :
rnk =
αk (∏tm ∈dn qmk )(∏tm ∈/ dn (1 − qmk )) K ∑k=1 αk (∏tm ∈dn qmk )(∏tm ∈/ dn (1 − qmk ))
This expectation step applies Equations (16.14) and (16.15) to computing the likelihood that ωk generated document dn . It is the classification procedure for the multivariate Bernoulli in Table 13.3. Thus, the expectation step is nothing else but Bernoulli Naive Bayes classification (including normalization, i.e. dividing by the denominator, to get a probability distribution over clusters). We clustered a set of 11 documents into two clusters using EM in Table 16.3. After convergence in iteration 25, the first 5 documents are assigned to cluster 1 (ri,1 = 1.00) and the last 6 to cluster 2 (ri,1 = 0.00). Somewhat atypically, the final assignment is a hard assignment here. EM usually converges to a soft assignment. In iteration 25, the prior α1 for cluster 1 is 5/11 ≈ 0.45 because 5 of the 11 documents are in cluster 1. Some terms are quickly associated with one cluster because the initial assignment can "spread" to them unambiguously. For example, membership in cluster 2 spreads from document 7 to document 8 in the first iteration because they share sugar (r8,1 = 0 in iteration 1). For parameters of terms occurring in ambiguous contexts, convergence takes longer. Seed documents 6 and 7
docID 1 2 3 4 5 6
document text hot chocolate cocoa beans cocoa ghana africa beans harvest ghana cocoa butter butter truffles sweet chocolate
Parameter 0 α1 r1,1 r2,1 r3,1 r4,1 r5,1 r6,1 r7,1 r8,1 r9,1 r10,1 r11,1 qafrica,1 qafrica,2 qbrazil,1 qbrazil,2 qcocoa,1 qcocoa,2 qsugar,1 qsugar,2 qsweet,1 qsweet,2
1 0.50 1.00 0.50 0.50 0.50 0.50 1.00 0.00 0.00 0.00 0.50 0.50 0.000 0.000 0.000 0.000 0.000 0.000 0.000 1.000 1.000 1.000
docID 7 8 9 10 11
document text sweet sugar sugar cane brazil sweet sugar beet sweet cake icing cake black forest
Iteration of clustering 2 3 4 5 0.45 0.53 0.57 0.58 1.00 1.00 1.00 1.00 0.79 0.99 1.00 1.00 0.84 1.00 1.00 1.00 0.75 0.94 1.00 1.00 0.52 0.66 0.91 1.00 1.00 1.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.40 0.14 0.01 0.00 0.57 0.58 0.41 0.07 0.100 0.134 0.158 0.158 0.083 0.042 0.001 0.000 0.000 0.000 0.000 0.000 0.167 0.195 0.213 0.214 0.400 0.432 0.465 0.474 0.167 0.090 0.014 0.001 0.000 0.000 0.000 0.000 0.500 0.585 0.640 0.642 0.300 0.238 0.180 0.159 0.417 0.507 0.610 0.640
15 0.54 1.00 1.00 1.00 1.00 1.00 0.83 0.00 0.00 0.00 0.00 0.00 0.169 0.000 0.000 0.196 0.508 0.000 0.000 0.589 0.153 0.608
◮ Table 16.3 The EM clustering algorithm. The table shows a set of documents (a) and parameter values for selected iterations during EM clustering (b). Parameters shown are prior α1 , soft assignment scores rn,1 (both omitted for cluster 2), and lexical parameters q m,k for a few terms. The authors initially assigned document 6 to cluster 1 and document 7 to cluster 2 (iteration 0). EM converges after 25 iterations. For smoothing, the rnk in Equation (16.16) were replaced with rnk + ǫ where ǫ = 0.0001.
both contain sweet. As a result, it takes 25 iterations for the term to be unambiguously associated with cluster 2. (qsweet,1 = 0 in iteration 25.) Finding good seeds is even more critical for EM than for K-means. EM is prone to get stuck in local optima if the seeds are not chosen well. This is a general problem that also occurs in other applications of EM.4 Therefore, as with K-means, the initial assignment of documents to clusters is often computed by a different algorithm. For example, a hard K-means clustering may provide the initial assignment, which EM can then "soften up."
Exercise 16.6 We saw above that the time complexity of K-means is Θ ( IKNM ). What is the time complexity of EM?
References and further reading Berkhin (2006b) gives a general up-to-date survey of clustering methods with special attention to scalability. The classic reference for clustering in pattern recognition, covering both K-means and EM, is (Duda et al. 2000). Rasmussen (1992) introduces clustering from an information retrieval perspective. Anderberg (1973) provides a general introduction to clustering for applications. In addition to Euclidean distance and cosine similarity, KullbackLeibler divergence is often used in clustering as a measure of how (dis)similar documents and clusters are (Xu and Croft 1999, Muresan and Harper 2004, Kurland and Lee 2004). The cluster hypothesis is due to Jardine and van Rijsbergen (1971) who state it as follows: Associations between documents convey information about the relevance of documents to requests. Salton (1971a; 1975), Croft (1978), Voorhees (1985a), Can and Ozkarahan (1990), Cacheda et al. (2003), Can et al. (2004), Singitham et al. (2004) and Altingövde et al. (2008) investigate the efficiency and effectiveness of cluster-based retrieval. While some of these studies show improvements in effectiveness, efficiency or both, there is no consensus that cluster-based retrieval works well consistently across scenarios. Clusterbased language modeling was pioneered by Liu and Croft (2004). There is good evidence that clustering of search results improves user experience and search result quality (Hearst and Pedersen 1996, Zamir and Etzioni 1999, Tombros et al. 2002, Käki 2005, Toda and Kataoka 2005), although not as much as search result structuring based on carefully edited category hierarchies (Hearst 2006). The Scatter-Gather interface for browsing collections was presented by Cutting et al. (1992). A theoretical framework for an4. For example, this problem is common when EM is used to estimate parameters of hidden Markov models, probabilistic grammars, and machine translation models in natural language processing (Manning and Schütze 1999).
ADJUSTED R AND INDEX
alyzing the properties of Scatter/Gather and other information seeking user interfaces is presented by Pirolli (2007). Schütze and Silverstein (1997) evaluate LSI (Chapter 18) and truncated representations of centroids for efficient K-means clustering. The Columbia NewsBlaster system (McKeown et al. 2002), a forerunner to the now much more famous and refined Google News (http://news.google.com), used hierarchical clustering (Chapter 17) to give two levels of news topic granularity. See Hatzivassiloglou et al. (2000) for details, and Chen and Lin (2000) and Radev et al. (2001) for related systems. Other applications of clustering in information retrieval are duplicate detection (Yang and Callan (2006), Section 19.6, page 438), novelty detection (see references in Section 17.9, page 399) and metadata discovery on the semantic web (Alonso et al. 2006). The discussion of external evaluation measures is partially based on Strehl (2002). Dom (2002) proposes a measure Q0 that is better motivated theoretically than NMI. Q0 is the number of bits needed to transmit class memberships assuming cluster memberships are known. The Rand index is due to Rand (1971). Hubert and Arabie (1985) propose an adjusted Rand index that ranges between −1 and 1 and is 0 if there is only chance agreement between clusters and classes (similar to κ in Chapter 8, page 165). Basu et al. (2004) argue that the three evaluation measures NMI, Rand index and F measure give very similar results. Stein et al. (2003) propose expected edge density as an internal measure and give evidence that it is a good predictor of the quality of a clustering. Kleinberg (2002) and Meil˘a (2005) present axiomatic frameworks for comparing clusterings. Authors that are often credited with the invention of the K-means algorithm include Lloyd (1982) (first distributed in 1957), Ball (1965), MacQueen (1967), and Hartigan and Wong (1979). Arthur and Vassilvitskii (2006) investigate the worst-case complexity of K-means. Bradley and Fayyad (1998), Pelleg and Moore (1999) and Davidson and Satyanarayana (2003) investigate the convergence properties of K-means empirically and how it depends on initial seed selection. Dhillon and Modha (2001) compare K-means clusters with SVD-based clusters (Chapter 18). The K-medoid algorithm was presented by Kaufman and Rousseeuw (1990). The EM algorithm was originally introduced by Dempster et al. (1977). An in-depth treatment of EM is (McLachlan and Krishnan 1996). See Section 18.5 (page 417) for publications on latent analysis, which can also be viewed as soft clustering. AIC is due to Akaike (1974) (see also Burnham and Anderson (2002)). An alternative to AIC is BIC, which can be motivated as a Bayesian model selection procedure (Schwarz 1978). Fraley and Raftery (1998) show how to choose an optimal number of clusters based on BIC. An application of BIC to K-means is (Pelleg and Moore 2000). Hamerly and Elkan (2003) propose an alternative to BIC that performs better in their experiments. Another influential Bayesian approach for determining the number of clusters (simultane-
CO - CLUSTERING
ously with cluster assignment) is described by Cheeseman and Stutz (1996). Two methods for determining cardinality without external criteria are presented by Tibshirani et al. (2001). We only have space here for classical completely unsupervised clustering. An important current topic of research is how to use prior knowledge to guide clustering (e.g., Ji and Xu (2006)) and how to incorporate interactive feedback during clustering (e.g., Huang and Mitchell (2006)). Fayyad et al. (1998) propose an initialization for EM clustering. For algorithms that can cluster very large data sets in one scan through the data see Bradley et al. (1998). The applications in Table 16.1 all cluster documents. Other information retrieval applications cluster words (e.g., Crouch 1988), contexts of words (e.g., Schütze and Pedersen 1995) or words and documents simultaneously (e.g., Tishby and Slonim 2000, Dhillon 2001, Zha et al. 2001). Simultaneous clustering of words and documents is an example of co-clustering or biclustering.
Exercise 16.7 Let Ω be a clustering that exactly reproduces a class structure C and Ω′ a clustering that further subdivides some clusters in Ω. Show that I (Ω; C ) = I (Ω′ ; C ). Exercise 16.8 Show that I (Ω; C ) ≤ [ H (Ω) + H (C )] /2. Exercise 16.9 Mutual information is symmetric in the sense that its value does not change if the roles of clusters and classes are switched: I (Ω; C ) = I (C; Ω). Which of the other three evaluation measures are symmetric in this sense? Exercise 16.10 Compute RSS for the two clusterings in Figure 16.7. Exercise 16.11 (i) Give an example of a set of points and three initial centroids (which need not be members of the set of points) for which 3-means converges to a clustering with an empty cluster. (ii) Can a clustering with an empty cluster be the global optimum with respect to RSS? Exercise 16.12 Download Reuters-21578. Discard documents that do not occur in one of the 10 classes acquisitions, corn, crude, earn, grain, interest, money-fx, ship, trade, and wheat. Discard documents that occur in two of these 10 classes. (i) Compute a K-means clustering of this subset into 10 clusters. There are a number of software packages that implement K-means, such as WEKA (Witten and Frank 2005) and R (R Development Core Team 2005). (ii) Compute purity, normalized mutual information, F1 and RI for
the clustering with respect to the 10 classes. (iii) Compile a confusion matrix (Table 14.5, page 308) for the 10 classes and 10 clusters. Identify classes that give rise to false positives and false negatives. Exercise 16.13 Prove that RSSmin (K ) is monotonically decreasing in K. Exercise 16.14 There is a soft version of K-means that computes the fractional membership of a document in a cluster as a monotonically decreasing function of the distance ∆ from its centroid, e.g., as e−∆ . Modify reassignment and recomputation steps of hard K-means for this soft version. Exercise 16.15 In the last iteration in Table 16.3, document 6 is in cluster 2 even though it was the initial seed for cluster 1. Why does the document change membership? Exercise 16.16 The values of the parameters q mk in iteration 25 in Table 16.3 are rounded. What are the exact values that EM will converge to? Exercise 16.17 Perform a K-means clustering for the documents in Table 16.3. After how many iterations does K-means converge? Compare the result with the EM clustering in Table 16.3 and discuss the differences. Exercise 16.18 [ ⋆ ⋆ ⋆] Modify the expectation and maximization steps of EM for a Gaussian mixture. The maximization step computes the maximum likelihood parameter estimates αk , ~µ k , and Σk for each of the clusters. The expectation step computes for each vector a soft assignment to clusters (Gaussians) based on their current parameters. Write down the equations for Gaussian mixtures corresponding to Equations (16.16) and (16.17). Exercise 16.19 [ ⋆ ⋆ ⋆] Show that K-means can be viewed as the limiting case of EM for Gaussian mixtures if variance is very small and all covariances are 0. [ ⋆ ⋆ ⋆]
Exercise 16.20 WITHIN - POINT SCATTER
The within-point scatter of a clustering is defined as ∑k 12
∑~xi ∈ωk ∑~x j ∈ωk |~xi −~x j that minimizing RSS and minimizing within-point scatter are equivalent.
|2 .
Exercise 16.21 [ ⋆ ⋆ ⋆] Derive an AIC criterion for the multivariate Bernoulli mixture model from Equation (16.12).
17 HIERARCHICAL CLUSTERING
Hierarchical clustering
Flat clustering is efficient and conceptually simple, but as we saw in Chapter 16 it has a number of drawbacks. The algorithms introduced in Chapter 16 return a flat unstructured set of clusters, require a prespecified number of clusters as input and are nondeterministic. Hierarchical clustering (or hierarchic clustering) outputs a hierarchy, a structure that is more informative than the unstructured set of clusters returned by flat clustering.1 Hierarchical clustering does not require us to prespecify the number of clusters and most hierarchical algorithms that have been used in IR are deterministic. These advantages of hierarchical clustering come at the cost of lower efficiency. The most common hierarchical clustering algorithms have a complexity that is at least quadratic in the number of documents compared to the linear complexity of K-means and EM (cf. Section 16.4, page 364). This chapter first introduces agglomerative hierarchical clustering (Section 17.1) and presents four different agglomerative algorithms, in Sections 17.2–17.4, which differ in the similarity measures they employ: single-link, completelink, group-average, and centroid similarity. We then discuss the optimality conditions of hierarchical clustering in Section 17.5. Section 17.6 introduces top-down (or divisive) hierarchical clustering. Section 17.7 looks at labeling clusters automatically, a problem that must be solved whenever humans interact with the output of clustering. We discuss implementation issues in Section 17.8. Section 17.9 provides pointers to further reading, including references to soft hierarchical clustering, which we do not cover in this book. There are few differences between the applications of flat and hierarchical clustering in information retrieval. In particular, hierarchical clustering is appropriate for any of the applications shown in Table 16.1 (page 351; see also Section 16.6, page 372). In fact, the example we gave for collection clustering is hierarchical. In general, we select flat clustering when efficiency is important and hierarchical clustering when one of the potential problems 1. In this chapter, we only consider hierarchies that are binary trees like the one shown in Figure 17.1 – but hierarchical clustering can be easily extended to other types of trees.
of flat clustering (not enough structure, predetermined number of clusters, non-determinism) is a concern. In addition, many researchers believe that hierarchical clustering produces better clusters than flat clustering. However, there is no consensus on this issue (see references in Section 17.9).
HIERARCHICAL AGGLOMERATIVE CLUSTERING
DENDROGRAM
COMBINATION SIMILARITY
Hierarchical agglomerative clustering Hierarchical clustering algorithms are either top-down or bottom-up. Bottomup algorithms treat each document as a singleton cluster at the outset and then successively merge (or agglomerate) pairs of clusters until all clusters have been merged into a single cluster that contains all documents. Bottomup hierarchical clustering is therefore called hierarchical agglomerative clustering or HAC. Top-down clustering requires a method for splitting a cluster. It proceeds by splitting clusters recursively until individual documents are reached. See Section 17.6. HAC is more frequently used in IR than top-down clustering and is the main subject of this chapter. Before looking at specific similarity measures used in HAC in Sections 17.2–17.4, we first introduce a method for depicting hierarchical clusterings graphically, discuss a few key properties of HACs and present a simple algorithm for computing an HAC. An HAC clustering is typically visualized as a dendrogram as shown in Figure 17.1. Each merge is represented by a horizontal line. The y-coordinate of the horizontal line is the similarity of the two clusters that were merged, where documents are viewed as singleton clusters. We call this similarity the combination similarity of the merged cluster. For example, the combination similarity of the cluster consisting of Lloyd's CEO questioned and Lloyd's chief / U.S. grilling in Figure 17.1 is ≈ 0.56. We define the combination similarity of a singleton cluster as its document's self-similarity (which is 1.0 for cosine similarity). By moving up from the bottom layer to the top node, a dendrogram allows us to reconstruct the history of merges that resulted in the depicted clustering. For example, we see that the two documents entitled War hero Colin Powell were merged first in Figure 17.1 and that the last merge added Ag trade reform to a cluster consisting of the other 29 documents. A fundamental assumption in HAC is that the merge operation is monotonic. Monotonic means that if s1 , s2 , . . . , sK −1 are the combination similarities of the successive merges of an HAC, then s1 ≥ s2 ≥ . . . ≥ sK −1 holds. A nonmonotonic hierarchical clustering contains at least one inversion si < si+1 and contradicts the fundamental assumption that we chose the best merge available at each step. We will see an example of an inversion in Figure 17.12. Hierarchical clustering does not require a prespecified number of clusters. However, in some applications we want a partition of disjoint clusters just as
Ag trade reform. Back−to−school spending is up Lloyd's CEO questioned Lloyd's chief / U.S. grilling Viag stays positive Chrysler / Latin America Ohio Blue Cross Japanese prime minister / Mexico CompuServe reports loss Sprint / Internet access service Planet Hollywood Trocadero: tripling of revenues German unions split War hero Colin Powell War hero Colin Powell Oil prices slip Chains may raise prices Clinton signs law Lawsuit against tobacco companies suits against tobacco firms Indiana tobacco lawsuit Most active stocks Mexican markets Hog prices tumble NYSE closing averages British FTSE index Fed holds interest rates steady Fed to keep interest rates steady Fed keeps interest rates steady Fed keeps interest rates steady
17.1 Hierarchical agglomerative clustering
◮ Figure 17.1 A dendrogram of a single-link clustering of 30 documents from Reuters-RCV1. Two possible cuts of the dendrogram are shown: at 0.4 into 24 clusters and at 0.1 into 12 clusters.
in flat clustering. In those cases, the hierarchy needs to be cut at some point. A number of criteria can be used to determine the cutting point: • Cut at a prespecified level of similarity. For example, we cut the dendrogram at 0.4 if we want clusters with a minimum combination similarity of 0.4. In Figure 17.1, cutting the diagram at y = 0.4 yields 24 clusters (grouping only documents with high similarity together) and cutting it at y = 0.1 yields 12 clusters (one large financial news cluster and 11 smaller clusters). • Cut the dendrogram where the gap between two successive combination similarities is largest. Such large gaps arguably indicate "natural" clusterings. Adding one more cluster decreases the quality of the clustering significantly, so cutting before this steep decrease occurs is desirable. This strategy is analogous to looking for the knee in the K-means graph in Figure 16.8 (page 366). • Apply Equation (16.11) (page 366): K = arg min[RSS(K ′ ) + λK ′ ] K′
where K ′ refers to the cut of the hierarchy that results in K ′ clusters, RSS is the residual sum of squares and λ is a penalty for each additional cluster. Instead of RSS, another measure of distortion can be used. • As in flat clustering, we can also prespecify the number of clusters K and select the cutting point that produces K clusters. A simple, naive HAC algorithm is shown in Figure 17.2. We first compute the N × N similarity matrix C. The algorithm then executes N − 1 steps of merging the currently most similar clusters. In each iteration, the two most similar clusters are merged and the rows and columns of the merged cluster i in C are updated.2 The clustering is stored as a list of merges in A. I indicates which clusters are still available to be merged. The function SIM (i, m, j) computes the similarity of cluster j with the merge of clusters i and m. For some HAC algorithms, SIM (i, m, j) is simply a function of C [ j][i ] and C [ j][m], for example, the maximum of these two values for single-link. We will now refine this algorithm for the different similarity measures of single-link and complete-link clustering (Section 17.2) and group-average and centroid clustering (Sections 17.3 and 17.4). The merge criteria of these four variants of HAC are shown in Figure 17.3. 2. We assume that we use a deterministic method for breaking ties, such as always choose the merge that is the first cluster with respect to a total ordering of the subsets of the document set D.
S IMPLE HAC(d1 , . . . , d N ) 1 for n ← 1 to N 2 do for i ← 1 to N 3 do C [n][i ] ← S IM (dn , di ) 4 I [n] ← 1 (keeps track of active clusters) 5 A ← [] (assembles clustering as a sequence of merges) 6 for k ← 1 to N − 1 7 do hi, mi ← arg max{hi,mi:i6=m∧ I [i]=1∧ I [m]=1} C [i ][m] 8 A.A PPEND (hi, mi) (store merge) 9 for j ← 1 to N 10 do C [i ][ j] ← S IM (i, m, j) 11 C [ j][i ] ← S IM (i, m, j) 12 I [m] ← 0 (deactivate cluster) 13 return A ◮ Figure 17.2 A simple, but inefficient HAC algorithm.
(a) single-link: maximum similarity
(b) complete-link: minimum similarity
(c) centroid: average inter-similarity (d) group-average: average of all similarities
◮ Figure 17.3 The different notions of cluster similarity used by the four HAC algorithms. An inter-similarity is a similarity between two documents from different clusters.
◮ Figure 17.4 A single-link (left) and complete-link (right) clustering of eight documents. The ellipses correspond to successive clustering stages. Left: The single-link similarity of the two upper two-point clusters is the similarity of d2 and d3 (solid line), which is greater than the single-link similarity of the two left two-point clusters (dashed line). Right: The complete-link similarity of the two upper two-point clusters is the similarity of d1 and d4 (dashed line), which is smaller than the complete-link similarity of the two left two-point clusters (solid line).
17.2 SINGLE - LINK CLUSTERING
COMPLETE - LINK CLUSTERING
Single-link and complete-link clustering In single-link clustering or single-linkage clustering, the similarity of two clusters is the similarity of their most similar members (see Figure 17.3, (a))3. This single-link merge criterion is local. We pay attention solely to the area where the two clusters come closest to each other. Other, more distant parts of the cluster and the clusters' overall structure are not taken into account. In complete-link clustering or complete-linkage clustering, the similarity of two clusters is the similarity of their most dissimilar members (see Figure 17.3, (b)). This is equivalent to choosing the cluster pair whose merge has the smallest diameter. This complete-link merge criterion is non-local; the entire structure of the clustering can influence merge decisions. This results in a preference for compact clusters with small diameters over long, straggly clusters, but also causes sensitivity to outliers. A single document far from the center can increase diameters of candidate merge clusters dramatically and completely change the final clustering. Figure 17.4 depicts a single-link and a complete-link clustering of eight documents. The first four steps, each producing a cluster consisting of a pair of two documents, are identical. Then single-link clustering joins the upper two pairs (and after that the lower two pairs) because on the maximumsimilarity definition of cluster similarity, those two clusters are closest. Complete3. Throughout this chapter, we equate similarity with proximity in 2D depictions of clustering.
NYSE closing averages Hog prices tumble Oil prices slip Ag trade reform. Chrysler / Latin America Japanese prime minister / Mexico Fed holds interest rates steady Fed to keep interest rates steady Fed keeps interest rates steady Fed keeps interest rates steady Mexican markets British FTSE index War hero Colin Powell War hero Colin Powell Lloyd's CEO questioned Lloyd's chief / U.S. grilling Ohio Blue Cross Lawsuit against tobacco companies suits against tobacco firms Indiana tobacco lawsuit Viag stays positive Most active stocks CompuServe reports loss Sprint / Internet access service Planet Hollywood Trocadero: tripling of revenues Back−to−school spending is up German unions split Chains may raise prices Clinton signs law
17.2 Single-link and complete-link clustering
◮ Figure 17.5 A dendrogram of a complete-link clustering. The same 30 documents were clustered with single-link clustering in Figure 17.1.
17 Hierarchical clustering ×
◮ Figure 17.6 Chaining in single-link clustering. The local criterion in single-link clustering can cause undesirable elongated clusters.
CONNECTED COMPONENT CLIQUE
link clustering joins the left two pairs (and then the right two pairs) because those are the closest pairs according to the minimum-similarity definition of cluster similarity.4 Figure 17.1 is an example of a single-link clustering of a set of documents and Figure 17.5 is the complete-link clustering of the same set. When cutting the last merge in Figure 17.5, we obtain two clusters of similar size (documents 1–16, from NYSE closing averages to Lloyd's chief / U.S. grilling, and documents 17–30, from Ohio Blue Cross to Clinton signs law). There is no cut of the dendrogram in Figure 17.1 that would give us an equally balanced clustering. Both single-link and complete-link clustering have graph-theoretic interpretations. Define sk to be the combination similarity of the two clusters merged in step k, and G (sk ) the graph that links all data points with a similarity of at least sk . Then the clusters after step k in single-link clustering are the connected components of G (sk ) and the clusters after step k in complete-link clustering are maximal cliques of G (sk ). A connected component is a maximal set of connected points such that there is a path connecting each pair. A clique is a set of points that are completely linked with each other. These graph-theoretic interpretations motivate the terms single-link and complete-link clustering. Single-link clusters at step k are maximal sets of points that are linked via at least one link (a single link) of similarity s ≥ sk ; complete-link clusters at step k are maximal sets of points that are completely linked with each other via links of similarity s ≥ sk . Single-link and complete-link clustering reduce the assessment of cluster quality to a single similarity between a pair of documents: the two most similar documents in single-link clustering and the two most dissimilar documents in complete-link clustering. A measurement based on one pair cannot fully reflect the distribution of documents in a cluster. It is therefore not surprising that both algorithms often produce undesirable clusters. Single-link clustering can produce straggling clusters as shown in Figure 17.6. Since the merge criterion is strictly local, a chain of points can be extended for long 4. If you are bothered by the possibility of ties, assume that d1 has coordinates (1 + ǫ, 3 − ǫ ) and that all other points have integer coordinates.
d1 1 0
d2 d3 d4 d5 ×
0 1 2 3 4 5 6 7 ◮ Figure 17.7 Outliers in complete-link clustering. The five documents have the x-coordinates 1 + 2ǫ, 4, 5 + 2ǫ, 6 and 7 − ǫ. Complete-link clustering creates the two clusters shown as ellipses. The most intuitive two-cluster clustering is {{d1 }, {d2 , d3 , d4 , d5 }}, but in complete-link clustering, the outlier d1 splits {d2 , d3 , d4 , d5 } as shown.
distances without regard to the overall shape of the emerging cluster. This effect is called chaining. The chaining effect is also apparent in Figure 17.1. The last eleven merges of the single-link clustering (those above the 0.1 line) add on single documents or pairs of documents, corresponding to a chain. The complete-link clustering in Figure 17.5 avoids this problem. Documents are split into two groups of roughly equal size when we cut the dendrogram at the last merge. In general, this is a more useful organization of the data than a clustering with chains. However, complete-link clustering suffers from a different problem. It pays too much attention to outliers, points that do not fit well into the global structure of the cluster. In the example in Figure 17.7 the four documents d2 , d3 , d4 , d5 are split because of the outlier d1 at the left edge (Exercise 17.1). Complete-link clustering does not find the most intuitive cluster structure in this example.
Time complexity of HAC The complexity of the naive HAC algorithm in Figure 17.2 is Θ( N 3 ) because we exhaustively scan the N × N matrix C for the largest similarity in each of N − 1 iterations. For the four HAC methods discussed in this chapter a more efficient algorithm is the priority-queue algorithm shown in Figure 17.8. Its time complexity is Θ( N 2 log N ). The rows C [k] of the N × N similarity matrix C are sorted in decreasing order of similarity in the priority queues P. P[k].MAX () then returns the cluster in P[k] that currently has the highest similarity with ωk , where we use ωk to denote the kth cluster as in Chapter 16. After creating the merged cluster of ωk1 and ωk2 , ωk1 is used as its representative. The function SIM computes the similarity function for potential merge pairs: largest similarity for single-link, smallest similarity for complete-link, average similarity for GAAC (Section 17.3), and centroid similarity for centroid clustering (Sec-
E FFICIENT HAC (d~1 , . . . , d~N ) 1 for n ← 1 to N 2 do for i ← 1 to N 3 do C [n][i ].sim ← d~n · d~i 4 C [n][i ].index ← i 5 I [n] ← 1 6 P[n] ← priority queue for C [n] sorted on sim 7 P[n].D ELETE(C [n][n]) (don't want self-similarities) 8 A ← [] 9 for k ← 1 to N − 1 10 do k1 ← arg max{k:I [k]=1} P[k].M AX ().sim 11 k2 ← P[k1 ].M AX ().index 12 A.A PPEND (hk1 , k2 i) 13 I [k2 ] ← 0 14 P[k1 ] ← [] 15 for each i with I [i ] = 1 ∧ i 6= k1 16 do P[i ].D ELETE(C [i ][k1 ]) 17 P[i ].D ELETE(C [i ][k2 ]) 18 C [i ][k1 ].sim ← S IM (i, k1 , k2 ) 19 P[i ].I NSERT(C [i ][k1 ]) 20 C [k1 ][i ].sim ← S IM (i, k1 , k2 ) 21 P[k1 ].I NSERT (C [k1 ][i ]) 22 return A clustering algorithm single-link complete-link centroid group-average
SIM (i, k 1 , k 2 ) max( SIM (i, k1 ), SIM(i, k2 )) min( SIM (i, k1 ), SIM(i, k2 )) ( N1m ~vm ) · ( N1i ~vi ) 1 [(~vm + ~vi )2 − ( Nm + Ni )] ( N + N )( N + N −1) m
compute C [5] create P[5] (by sorting) merge 2 and 3, update similarity of 2, delete 3 delete and reinsert 2
◮ Figure 17.8 The priority-queue algorithm for HAC. Top: The algorithm. Center: Four different similarity measures. Bottom: An example for processing steps 6 and 16–19. This is a made up example showing P [5] for a 5 × 5 matrix C.
S INGLE L INK C LUSTERING (d1 , . . . , d N ) 1 for n ← 1 to N 2 do for i ← 1 to N 3 do C [n][i ].sim ← SIM(dn , di ) 4 C [n][i ].index ← i 5 I [n] ← n 6 NBM [n] ← arg maxX ∈{C[n][i]:n6=i} X.sim 7 A ← [] 8 for n ← 1 to N − 1 9 do i1 ← arg max{i:I [i]=i} NBM [i ].sim 10 i2 ← I [ NBM [i1 ].index] 11 A.A PPEND (hi1 , i2 i) 12 for i ← 1 to N 13 do if I [i ] = i ∧ i 6= i1 ∧ i 6= i2 14 then C [i1 ][i ].sim ← C [i ][i1 ].sim ← max(C [i1 ][i ].sim, C [i2 ][i ].sim) 15 if I [i ] = i2 16 then I [i ] ← i1 17 NBM [i1 ] ← arg maxX ∈{C[i1][i]:I [i]=i∧i6=i1} X.sim 18 return A ◮ Figure 17.9 Single-link clustering algorithm using an NBM array. After merging two clusters i1 and i2 , the first one (i1 ) represents the merged cluster. If I [i ] = i, then i is the representative of its current cluster. If I [i ] 6= i, then i has been merged into the cluster represented by I [i ] and will therefore be ignored when updating NBM [i1 ].
tion 17.4). We give an example of how a row of C is processed (Figure 17.8, bottom panel). The loop in lines 1–7 is Θ( N 2 ) and the loop in lines 9–21 is Θ( N 2 log N ) for an implementation of priority queues that supports deletion and insertion in Θ(log N ). The overall complexity of the algorithm is therefore Θ( N 2 log N ). In the definition of the function SIM, ~vm and ~vi are the vector sums of ωk1 ∪ ωk2 and ωi , respectively, and Nm and Ni are the number of documents in ωk1 ∪ ωk2 and ωi , respectively. The argument of E FFICIENT HAC in Figure 17.8 is a set of vectors (as opposed to a set of generic documents) because GAAC and centroid clustering (Sections 17.3 and 17.4) require vectors as input. The complete-link version of E FFICIENT HAC can also be applied to documents that are not represented as vectors. For single-link, we can introduce a next-best-merge array (NBM) as a further optimization as shown in Figure 17.9. NBM keeps track of what the best merge is for each cluster. Each of the two top level for-loops in Figure 17.9 are Θ( N 2 ), thus the overall complexity of single-link clustering is Θ( N 2 ).
0 1 2 3 4 5 6 7 8 9 10 ◮ Figure 17.10 Complete-link clustering is not best-merge persistent. At first, d2 is the best-merge cluster for d3 . But after merging d1 and d2 , d4 becomes d3 's best-merge candidate. In a best-merge persistent algorithm like single-link, d3 's best-merge cluster would be {d1 , d2 }.
BEST- MERGE PERSISTENCE
? 17.3 GROUP - AVERAGE AGGLOMERATIVE CLUSTERING
Can we also speed up the other three HAC algorithms with an NBM array? We cannot because only single-link clustering is best-merge persistent. Suppose that the best merge cluster for ωk is ω j in single-link clustering. Then after merging ω j with a third cluster ωi 6= ωk , the merge of ωi and ω j will be ωk 's best merge cluster (Exercise 17.6). In other words, the best-merge candidate for the merged cluster is one of the two best-merge candidates of its components in single-link clustering. This means that C can be updated in Θ( N ) in each iteration – by taking a simple max of two values on line 14 in Figure 17.9 for each of the remaining ≤ N clusters. Figure 17.10 demonstrates that best-merge persistence does not hold for complete-link clustering, which means that we cannot use an NBM array to speed up clustering. After merging d3 's best merge candidate d2 with cluster d1 , an unrelated cluster d4 becomes the best merge candidate for d3 . This is because the complete-link merge criterion is non-local and can be affected by points at a great distance from the area where two merge candidates meet. In practice, the efficiency penalty of the Θ( N 2 log N ) algorithm is small compared with the Θ( N 2 ) single-link algorithm since computing the similarity between two documents (e.g., as a dot product) is an order of magnitude slower than comparing two scalars in sorting. All four HAC algorithms in this chapter are Θ( N 2 ) with respect to similarity computations. So the difference in complexity is rarely a concern in practice when choosing one of the algorithms. Exercise 17.1 Show that complete-link clustering creates the two-cluster clustering depicted in Figure 17.7.
Group-average agglomerative clustering Group-average agglomerative clustering or GAAC (see Figure 17.3, (d)) evaluates cluster quality based on all similarities between documents, thus avoiding the pitfalls of the single-link and complete-link criteria, which equate cluster
17.3 Group-average agglomerative clustering
similarity with the similarity of a single pair of documents. GAAC is also called group-average clustering and average-link clustering. GAAC computes the average similarity SIM - GA of all pairs of documents, including pairs from the same cluster. But self-similarities are not included in the average: (17.1)
SIM - GA ( ω i , ω j )
1 ( Ni + Nj )( Ni + Nj − 1) dm ∈∑ ω ∪ω i
∑ d n ∈ ω i ∪ ω j ,d n 6 =d m
d~m · d~n
where d~ is the length-normalized vector of document d, · denotes the dot product, and Ni and Nj are the number of documents in ωi and ω j , respectively. The motivation for GAAC is that our goal in selecting two clusters ωi and ω j as the next merge in HAC is that the resulting merge cluster ωk = ωi ∪ ω j should be coherent. To judge the coherence of ωk , we need to look at all document-document similarities within ωk , including those that occur within ωi and those that occur within ω j . We can compute the measure SIM - GA efficiently because the sum of individual vector similarities is equal to the similarities of their sums: (17.2)
∑ ∑ dm ∈ωi dn ∈ω j
(d~m · d~n ) = (
∑ dm ∈ωi
d~m ) · (
d~n )
∑ dn ∈ω j
With (17.2), we have: (17.3)
1 [( d~m )2 − ( Ni + Nj )] ( Ni + Nj )( Ni + Nj − 1) dm ∈∑ ω ∪ω i
The term ( Ni + Nj ) on the right is the sum of Ni + Nj self-similarities of value 1.0. With this trick we can compute cluster similarity in constant time (assuming we have available the two vector sums ∑dm ∈ωi d~m and ∑dm ∈ω j d~m ) instead of in Θ( Ni Nj ). This is important because we need to be able to compute the function SIM on lines 18 and 20 in E FFICIENT HAC (Figure 17.8) in constant time for efficient implementations of GAAC. Note that for two singleton clusters, Equation (17.3) is equivalent to the dot product. Equation (17.2) relies on the distributivity of the dot product with respect to vector addition. Since this is crucial for the efficient computation of a GAAC clustering, the method cannot be easily applied to representations of documents that are not real-valued vectors. Also, Equation (17.2) only holds for the dot product. While many algorithms introduced in this book have near-equivalent descriptions in terms of dot product, cosine similarity and Euclidean distance (cf. Section 14.1, page 291), Equation (17.2) can only be expressed using the dot product. This is a fundamental difference between single-link/complete-link clustering and GAAC. The first two only require a
square matrix of similarities as input and do not care how these similarities were computed. To summarize, GAAC requires (i) documents represented as vectors, (ii) length normalization of vectors, so that self-similarities are 1.0, and (iii) the dot product as the measure of similarity between vectors and sums of vectors. The merge algorithms for GAAC and complete-link clustering are the same except that we use Equation (17.3) as similarity function in Figure 17.8. Therefore, the overall time complexity of GAAC is the same as for complete-link clustering: Θ( N 2 log N ). Like complete-link clustering, GAAC is not bestmerge persistent (Exercise 17.6). This means that there is no Θ( N 2 ) algorithm for GAAC that would be analogous to the Θ( N 2 ) algorithm for single-link in Figure 17.9. We can also define group-average similarity as including self-similarities:
(17.4)SIM - GA′ (ωi , ω j ) =
1 ( ( Ni+Nj )2 d
∑ m ∈ωi ∪ω j
d~m )2 =
1 Ni+Nj
[d~m · ~µ(ωi ∪ ω j )]
dm ∈ωi ∪ω j
where the centroid ~µ (ω ) is defined as in Equation (14.1) (page 292). This definition is equivalent to the intuitive definition of cluster quality as average similarity of documents d~m to the cluster's centroid ~µ. Self-similarities are always equal to 1.0, the maximum possible value for length-normalized vectors. The proportion of self-similarities in Equation (17.4) is i/i2 = 1/i for a cluster of size i. This gives an unfair advantage to small clusters since they will have proportionally more self-similarities. For two documents d1 , d2 with a similarity s, we have SIM - GA ′ (d1 , d2 ) = (1 + s)/2. In contrast, SIM - GA(d1 , d2 ) = s ≤ (1 + s)/2. This similarity SIM - GA (d1 , d2 ) of two documents is the same as in single-link, complete-link and centroid clustering. We prefer the definition in Equation (17.3), which excludes selfsimilarities from the average, because we do not want to penalize large clusters for their smaller proportion of self-similarities and because we want a consistent similarity value s for document pairs in all four HAC algorithms.
Exercise 17.2 Apply group-average clustering to the points in Figures 17.6 and 17.7. Map them onto the surface of the unit sphere in a three-dimensional space to get length-normalized vectors. Is the group-average clustering different from the single-link and completelink clusterings?
17.4 Centroid clustering
bc µ2
bc c b × ×
µ3
d6 7
◮ Figure 17.11 Three iterations of centroid clustering. Each iteration merges the two clusters whose centroids are closest.
Centroid clustering In centroid clustering, the similarity of two clusters is defined as the similarity of their centroids:
SIM - CENT ( ω i , ω j )
= ~µ (ωi ) · ~µ (ω j ) 1 1 d~m ) · ( = ( Ni d ∑ N j ∈ω m
1 Ni Nj
dn ∈ω j
Equation (17.5) is centroid similarity. Equation (17.6) shows that centroid similarity is equivalent to average similarity of all pairs of documents from different clusters. Thus, the difference between GAAC and centroid clustering is that GAAC considers all pairs of documents in computing average pairwise similarity (Figure 17.3, (d)) whereas centroid clustering excludes pairs from the same cluster (Figure 17.3, (c)). Figure 17.11 shows the first three steps of a centroid clustering. The first two iterations form the clusters {d5 , d6 } with centroid µ1 and {d1 , d2 } with centroid µ2 because the pairs hd5 , d6 i and hd1 , d2 i have the highest centroid similarities. In the third iteration, the highest centroid similarity is between µ1 and d4 producing the cluster {d4 , d5 , d6 } with centroid µ3 . Like GAAC, centroid clustering is not best-merge persistent and therefore Θ( N 2 log N ) (Exercise 17.6). In contrast to the other three HAC algorithms, centroid clustering is not monotonic. So-called inversions can occur: Similarity can increase during
c b
−4 −3 −2 −1 0
◮ Figure 17.12 Centroid clustering is not monotonic. The documents d1 at (1 + ǫ, 1), √ d2 at (5, 1), and d3 at (3, 1 + 2 3) are almost equidistant, with d1 and d2 closer to each other than to d3 . The non-monotonic inversion in the hierarchical clustering of the three points appears as an intersecting merge line in the dendrogram. The intersection is circled.
clustering as in the example in Figure 17.12, where we define similarity as negative distance. In the first merge, the similarity of d1 and d2 is −(4 − ǫ). In the second merge, the similarity of the centroid of d1 and d2 (the circle) and d3 √ is ≈ − cos(π