  The New York Public Library is participating in the Chronicling America
initiative to develop an online searchable database of historically significant
newspaper articles. Microfilm copies of the newspapers are scanned and high
resolution Optical Character Recognition (OCR) software is run on them. The
text from the OCR provides a wealth of data and opinion for researchers and
historians. However, categorization of articles provided by the OCR engine is
rudimentary and a large number of the articles are labeled editorial without
further grouping. Manually sorting articles into fine-grained categories is
time consuming if not impossible given the size of the corpus. This paper
studies techniques for automatic categorization of newspaper articles so as to
enhance search and retrieval on the archive. We explore unsupervised (e.g.
KMeans) and semi-supervised (e.g. constrained clustering) learning algorithms
to develop article categorization schemes geared towards the needs of
end-users. A pilot study was designed to understand whether there was unanimous
agreement amongst patrons regarding how articles can be categorized. It was
found that the task was very subjective and consequently automated algorithms
that could deal with subjective labels were used. While the small scale pilot
study was extremely helpful in designing machine learning algorithms, a much
larger system needs to be developed to collect annotations from users of the
archive. The "BODHI" system currently being developed is a step in that
direction, allowing users to correct wrongly scanned OCR and providing keywords
and tags for newspaper articles used frequently. On successful implementation
of the beta version of this system, we hope that it can be integrated with
existing software being developed for the Chronicling America project.
