  Web archive collections are created with a particular purpose in mind. A
curator selects seeds, or original resources, which are then captured by an
archiving system and stored as archived web pages, or mementos. The systems
that build web archive collections are often configured to revisit the same
original resource multiple times. This is incredibly useful for understanding
an unfolding news story or the evolution of an organization. Unfortunately,
over time, some of these original resources can go off-topic and no longer suit
the purpose for which the collection was originally created. They can go
off-topic due to web site redesigns, changes in domain ownership, financial
issues, hacking, technical problems, or because their content has moved on from
the original topic. Even though they are off-topic, the archiving system will
still capture them, thus it becomes imperative to anyone performing research on
these collections to identify these off-topic mementos. Hence, we present the
Off-Topic Memento Toolkit, which allows users to detect off-topic mementos
within web archive collections. The mementos identified by this toolkit can
then be separately removed from a collection or merely excluded from downstream
analysis. The following similarity measures are available: byte count, word
count, cosine similarity, Jaccard distance, S{\o}rensen-Dice distance, Simhash
using raw text content, Simhash using term frequency, and Latent Semantic
Indexing via the gensim library. We document the implementation of each of
these similarity measures. We possess a gold standard dataset generated by
manual analysis, which contains both off-topic and on-topic mementos. Using
this gold standard dataset, we establish a default threshold corresponding to
the best F1 score for each measure. We also provide an overview of potential
future directions that the toolkit may take.
