  This paper introduces an agent-centric approach to handle novelty in the
visual recognition domain of handwriting recognition (HWR). An ideal
transcription agent would rival or surpass human perception, being able to
recognize known and new characters in an image, and detect any stylistic
changes that may occur within or across documents. A key confound is the
presence of novelty, which has continued to stymie even the best machine
learning-based algorithms for these tasks. In handwritten documents, novelty
can be a change in writer, character attributes, writing attributes, or overall
document appearance, among other things. Instead of looking at each aspect
independently, we suggest that an integrated agent that can process known
characters and novelties simultaneously is a better strategy. This paper
formalizes the domain of handwriting recognition with novelty, describes a
baseline agent, introduces an evaluation protocol with benchmark data, and
provides experimentation to set the state-of-the-art. Results show feasibility
for the agent-centric approach, but more work is needed to approach
human-levels of reading ability, giving the HWR community a formal basis to
build upon as they solve this challenging problem.
