  Filler words like ``um" or ``uh" are common in spontaneous speech. It is
desirable to automatically detect and remove them in recordings, as they affect
the fluency, confidence, and professionalism of speech. Previous studies and
our preliminary experiments reveal that the biggest challenge in filler word
detection is that fillers can be easily confused with other hard categories
like ``a" or ``I". In this paper, we propose a novel filler word detection
method that effectively addresses this challenge by adding auxiliary categories
dynamically and applying an additional inter-category focal loss. The auxiliary
categories force the model to explicitly model the confusing words by mining
hard categories. In addition, inter-category focal loss adaptively adjusts the
penalty weight between ``filler" and ``non-filler" categories to deal with
other confusing words left in the ``non-filler" category. Our system achieves
the best results, with a huge improvement compared to other methods on the
PodcastFillers dataset.
