  Various domain users are increasingly leveraging real-time social media data
to gain rapid situational awareness. However, due to the high noise in the
deluge of data, effectively determining semantically relevant information can
be difficult, further complicated by the changing definition of relevancy by
each end user for different events. The majority of existing methods for short
text relevance classification fail to incorporate users' knowledge into the
classification process. Existing methods that incorporate interactive user
feedback focus on historical datasets. Therefore, classifiers cannot be
interactively retrained for specific events or user-dependent needs in
real-time. This limits real-time situational awareness, as streaming data that
is incorrectly classified cannot be corrected immediately, permitting the
possibility for important incoming data to be incorrectly classified as well.
We present a novel interactive learning framework to improve the classification
process in which the user iteratively corrects the relevancy of tweets in
real-time to train the classification model on-the-fly for immediate predictive
improvements. We computationally evaluate our classification model adapted to
learn at interactive rates. Our results show that our approach outperforms
state-of-the-art machine learning models. In addition, we integrate our
framework with the extended Social Media Analytics and Reporting Toolkit
(SMART) 2.0 system, allowing the use of our interactive learning framework
within a visual analytics system tailored for real-time situational awareness.
To demonstrate our framework's effectiveness, we provide domain expert feedback
from first responders who used the extended SMART 2.0 system.
