  An essential part of building a data-driven organization is the ability to
handle and process continuous streams of data to discover actionable insights.
The explosive growth of interconnected devices and the social Web has led to a
large volume of data being generated on a continuous basis. Streaming data
sources such as stock quotes, credit card transactions, trending news, traffic
conditions, time-sensitive patients data are not only very common but can
rapidly depreciate if not processed quickly. The ever-increasing volume and
highly irregular nature of data rates pose new challenges to data stream
processing systems. One such challenging but important task is how to
accurately ingest and integrate data streams from various sources and locations
into an analytics platform. These challenges demand new strategies and systems
that can offer the desired degree of scalability and robustness in handling
failures. This paper investigates the fundamental requirements and the state of
the art of existing data stream ingestion systems, propose a scalable and
fault-tolerant data stream ingestion and integration framework that can serve
as a reusable component across many feeds of structured and unstructured input
data in a given platform, and demonstrate the utility of the framework in a
real-world data stream processing case study that integrates Apache NiFi and
Kafka for processing high velocity news articles from across the globe. The
study also identifies best practices and gaps for future research in developing
large-scale data stream processing infrastructure.
