We have a upcoming project and for that I am learning Spark Streaming (with focus on pyspark).
So far I have completed few simple case studies from online.
But I am stuck with 2 scenarios and they are described below.
I shall be highly obliged if you guys kindly share your thoughts or guide me to any web page for help on solution.
Friends, any update for these 2 questions, sadly after many days still no reply.
In order to use append mode with aggregations, you need to set an event time watermark (using "withWatermark"). Otherwise, Spark doesn't know when to output an aggregation result as "final".
A watermark is a threshold to specify how long the system waits for late events.
You have to set SPARK_KAFKA_VERSION environment variable.
When running jobs that require the new Kafka integration, set SPARK_KAFKA_VERSION=0.10 in the shell before launching spark-submit.