  This paper presents BigDL (a distributed deep learning framework for Apache
Spark), which has been used by a variety of users in the industry for building
deep learning applications on production big data platforms. It allows deep
learning applications to run on the Apache Hadoop/Spark cluster so as to
directly process the production data, and as a part of the end-to-end data
analysis pipeline for deployment and management. Unlike existing deep learning
frameworks, BigDL implements distributed, data parallel training directly on
top of the functional compute model (with copy-on-write and coarse-grained
operations) of Spark. We also share real-world experience and "war stories" of
users that have adopted BigDL to address their challenges(i.e., how to easily
build end-to-end data analysis and deep learning pipelines for their production
data).
