  Learning from continuous data streams via classification/regression is
prevalent in many domains. Adapting to evolving data characteristics (concept
drift) while protecting data owners' private information is an open challenge.
We present a differentially private ensemble solution to this problem with two
distinguishing features: it allows an \textit{unbounded} number of ensemble
updates to deal with the potentially never-ending data streams under a fixed
privacy budget, and it is \textit{model agnostic}, in that it treats any
pre-trained differentially private classification/regression model as a
black-box. Our method outperforms competitors on real-world and simulated
datasets for varying settings of privacy, concept drift, and data distribution.
