  Recent advancements in artificial intelligence (AI) have seen the emergence
of smart video surveillance (SVS) in many practical applications, particularly
for building safer and more secure communities in our urban environments.
Cognitive tasks, such as identifying objects, recognizing actions, and
detecting anomalous behaviors, can produce data capable of providing valuable
insights to the community through statistical and analytical tools. However,
artificially intelligent surveillance systems design requires special
considerations for ethical challenges and concerns. The use and storage of
personally identifiable information (PII) commonly pose an increased risk to
personal privacy. To address these issues, this paper identifies the privacy
concerns and requirements needed to address when designing AI-enabled smart
video surveillance. Further, we propose the first end-to-end AI-enabled
privacy-preserving smart video surveillance system that holistically combines
computer vision analytics, statistical data analytics, cloud-native services,
and end-user applications. Finally, we propose quantitative and qualitative
metrics to evaluate intelligent video surveillance systems. The system shows
the 17.8 frame-per-second (FPS) processing in extreme video scenes. However,
considering privacy in designing such a system results in preferring the
pose-based algorithm to the pixel-based one. This choice resulted in dropping
accuracy in both action and anomaly detection tasks. The results drop from
97.48 to 73.72 in anomaly detection and 96 to 83.07 in the action detection
task. On average, the latency of the end-to-end system is 36.1 seconds.
