  Although AI has significant potential to transform society, there are serious
concerns about its ability to behave and make decisions responsibly. Many
ethical regulations, principles, and guidelines for responsible AI have been
issued recently. However, these principles are high-level and difficult to put
into practice. In the meantime much effort has been put into responsible AI
from the algorithm perspective, but they are limited to a small subset of
ethical principles amenable to mathematical analysis. Responsible AI issues go
beyond data and algorithms and are often at the system-level crosscutting many
system components and the entire software engineering lifecycle. Based on the
result of a systematic literature review, this paper identifies one missing
element as the system-level guidance - how to design the architecture of
responsible AI systems. We present a summary of design patterns that can be
embedded into the AI systems as product features to contribute to
responsible-AI-by-design.
