  Conversational AI systems are becoming famous in day to day lives. In this
paper, we are trying to address the following key question: To identify whether
design, as well as development efforts for search oriented conversational AI
are successful or not.It is tricky to define 'success' in the case of
conversational AI and equally tricky part is to use appropriate metrics for the
evaluation of conversational AI. We propose four different perspectives namely
user experience, information retrieval, linguistic and artificial intelligence
for the evaluation of conversational AI systems. Additionally, background
details of conversational AI systems are provided including desirable
characteristics of personal assistants, differences between chatbot and an AI
based personal assistant. An importance of personalization and how it can be
achieved is explained in detail. Current challenges in the development of an
ideal conversational AI (personal assistant) are also highlighted along with
guidelines for achieving personalized experience for users.
