  Many hyper-personalized AI systems profile people's characteristics (e.g.,
personality traits) to provide personalized recommendations. These systems are
increasingly used to facilitate interactions among people, such as providing
teammate recommendations. Despite improved accuracy, such systems are not
immune to errors when making inferences about people's most personal traits.
These errors manifested as AI misrepresentations. However, the repercussions of
such AI misrepresentations are unclear, especially on people's reactions and
perceptions of the AI. We present two studies to examine how people react and
perceive the AI after encountering personality misrepresentations in
AI-facilitated team matching in a higher education context. Through
semi-structured interviews (n=20) and a survey experiment (n=198), we pinpoint
how people's existing and newly acquired AI knowledge could shape their
perceptions and reactions of the AI after encountering AI misrepresentations.
Specifically, we identified three rationales that people adopted through
knowledge acquired from AI (mis)representations: AI works like a machine,
human, and/or magic. These rationales are highly connected to people's
reactions of over-trusting, rationalizing, and forgiving of AI
misrepresentations. Finally, we found that people's existing AI knowledge,
i.e., AI literacy, could moderate people's changes in their trust in AI after
encountering AI misrepresentations, but not changes in people's social
perceptions of AI. We discuss the role of people's AI knowledge when facing AI
fallibility and implications for designing responsible mitigation and repair
strategies.
