  Many explainable AI (XAI) techniques strive for interpretability by providing
concise salient information, such as sparse linear factors. However, users
either only see inaccurate global explanations, or highly-varying local
explanations. We propose to provide more detailed explanations by leveraging
the human cognitive capacity to accumulate knowledge by incrementally receiving
more details. Focusing on linear factor explanations (factors $\times$ values =
outcome), we introduce Incremental XAI to automatically partition explanations
for general and atypical instances by providing Base + Incremental factors to
help users read and remember more faithful explanations. Memorability is
improved by reusing base factors and reducing the number of factors shown in
atypical cases. In modeling, formative, and summative user studies, we
evaluated the faithfulness, memorability and understandability of Incremental
XAI against baseline explanation methods. This work contributes towards more
usable explanation that users can better ingrain to facilitate intuitive
engagement with AI.
