  Commercial AI solutions provide analysts and managers with data-driven
business intelligence for a wide range of decisions, such as demand forecasting
and pricing. However, human analysts may have their own insights and
experiences about the decision-making that is at odds with the algorithmic
recommendation. In view of such a conflict, we provide a general analytical
framework to study the augmentation of algorithmic decisions with human
knowledge: the analyst uses the knowledge to set a guardrail by which the
algorithmic decision is clipped if the algorithmic output is out of bound, and
seems unreasonable. We study the conditions under which the augmentation is
beneficial relative to the raw algorithmic decision. We show that when the
algorithmic decision is asymptotically optimal with large data, the
non-data-driven human guardrail usually provides no benefit. However, we point
out three common pitfalls of the algorithmic decision: (1) lack of domain
knowledge, such as the market competition, (2) model misspecification, and (3)
data contamination. In these cases, even with sufficient data, the augmentation
from human knowledge can still improve the performance of the algorithmic
decision.
