  Food is not only a basic human necessity but also a key factor driving a
society's health and economic well-being. As a result, the cooking domain is a
popular use-case to demonstrate decision-support (AI) capabilities in service
of benefits like precision health with tools ranging from information retrieval
interfaces to task-oriented chatbots. An AI here should understand concepts in
the food domain (e.g., recipes, ingredients), be tolerant to failures
encountered while cooking (e.g., browning of butter), handle allergy-based
substitutions, and work with multiple data modalities (e.g. text and images).
However, the recipes today are handled as textual documents which makes it
difficult for machines to read, reason and handle ambiguity. This demands a
need for better representation of the recipes, overcoming the ambiguity and
sparseness that exists in the current textual documents. In this paper, we
discuss the construction of a machine-understandable rich recipe representation
(R3), in the form of plans, from the recipes available in natural language. R3
is infused with additional knowledge such as information about allergens and
images of ingredients, possible failures and tips for each atomic cooking step.
To show the benefits of R3, we also present TREAT, a tool for recipe retrieval
which uses R3 to perform multi-modal reasoning on the recipe's content (plan
objects - ingredients and cooking tools), food preparation process (plan
actions and time), and media type (image, text). R3 leads to improved retrieval
efficiency and new capabilities that were hither-to not possible in textual
representation.
