  In an efficient and flexible human-robot collaborative work environment, a
robot team member must be able to recognize both explicit requests and implied
actions from human users. Identifying "what to do" in such cases requires an
agent to have the ability to construct associations between objects, their
actions, and the effect of actions on the environment. In this regard, semantic
memory is being introduced to understand the explicit cues and their
relationships with available objects and required skills to make "tea" and
"sandwich". We have extended our previous hierarchical robot control
architecture to add the capability to execute the most appropriate task based
on both feedback from the user and the environmental context. To validate this
system, two types of skills were implemented in the hierarchical task tree: 1)
Tea making skills and 2) Sandwich making skills. During the conversation
between the robot and the human, the robot was able to determine the hidden
context using ontology and began to act accordingly. For instance, if the
person says "I am thirsty" or "It is cold outside" the robot will start to
perform the tea-making skill. In contrast, if the person says, "I am hungry" or
"I need something to eat", the robot will make the sandwich. A humanoid robot
Baxter was used for this experiment. We tested three scenarios with objects at
different positions on the table for each skill. We observed that in all cases,
the robot used only objects that were relevant to the skill.
