  Consumers often react expressively to products such as food samples, perfume,
jewelry, sunglasses, and clothing accessories. This research discusses a
multimodal affect recognition system developed to classify whether a consumer
likes or dislikes a product tested at a counter or kiosk, by analyzing the
consumer's facial expression, body posture, hand gestures, and voice after
testing the product. A depth-capable camera and microphone system - Kinect for
Windows - is utilized. An emotion identification engine has been developed to
analyze the images and voice to determine affective state of the customer. The
image is segmented using skin color and adaptive threshold. Face, body and
hands are detected using the Haar cascade classifier. Canny edges are
identified and the lip, body and hand contours are extracted using spatial
filtering. Edge count and orientation around the mouth, cheeks, eyes,
shoulders, fingers and the location of the edges are used as features.
Classification is done by an emotion template mapping algorithm and training a
classifier using support vector machines. The real-time performance, accuracy
and feasibility for multimodal affect recognition in feedback assessment are
evaluated.
