  Current gesture interfaces typically demand users to learn and perform
gestures from a predefined set, which leads to a less natural experience.
Interfaces supporting user-defined gestures eliminate the learning process, but
users still need to demonstrate and associate the gesture to a specific system
function themselves. We introduce GestureGPT, a free-form hand gesture
understanding framework that does not require users to learn, demonstrate, or
associate gestures. Our framework leverages the large language model's (LLM)
astute common sense and strong inference ability to understand a spontaneously
performed gesture from its natural language descriptions, and automatically
maps it to a function provided by the interface. More specifically, our
triple-agent framework involves a Gesture Description Agent that automatically
segments and formulates natural language descriptions of hand poses and
movements based on hand landmark coordinates. The description is deciphered by
a Gesture Inference Agent through self-reasoning and querying about the
interaction context (e.g., interaction history, gaze data), which a Context
Management Agent organizes and provides. Following iterative exchanges, the
Gesture Inference Agent discerns user intent, grounding it to an interactive
function. We validated our conceptual framework under two real-world scenarios:
smart home controlling and online video streaming. The average zero-shot Top-5
grounding accuracies are 83.59% for smart home tasks and 73.44% for video
streaming. We also provided an extensive discussion of our framework including
model selection rationale, generated description quality, generalizability etc.
