  Commanding a drone with a natural language is not only user-friendly but also
opens the door for emerging language agents to control the drone. Emerging
large language models (LLMs) provide a previously impossible opportunity to
automatically translate a task description in a natural language to a program
that can be executed by the drone. However, powerful LLMs and their vision
counterparts are limited in three important ways. First, they are only
available as cloud-based services. Sending images to the cloud raises privacy
concerns. Second, they are expensive, costing proportionally to the request
size. Finally, without expensive fine-tuning, existing LLMs are quite limited
in their capability of writing a program for specialized systems like drones.
  In this paper, we present a system called TypeFly that tackles the above
three problems using a combination of edge-based vision intelligence, novel
programming language design, and prompt engineering. Instead of the familiar
Python, TypeFly gets a cloud-based LLM service to write a program in a small,
custom language called MiniSpec, based on task and scene descriptions in
English. Such MiniSpec programs are not only succinct (and therefore efficient)
but also able to consult the LLM during their execution using a special skill
called query. Using a set of increasingly challenging drone tasks, we show that
design choices made by TypeFly can reduce both the cost of LLM service and the
task execution time by more than 2x. More importantly, query and prompt
engineering techniques contributed by TypeFly significantly improve the chance
of success of complex tasks.
