  Automation systems that can autonomously drive application user interfaces to
complete user tasks are of great benefit, especially when users are
situationally or permanently impaired. Prior automation systems do not produce
generalizable models while AI-based automation agents work reliably only in
simple, hand-crafted applications or incur high computation costs. We propose
UINav, a demonstration-based approach to train automation agents that fit
mobile devices, yet achieving high success rates with modest numbers of
demonstrations. To reduce the demonstration overhead, UINav uses a referee
model that provides users with immediate feedback on tasks where the agent
fails, and automatically augments human demonstrations to increase diversity in
training data. Our evaluation shows that with only 10 demonstrations UINav can
achieve 70% accuracy, and that with enough demonstrations it can surpass 90%
accuracy.
