  With the rapid advance of machine learning (ML) technology, large language
models (LLMs) are increasingly explored as an intelligent tool to generate
program code from natural language specifications. However, existing
evaluations of LLMs have focused on their capabilities in comparison with
humans. It is desirable to evaluate their usability when deciding on whether to
use a LLM in software production. This paper proposes a user centric method for
this purpose. It includes metadata in the test cases of a benchmark to describe
their usages, conducts testing in a multi-attempt process that mimics the uses
of LLMs, measures LLM generated solutions on a set of quality attributes that
reflect usability, and evaluates the performance based on user experiences in
the uses of LLMs as a tool.
  The paper also reports a case study with the method in the evaluation of
ChatGPT's usability as a code generation tool for the R programming language.
Our experiments demonstrated that ChatGPT is highly useful for generating R
program code although it may fail on hard programming tasks. The user
experiences are good with overall average number of attempts being 1.61 and the
average time of completion being 47.02 seconds. Our experiments also found that
the weakest aspect of usability is conciseness, which has a score of 3.80 out
of 5.
