  Tens of thousands of engineers use Sourcegraph day-to-day to search for code
and rely on it to make progress on software development tasks. We face a key
challenge in designing a query language that accommodates the needs of a broad
spectrum of users. Our experience shows that users express different and often
contradictory preferences for how queries should be interpreted. These
preferences stem from users with differing usage contexts, technical
experience, and implicit expectations from using prior tools. At the same time,
designing a code search query language poses unique challenges because it
intersects traditional search engines and full-fledged programming languages.
For example, code search queries adopt certain syntactic conventions in the
interest of simplicity and terseness but invariably risk encoding implicit
semantics that are ambiguous at face-value (a single space in a query could
mean three or more semantically different things depending on surrounding
terms). Users often need to disambiguate intent with additional syntax so that
a query expresses what they actually want to search. This need to disambiguate
is one of the primary frustrations we've seen users experience with writing
search queries in the last three years. We share our observations that lead us
to a fresh perspective where code search behavior can straddle seemingly
ambiguous queries. We develop Automated Query Evaluation (AQE), a new technique
that automatically generates and adaptively runs alternative query
interpretations in frustration-prone conditions. We evaluate AQE with an A/B
test across more than 10,000 unique users on our publicly-available code search
instance. Our main result shows that relative to the control group, users are
on average 22% more likely to click on a search result at all on any given day
when AQE is active.
