  Search engines intentionally influence user behavior by picking and ranking
the list of results. Users engage with the highest results both because of
their prominent placement and because they are typically the most relevant
documents. Search engine ranking algorithms need to identify relevance while
incorporating the influence of the search engine itself. This paper describes
our efforts at Thumbtack to understand the impact of ranking, including the
empirical results of a randomization program. In the context of a consumer
marketplace we discuss practical details of model choice, experiment design,
bias calculation, and machine learning model adaptation. We include a novel
discussion of how ranking bias may not only affect labels, but also model
features. The randomization program led to improved models, motivated internal
scenario analysis, and enabled user-facing scenario tooling.
