  With the impressive performance in various downstream tasks, large language
models (LLMs) have been widely integrated into production pipelines, like
recruitment and recommendation systems. A known issue of models trained on
natural language data is the presence of human biases, which can impact the
fairness of the system. This paper investigates LLMs' behavior with respect to
gender stereotypes, in the context of occupation decision making. Our framework
is designed to investigate and quantify the presence of gender stereotypes in
LLMs' behavior via multi-round question answering. Inspired by prior works, we
construct a dataset by leveraging a standard occupation classification
knowledge base released by authoritative agencies. We tested three LLMs
(RoBERTa-large, GPT-3.5-turbo, and Llama2-70b-chat) and found that all models
exhibit gender stereotypes analogous to human biases, but with different
preferences. The distinct preferences of GPT-3.5-turbo and Llama2-70b-chat may
imply the current alignment methods are insufficient for debiasing and could
introduce new biases contradicting the traditional gender stereotypes.
