  We analyze 6.7 million case law documents to determine the presence of gender
bias within our judicial system. We find that current bias detectino methods in
NLP are insufficient to determine gender bias in our case law database and
propose an alternative approach. We show that existing algorithms' inconsistent
results are consequences of prior research's definition of biases themselves.
Bias detection algorithms rely on groups of words to represent bias (e.g.,
'salary,' 'job,' and 'boss' to represent employment as a potentially biased
theme against women in text). However, the methods to build these groups of
words have several weaknesses, primarily that the word lists are based on the
researchers' own intuitions. We suggest two new methods of automating the
creation of word lists to represent biases. We find that our methods outperform
current NLP bias detection methods. Our research improves the capabilities of
NLP technology to detect bias and highlights gender biases present in
influential case law. In order test our NLP bias detection method's
performance, we regress our results of bias in case law against U.S census data
of women's participation in the workforce in the last 100 years.
