  We perform a socio-computational interrogation of the google search by image
algorithm, a main component of the google search engine. We audit the algorithm
by presenting it with more than 40 thousands faces of all ages and more than
four races and collecting and analyzing the assigned labels with the
appropriate statistical tools. We find that the algorithm reproduces white male
patriarchal structures, often simplifying, stereotyping and discriminating
females and non-white individuals, while providing more diverse and positive
descriptions of white men. By drawing from Bourdieu's theory of cultural
reproduction, we link these results to the attitudes of the algorithm's
designers, owners, and the dataset the algorithm was trained on. We further
underpin the problematic nature of the algorithm by using the ethnographic
practice of studying-up: We show how the algorithm places individuals at the
top of the tech industry within the socio-cultural reality that they shaped,
many times creating biased representations of them. We claim that the use of
social-theoretic frameworks such as the above are able to contribute to
improved algorithmic accountability, algorithmic impact assessment and provide
additional and more critical depth in algorithmic bias and auditing studies.
Based on the analysis, we discuss the scientific and design implications and
provide suggestions for alternative ways to design just socioalgorithmic
systems.
