  Algorithms now permeate multiple aspects of human lives and multiple recent
results have reported that these algorithms may have biases pertaining to
gender, race, and other demographic characteristics. The metrics used to
quantify such biases have still focused on a static notion of algorithms.
However, algorithms evolve over time. For instance, Tay (a conversational bot
launched by Microsoft) was arguably not biased at its launch but quickly became
biased, sexist, and racist over time. We suggest a set of intuitive metrics to
study the variations in biases over time and present the results for a case
study for genders represented in images resulting from a Twitter image search
for #Nurse and #Doctor over a period of 21 days. Results indicate that biases
vary significantly over time and the direction of bias could appear to be
different on different days. Hence, one-shot measurements may not suffice for
understanding algorithmic bias, thus motivating further work on studying biases
in algorithms over time.
