  Class-imbalanced datasets are known to cause the problem of model being
biased towards the majority classes. In this project, we set up two research
questions: 1) when is the class-imbalance problem more prevalent in
self-supervised pre-training? and 2) can offline clustering of feature
representations help pre-training on class-imbalanced data? Our experiments
investigate the former question by adjusting the degree of {\it
class-imbalance} when training the baseline models, namely SimCLR and SimSiam
on CIFAR-10 database. To answer the latter question, we train each expert model
on each subset of the feature clusters. We then distill the knowledge of expert
models into a single model, so that we will be able to compare the performance
of this model to our baselines.
