  Deep learning models are known to suffer from the problem of bias, and
researchers have been exploring methods to address this issue. However, most of
these methods require prior knowledge of the bias and are not always practical.
In this paper, we focus on a more practical setting with no prior information
about the bias. Generally, in this setting, there are a large number of
bias-aligned samples that cause the model to produce biased predictions and a
few bias-conflicting samples that do not conform to the bias. If the training
data is limited, the influence of the bias-aligned samples may become even
stronger on the model predictions, and we experimentally demonstrate that
existing debiasing techniques suffer severely in such cases. In this paper, we
examine the effects of unknown bias in small dataset regimes and present a
novel approach to mitigate this issue. The proposed approach directly addresses
the issue of the extremely low occurrence of bias-conflicting samples in
limited data settings through the synthesis of hybrid samples that can be used
to reduce the effect of bias. We perform extensive experiments on several
benchmark datasets and experimentally demonstrate the effectiveness of our
proposed approach in addressing any unknown bias in the presence of limited
data. Specifically, our approach outperforms the vanilla, LfF, LDD, and DebiAN
debiasing methods by absolute margins of 10.39%, 9.08%, 8.07%, and 9.67% when
only 10% of the Corrupted CIFAR-10 Type 1 dataset is available with a
bias-conflicting sample ratio of 0.05.
