  Data augmentation is crucial in training deep models, preventing them from
overfitting to limited data. Recent advances in generative AI, e.g., diffusion
models, have enabled more sophisticated augmentation techniques that produce
data resembling natural images. We introduce GeNIe a novel augmentation method
which leverages a latent diffusion model conditioned on a text prompt to merge
contrasting data points (an image from the source category and a text prompt
from the target category) to generate challenging samples. To achieve this,
inspired by recent diffusion based image editing techniques, we limit the
number of diffusion iterations to ensure the generated image retains low-level
and background features from the source image while representing the target
category, resulting in a hard negative sample for the source category. We
further enhance the proposed approach by finding the appropriate noise level
adaptively for each image (coined as GeNIe-Ada) leading to further performance
improvement. Our extensive experiments, in both few-shot and long-tail
distribution settings, demonstrate the effectiveness of our novel augmentation
method and its superior performance over the prior art. Our code is available
here: https://github.com/UCDvision/GeNIe
