  Although the design and application of audio effects is well understood, the
inverse problem of removing these effects is significantly more challenging and
far less studied. Recently, deep learning has been applied to audio effect
removal; however, existing approaches have focused on narrow formulations
considering only one effect or source type at a time. In realistic scenarios,
multiple effects are applied with varying source content. This motivates a more
general task, which we refer to as general purpose audio effect removal. We
developed a dataset for this task using five audio effects across four
different sources and used it to train and evaluate a set of existing
architectures. We found that no single model performed optimally on all effect
types and sources. To address this, we introduced RemFX, an approach designed
to mirror the compositionality of applied effects. We first trained a set of
the best-performing effect-specific removal models and then leveraged an audio
effect classification model to dynamically construct a graph of our models at
inference. We found our approach to outperform single model baselines, although
examples with many effects present remain challenging.
