  Constructing an embedding space for musical instrument sounds that can
meaningfully represent new and unseen instruments is important for downstream
music generation tasks such as multi-instrument synthesis and timbre transfer.
The framework of Automatic Speaker Verification (ASV) provides us with
architectures and evaluation methodologies for verifying the identities of
unseen speakers, and these can be repurposed for the task of learning and
evaluating a musical instrument sound embedding space that can support unseen
instruments. Borrowing from state-of-the-art ASV techniques, we construct a
musical instrument recognition model that uses a SincNet front-end, a ResNet
architecture, and an angular softmax objective function. Experiments on the
NSynth and RWC datasets show our model's effectiveness in terms of equal error
rate (EER) for unseen instruments, and ablation studies show the importance of
data augmentation and the angular softmax objective. Experiments also show the
benefit of using a CQT-based filterbank for initializing SincNet over a Mel
filterbank initialization. Further complementary analysis of the learned
embedding space is conducted with t-SNE visualizations and probing
classification tasks, which show that including instrument family labels as a
multi-task learning target can help to regularize the embedding space and
incorporate useful structure, and that meaningful information such as playing
style, which was not included during training, is contained in the embeddings
of unseen instruments.
