  We expose the limitation of modular multilingual language models (MLMs) in
multilingual inference scenarios with unknown languages. Existing evaluations
of modular MLMs exclude the involvement of language identification (LID)
modules, which obscures the performance of real-case multilingual scenarios of
modular MLMs. In this work, we showcase the effect of adding LID on the
multilingual evaluation of modular MLMs and provide discussions for closing the
performance gap of caused by the pipelined approach of LID and modular MLMs.
