  Linear Mode Connectivity (LMC) refers to the phenomenon that performance
remains consistent for linearly interpolated models in the parameter space. For
independently optimized model pairs from different random initializations,
achieving LMC is considered crucial for validating the stable success of the
non-convex optimization in modern machine learning models and for facilitating
practical parameter-based operations such as model merging. While LMC has been
achieved for neural networks by considering the permutation invariance of
neurons in each hidden layer, its attainment for other models remains an open
question. In this paper, we first achieve LMC for soft tree ensembles, which
are tree-based differentiable models extensively used in practice. We show the
necessity of incorporating two invariances: subtree flip invariance and
splitting order invariance, which do not exist in neural networks but are
inherent to tree architectures, in addition to permutation invariance of trees.
Moreover, we demonstrate that it is even possible to exclude such additional
invariances while keeping LMC by designing decision list-based tree
architectures, where such invariances do not exist by definition. Our findings
indicate the significance of accounting for architecture-specific invariances
in achieving LMC.
