  Latent variable models serve as powerful tools to infer underlying dynamics
from observed neural activity. However, due to the absence of ground truth
data, prediction benchmarks are often employed as proxies. In this study, we
reveal the limitations of the widely-used 'co-smoothing' prediction framework
and propose an improved few-shot prediction approach that encourages more
accurate latent dynamics. Utilizing a student-teacher setup with Hidden Markov
Models, we demonstrate that the high co-smoothing model space can encompass
models with arbitrary extraneous dynamics within their latent representations.
To address this, we introduce a secondary metric -- a few-shot version of
co-smoothing. This involves performing regression from the latent variables to
held-out channels in the data using fewer trials. Our results indicate that
among models with near-optimal co-smoothing, those with extraneous dynamics
underperform in the few-shot co-smoothing compared to 'minimal' models devoid
of such dynamics. We also provide analytical insights into the origin of this
phenomenon. We further validate our findings on real neural data using two
state-of-the-art methods: LFADS and STNDT. In the absence of ground truth, we
suggest a proxy measure to quantify extraneous dynamics. By cross-decoding the
latent variables of all model pairs with high co-smoothing, we identify models
with minimal extraneous dynamics. We find a correlation between few-shot
co-smoothing performance and this new measure. In summary, we present a novel
prediction metric designed to yield latent variables that more accurately
reflect the ground truth, offering a significant improvement for latent
dynamics inference.
