  Most of the existing approaches focus on specific visual tasks while ignoring
the relations between them. Estimating task relation sheds light on the
learning of high-order semantic concepts, e.g., transfer learning. How to
reveal the underlying relations between different visual tasks remains largely
unexplored. In this paper, we propose a novel \textbf{L}earnable
\textbf{P}arameter \textbf{S}imilarity (\textbf{LPS}) method that learns an
effective metric to measure the similarity of second-order semantics hidden in
trained models. LPS is achieved by using a second-order neural network to align
high-dimensional model parameters and learning second-order similarity in an
end-to-end way. In addition, we create a model set called ModelSet500 as a
parameter similarity learning benchmark that contains 500 trained models.
Extensive experiments on ModelSet500 validate the effectiveness of the proposed
method. Code will be released at
\url{https://github.com/Wanggcong/learnable-parameter-similarity}.
