  While unsupervised skill discovery has shown promise in autonomously
acquiring behavioral primitives, there is still a large methodological
disconnect between task-agnostic skill pretraining and downstream, task-aware
finetuning. We present Intrinsic Reward Matching (IRM), which unifies these two
phases of learning via the $\textit{skill discriminator}$, a pretraining model
component often discarded during finetuning. Conventional approaches finetune
pretrained agents directly at the policy level, often relying on expensive
environment rollouts to empirically determine the optimal skill. However, often
the most concise yet complete description of a task is the reward function
itself, and skill learning methods learn an $\textit{intrinsic}$ reward
function via the discriminator that corresponds to the skill policy. We propose
to leverage the skill discriminator to $\textit{match}$ the intrinsic and
downstream task rewards and determine the optimal skill for an unseen task
without environment samples, consequently finetuning with greater
sample-efficiency. Furthermore, we generalize IRM to sequence skills for
complex, long-horizon tasks and demonstrate that IRM enables us to utilize
pretrained skills far more effectively than previous skill selection methods on
both the Fetch tabletop and Franka Kitchen robot manipulation benchmarks.
