  Due to its empirical success in few-shot classification and reinforcement
learning, meta-learning has recently received significant interest.
Meta-learning methods leverage data from previous tasks to learn a new task in
a sample-efficient manner. In particular, model-agnostic methods look for
initialization points from which gradient descent quickly adapts to any new
task. Although it has been empirically suggested that such methods perform well
by learning shared representations during pretraining, there is limited
theoretical evidence of such behavior. More importantly, it has not been shown
that these methods still learn a shared structure, despite architectural
misspecifications. In this direction, this work shows, in the limit of an
infinite number of tasks, that first-order ANIL with a linear two-layer network
architecture successfully learns linear shared representations. This result
even holds with overparametrization; having a width larger than the dimension
of the shared representations results in an asymptotically low-rank solution.
The learned solution then yields a good adaptation performance on any new task
after a single gradient step. Overall, this illustrates how well model-agnostic
methods such as first-order ANIL can learn shared representations.
