  Graph convolutional networks (GCNs) enable end-to-end learning on graph
structured data. However, many works assume a given graph structure. When the
input graph is noisy or unavailable, one approach is to construct or learn a
latent graph structure. These methods typically fix the choice of node degree
for the entire graph, which is suboptimal. Instead, we propose a novel
end-to-end differentiable graph generator which builds graph topologies where
each node selects both its neighborhood and its size. Our module can be readily
integrated into existing pipelines involving graph convolution operations,
replacing the predetermined or existing adjacency matrix with one that is
learned, and optimized, as part of the general objective. As such it is
applicable to any GCN. We integrate our module into trajectory prediction,
point cloud classification and node classification pipelines resulting in
improved accuracy over other structure-learning methods across a wide range of
datasets and GCN backbones.
