  Jet classification is an important ingredient in measurements and searches
for new physics at particle coliders, and secondary vertex reconstruction is a
key intermediate step in building powerful jet classifiers. We use a neural
network to perform vertex finding inside jets in order to improve the
classification performance, with a focus on separation of bottom vs. charm
flavor tagging. We implement a novel, universal set-to-graph model, which takes
into account information from all tracks in a jet to determine if pairs of
tracks originated from a common vertex. We explore different performance
metrics and find our method to outperform traditional approaches in accurate
secondary vertex reconstruction. We also find that improved vertex finding
leads to a significant improvement in jet classification performance.
