  We establish an open framework for developing plan optimization models for
knowledge-based planning (KBP) in radiotherapy. Our framework includes
reference plans for 100 patients with head-and-neck cancer and high-quality
dose predictions from 19 KBP models that were developed by different research
groups during the OpenKBP Grand Challenge. The dose predictions were input to
four optimization models to form 76 unique KBP pipelines that generated 7600
plans. The predictions and plans were compared to the reference plans via: dose
score, which is the average mean absolute voxel-by-voxel difference in dose a
model achieved; the deviation in dose-volume histogram (DVH) criterion; and the
frequency of clinical planning criteria satisfaction. We also performed a
theoretical investigation to justify our dose mimicking models. The range in
rank order correlation of the dose score between predictions and their KBP
pipelines was 0.50 to 0.62, which indicates that the quality of the predictions
is generally positively correlated with the quality of the plans. Additionally,
compared to the input predictions, the KBP-generated plans performed
significantly better (P<0.05; one-sided Wilcoxon test) on 18 of 23 DVH
criteria. Similarly, each optimization model generated plans that satisfied a
higher percentage of criteria than the reference plans. Lastly, our theoretical
investigation demonstrated that the dose mimicking models generated plans that
are also optimal for a conventional planning model. This was the largest
international effort to date for evaluating the combination of KBP prediction
and optimization models. In the interest of reproducibility, our data and code
is freely available at https://github.com/ababier/open-kbp-opt.
