  We analyze VeLO (versatile learned optimizer), the largest scale attempt to
train a general purpose "foundational" optimizer to date. VeLO was trained on
thousands of machine learning tasks using over 4000 TPU months with the goal of
producing an optimizer capable of generalizing to new problems while being
hyperparameter free, and outperforming industry standards such as Adam. We
independently evaluate VeLO on the MLCommons optimizer benchmark suite. We find
that, contrary to initial claims: (1) VeLO has a critical hyperparameter that
needs problem-specific tuning, (2) VeLO does not necessarily outperform
competitors in quality of solution found, and (3) VeLO is not faster than
competing optimizers at reducing the training loss. These observations call
into question VeLO's generality and the value of the investment in training it.
