  For a large class of variational quantum circuits, we show how
arbitrary-order derivatives can be analytically evaluated in terms of simple
parameter-shift rules, i.e., by running the same circuit with different shifts
of the parameters. As particular cases, we obtain parameter-shift rules for the
Hessian of an expectation value and for the metric tensor of a variational
state, both of which can be efficiently used to analytically implement
second-order optimization algorithms on a quantum computer. We also consider
the impact of statistical noise by studying the mean squared error of different
derivative estimators. In the second part of this work, some of the theoretical
techniques for evaluating quantum derivatives are applied to their typical use
case: the implementation of quantum optimizers. We find that the performance of
different estimators and optimizers is intertwined with the values of different
hyperparameters, such as a step size or a number of shots. Our findings are
supported by several numerical and hardware experiments, including an
experimental estimation of the Hessian of a simple variational circuit and an
implementation of the Newton optimizer.
