  Wavefront shaping has become a powerful tool for manipulating light
propagation in various complex media undergoing linear scattering. Controlling
nonlinear optical interactions with spatial degrees of freedom is a relatively
recent but growing area of research. A wavefront-shaping-based approach can be
used to suppress nonlinear stimulated Brillouin scattering (SBS) and transverse
mode instability (TMI), which are the two main limitations to power scaling in
high-power narrowband fiber amplifiers. Here we formulate both SBS and TMI
suppression as optimization problems with respect to coherent multimode input
excitation in a given multimode fiber. We develop an efficient method for
finding the globally optimal input excitation for SBS and TMI suppression using
linear programming. We theoretically show that optimally exciting a standard
multimode fiber leads to roughly an order of magnitude enhancement in output
power limited by SBS and TMI, compared to fundamental-mode-only excitation. We
find that the optimal mode content is robust to small perturbations and our
approach works even in the presence of mode dependent loss and gain. Optimal
mode content can be excited in real experiments using spatial light modulators,
creating a novel platform for instability-free ultrahigh-power fiber lasers.
