  Given the variety of the visual world there is not one true scale for
recognition: objects may appear at drastically different sizes across the
visual field. Rather than enumerate variations across filter channels or
pyramid levels, dynamic models locally predict scale and adapt receptive fields
accordingly. The degree of variation and diversity of inputs makes this a
difficult task. Existing methods either learn a feedforward predictor, which is
not itself totally immune to the scale variation it is meant to counter, or
select scales by a fixed algorithm, which cannot learn from the given task and
data. We extend dynamic scale inference from feedforward prediction to
iterative optimization for further adaptivity. We propose a novel entropy
minimization objective for inference and optimize over task and structure
parameters to tune the model to each input. Optimization during inference
improves semantic segmentation accuracy and generalizes better to extreme scale
variations that cause feedforward dynamic inference to falter.
