  Iron-based superconductor Sr$_2$VFeAsO$_3$ is composed of alternate stacking
of a superconducting FeAs layer and an insulating vanadium-oxide layer with a
perovskite-type structure. Electronic orders stemming from the spin and orbital
degrees of freedom of V $3d$ electrons can arise in the vanadium-oxide layer,
but such orders have not been confirmed so far. Here, we systematically
investigate the electronic state of Sr$_2$VFeAsO$_{3-\delta}$ with oxygen
deficiency and demonstrate the phase diagram of Sr$_2$VFeAsO$_{3-\delta}$ as a
function of the $c$-axis lattice parameter, which has turned out to be a
suitable measure of the amount of oxygen deficiency. We found a magnetic and
structural anomaly at $\sim 100$ K with a thermal hysteresis, which is
manifested with the introduction of oxygen deficiency. The presence of
orthorhombic distortion was revealed below the temperature at which the anomaly
appears, suggestive of V orbital ordering involving the $d_{xz}$ and $d_{yz}$
orbitals. It seems that substantial fluctuations associated with the
orthorhombic distortion significantly influence the electronic state of the
FeAs layer. Our findings indicate that the vanadium-oxide layer plays a
significant role in the electronic state of Sr$_2$VFeAsO$_{3-\delta}$.
