  We have probed the crystalline electric-field ground states of pure $|J =
7/2, J_z = \pm 5/2\rangle$ as well as the anisotropic $c$-$f$ hybridization in
both valence fluctuating systems $\alpha$- and $\beta$-YbAlB$_4$ by linear
polarization dependence of angle-resolved core level photoemission
spectroscopy. Interestingly, the small but distinct difference between \abyb
was found in the polar angle dependence of linear dichroism, indicating the
difference in the anisotropy of $c$-$f$ hybridization which may be essential to
a heavy Fermi liquid state in $\alpha$-YbAlB$_4$ and a quantum critical state
in $\beta$-YbAlB$_4$.
