  We report a theoretical study on the phase transition between superconducting
states with and without spontaneous surface current. The phase transition takes
place due to the formation of surface Andreev bound states in unconventional
superconductors. Based on the quasiclassical theory of superconductivity, we
examine the influence of atomic-scale surface roughness on the surface phase
transition temperature $T_s$. To describe the surface effect, the boundary
condition for the quasiclassical Green's function is parameterized in terms of
specularity (the specular reflection probability in the normal state at the
Fermi level). This boundary condition allows systematic study of the surface
effect ranging from the specular limit to the diffuse limit. We show that
diffuse quasiparticle scattering at a rough surface causes substantial
reduction of $T_s$ in the d-wave pairing state of high-$T_c$ cuprate
superconductors. We also consider a p-wave pairing state in which Andreev bound
states similar to those in the d-wave state are generated. In contrast to the
d-wave case, $T_s$ in the p-wave state is insensitive to the specularity. This
is because the Andreev bound states in the p-wave superconductor are robust
against diffuse scattering, as implied from symmetry consideration for
odd-frequency Cooper pairs induced at the surface; the p-wave state has
odd-frequency pairs with s-wave symmetry, while the d-wave state does not.
