  In this paper, we describe the automorphism group of semidirect product of
two groups that fixes the non-normal subgroup of it. We have computed these
automorphisms for the non-abelian metacyclic $p$-group and non-abelian
$p$-groups $(p\ge 5)$ of order $p^{4}$, where $p$ is a prime.
