  If $G$ be a finite $p$-group and $\chi$ is a non-linear irreducible character
of $G$, then $\chi(1)\leq |G/Z(G)|^{\frac{1}{2}}$. In
\cite{fernandez2001groups}, Fern\'{a}ndez-Alcober and Moret\'{o} obtained the
relation between the character degree set of a finite $p$-group $G$ and its
normal subgroups depending on whether $|G/Z(G)|$ is a square or not. In this
paper we investigate the finite $p$-group $G$ where for any normal subgroup $N$
of $G$ with $G'\not \leq N$ either $N\leq Z(G)$ or $|NZ(G)/Z(G)|\leq p$ and
obtain some alternate characterizations of such groups. We find that if $G$ is
a finite $p$-group with $|G/Z(G)|=p^{2n+1}$ and $G$ satisfies the condition
that for any normal subgroup $N$ of $G$ either $G'\not \leq N$ or $N\leq Z(G)$,
then $cd(G)=\{1, p^{n}\}$. We also find that if $G$ is a finite $p$-group with
nilpotency class not equal to $3$ and $|G/Z(G)|=p^{2n}$ and $G$ satisfies the
condition that for any normal subgroup $N$ of $G$ either $G'\not \leq N$ or
$|NZ(G)/Z(G)|\leq p$, then $cd(G) \subseteq \{1, p^{n-1}, p^{n}\}$.
