Automorphisms of metacyclic groups
\Large Haimiao Chen \footnote{Email: chenhm@math.pku.edu.cn
Let $H$ be a metacyclic group which can be presented as $\langle a,b\mid a^{n=e, b^{m=a^{t, bab^{-1=a^{r\rangle$ for some $n,m,t,r$. Each endomorphism of $H$ is given by $\sigma:a\mapsto a^{x_{1b^{y_{1, b\mapsto a^{x_{2b^{y_{2$ for some integers $x_{1,x_{2,y_{1,y_{2$. We find sufficient and necessary conditions on $x_{1,x_{2,y_{1,y_{2$ for $\sigma$ to be an automorphism, so as to determine all automorphisms of $H$. \medskip \noindent {Keywords: automorphism, metacyclic group, congruence equation. \\ {MSC 2010: 20D45.
1506.02234.tex
\documentclass[11pt]{article}
\textwidth=450truept \textheight=580truept \oddsidemargin=1mm
\usepackage{amsfonts}
\usepackage{latexsym}
\usepackage{cite}
\usepackage{amsmath,amsfonts,latexsym,amssymb,color}
\usepackage[mathscr]{eucal}
\usepackage{cases}
\usepackage{amsthm}

\usepackage[bf,small]{caption2}
\usepackage{float}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage[all]{xy}

\newtheorem{theorem}{theorem}[section]
\newtheorem{thm}[theorem]{Theorem}
\newtheorem{lem}[theorem]{Lemma}
\newtheorem{clm}[theorem]{Claim}
\newtheorem{prop}[theorem]{Proposition}
\newtheorem{cor}[theorem]{Corollary}
\newtheorem{ass}[theorem]{Assertion}

\newtheorem{defn}[theorem]{Definition}
\newtheorem{exmp}[theorem]{Example}
\newtheorem{que}[theorem]{Question}
\newtheorem{conj}[theorem]{Conjecture}
\newtheorem{rmk}[theorem]{Remark}
\newtheorem{nota}[theorem]{Notation}



\begin{document}

\title{\textbf{Automorphisms of metacyclic groups}}
\author{\Large Haimiao Chen
\footnote{Email: \emph{chenhm@math.pku.edu.cn}}  \\
\normalsize \em{Mathematics, Beijing Technology and Business University, Beijing, China}\\
\Large Yueshan Xiong
\footnote{Email: \emph{xiongyueshan@gmail.com}}\\
\normalsize \em{Mathematics, Capital Normal University, Beijing, China} \\
\Large Zhongjian Zhu
\footnote{Email: \emph{zhuzhongjianzzj@126.com}}\\
\normalsize \em{Institute of Mathematics, Chinese Academy of Science, Beijing, China}}
\date{}
\maketitle


\begin{abstract}
  Let $H$ be a metacyclic group which can be presented as $\langle a,b\mid a^{n}=e, b^{m}=a^{t}, bab^{-1}=a^{r}\rangle$ for some $n,m,t,r$. Each endomorphism of $H$ is given by
  $\sigma:a\mapsto a^{x_{1}}b^{y_{1}}, b\mapsto a^{x_{2}}b^{y_{2}}$ for some integers $x_{1},x_{2},y_{1},y_{2}$. We find sufficient and necessary conditions on $x_{1},x_{2},y_{1},y_{2}$ for $\sigma$ to be an automorphism, so as to determine all automorphisms of $H$.

  \medskip
  \noindent {\bf Keywords:}  automorphism, metacyclic group, congruence equation. \\
  {\bf MSC 2010:} 20D45.
\end{abstract}



\section{Introduction}


A finite group $G$ is {\it metacyclic} if it contains a cyclic, normal subgroup $N$ such that $G/N$ is also cyclic. Metacyclic groups form an interesting family of groups, in that they are the simplest ones other than abelian groups.

As a natural object, the automorphism group of a metacyclic group has been widely studied.
In 1970, Davitt \cite{Dav70} shaw that if $G$ is a metacyclic $p$-group with $p\neq 2$, then the order of ${\rm Aut}(G)$ is divided by that of $G$.
In 2006, Bidwell and Curran \cite{BC06} found the order and the structure of ${\rm Aut}(G)$ when $G$ is a split metacyclic $p$-group with $p\neq 2$, and in 2007, Curran \cite{Cur07} obtained similar results for split metacyclic $2$-groups.
In 2008, Curran \cite{Cur08} determined ${\rm Aut}(G)$ when $G$ is a non-split metacyclic $p$-group with $p\neq 2$. In 2009, Golasi${\rm\acute{n}}$ski and Gon\c{c}alves \cite{GG09} determined ${\rm Aut}(G)$ for any split metacyclic group $G$.
The automorphism groups of non-split metacyclic 2-groups are left unknown, let alone general cases.

For abelian groups, all automorphisms were determined by Hillar and Rhea \cite{HR07} in 2007. In this paper, we aim to explicitly write down all of the automorphisms for a general metacyclic group; this should help people understand the automorphism group better.

It is well-known  (see Section 3.7 of \cite{Zas56}) that each metacyclic group can be presented as
\begin{align}
\langle a,b\mid a^{n}=e, b^{m}=a^{t}, bab^{-1}=a^{r}\rangle. \label{eq:presentation}
\end{align}
for some positive integers $n,m,r,t$ satisfying
\begin{align}
r^{m}-1\equiv t(r-1)\equiv 0\pmod{n}.   \label{eq:condition}
\end{align}
Denote this group by $H=H(n,m;t,r)$.
There is an extension
$$1\to\mathbb{Z}/n\mathbb{Z}\to H\to\mathbb{Z}/m\mathbb{Z}\to 1$$
where $\mathbb{Z}/n\mathbb{Z}\cong\langle a\rangle\lhd H$ and $\mathbb{Z}/m\mathbb{Z}\cong H/\langle a\rangle$.
It may happen that two groups given by different values of $n,m,t,r$ are isomorphic. A complete classification (up to isomorphism) for finite metacyclic groups was obtained by Hempel \cite{Hem00} in 2000.

In the presentation (\ref{eq:presentation}), we may assume $t\mid n$ which we do from now on. To see this, choose $u,v$ such that $un+vt=(n,t)$, then $(v,n/(n,t))=1$. Let $w$ be the product of all prime factors of $m$ that do not divide $v$ and let $v'=v+wn/(n,t)$, then $(v',m)=1$. Replacing $b$ by $\check{b}=b^{v'}$,
we get another presentation:
$H=\langle a,\check{b}\mid a^{n}=e, \check{b}^{m}=a^{(n,t)}, \check{b}a\check{b}^{-1}=a^{r^{v'}}\rangle.$


Obviously each element can be written as $a^{u}b^{v}$; note that $a^{u}b^{v}=e$ if and only if $m\mid v$ and $n\mid (u+tv/m)$.
Each automorphism $\sigma\in\textrm{Aut}(H)$ is determined by
$\sigma(a)=a^{x_{1}}b^{y_{1}}, \sigma(b)=a^{x_{2}}b^{y_{2}}$
for some integers $x_{1},x_{2},y_{1},y_{2}$.
The main result of this paper gives sufficient and necessary conditions on $x_{1},x_{2},y_{1},y_{2}$, for $\sigma$ to be an automorphism.
They consist of two parts, ensuring $\sigma$ to be invertible and well-defined, respectively. Skillfully using elementary number theoretical techniques, we manage to reduce the second part to linear congruence equations. It turns out that the situation concerning the prime 2 is quite subtle,
and this reflects the difficulty in determining the automorphism groups of metacyclic 2-groups.


%\textbf{Acknowledgement} \ \ We are grateful to Professor Yang Su at Institute of Mathematics, Chinese Academic of Science, for beneficial conversations.



\section{Determining all automorphisms}


\subsection{Preparation}


\begin{nota}
\rm For an integer $n>0$, let $\mathbb{Z}_{n}$ denote $\mathbb{Z}/n\mathbb{Z}$, and also regard it as a quotient ring of $\mathbb{Z}$.
For $u\in\mathbb{Z}$, denote its image under the quotient $\mathbb{Z}\twoheadrightarrow\mathbb{Z}_{n}$ also by $u$.

For integers $u,s$ with $u\ge 0$, set
$[u]_{s}=\left\{\begin{array}{ll}
1+s+\cdots+s^{u-1},  &u>0, \\
0,   &u=0.
\end{array}\right.$

For a prime number $p$ and an integer $u>0$, let $\deg_{p}(u)$ denote the largest integer $s$ with $p^{s}\mid u$;
set $\deg_{p}(0)=+\infty$ for convenience.

Denote $a^{u}$ by $\exp_{a}(u)$ when the expression for $u$ is too long.
\end{nota}


\begin{lem} \label{lem:simplify}
If $s>1$ with $\deg_{p}(s-1)=\ell>1$ and $x>0$ with $\deg_{p}(x)=u>0$, then
\begin{enumerate}
  \item[\rm(I)]
  $[x]_{s}\equiv \left\{\begin{array}{ll}
  x, &p\ne 2, \\
  (1+2^{\ell-1})x, &p=2
  \end{array}\right.
  \pmod{p^{\ell+u}}$.
  \item[\rm(II)]
  $s^{x}-1\equiv \left\{\begin{array}{ll}
  (s-1)x, &p\ne 2, \\
  (s-1+2^{2\ell-1})x, &p=2 \\
  \end{array}\right.
  \pmod{p^{2\ell+u}}$.
\end{enumerate}
\end{lem}

\begin{proof}
We only prove (I), then (II) follows immediately from the identity $(s-1)[u]_{s}=s^{u}-1$.

Write $s=1+p^{\ell}\mu$ with $p\nmid\mu$, then
\begin{align*}
[p^{u}]_{s}
&=\sum\limits_{i=0}^{p^{u}-1}(1+p^{\ell}\mu)^{i}=\sum\limits_{i=0}^{p^{u}-1}\sum\limits_{j=0}^{i}{i\choose j}(p^{\ell}\mu)^{j}
%=\sum\limits_{j=0}^{2^{u}-1}(2^{\ell}\mu)^{j}\sum\limits_{i=j}^{2^{u}-1}{i\choose j}
=\sum\limits_{j=1}^{p^{u}}(p^{\ell}\mu)^{j-1}{p^{u}\choose j} \\
&\equiv \left\{\begin{array}{ll}
p^{u}, &p\ne 2, \\
2^{u}(1+2^{\ell-1}), &p=2
\end{array}\right. \pmod{p^{\ell+u}},
\end{align*}
using
\begin{align*}
\deg_{p}\left({p^{u}\choose j}\right)
&=\deg_{p}\left(\frac{(p^{u})!}{j!(p^{u}-j+1)!}\right)=
\sum\limits_{i=0}^{j-1}\deg_{p}(p^{u}-i)-\sum\limits_{i=1}^{j}\deg_{p}(i) \\
&=u-\deg_{p}(j)\geq u-(j-2)\ell=(\ell+u)-(j-1)\ell
\end{align*}
for all $\left\{\begin{array}{ll} j\ge 2, &p\ne 2, \\ j\ge 3, &p=2 \\ \end{array}\right.$ and $(2^{u}-1)\mu\equiv 1\pmod{2}$ when $p=2$.

Thus $s^{p^{u}}=(s-1)[p^{u}]_{s}\equiv 1\pmod{p^{k}}$; writing $x=p^{u}x'$ with $p\nmid x'$, we have
$$[x]_{s}=[p^{u}]_{s}\left(\sum\limits_{j=0}^{x'-1}(s^{p^{u}})^{j}\right)=[p^{u}]_{s}\cdot x'\equiv
\left\{\begin{array}{ll}
x, &p\ne 2, \\
(1+2^{\ell-1})x, &p=2
\end{array}\right. \pmod{p^{\ell+u}}.$$
\end{proof}





\subsection{The method}



It follows from (\ref{eq:presentation}) that
\begin{align}
b^{v}a^{u}&=a^{r^{v}u}b^{v}, \label{eq:identity1} \\
(a^{u_{1}}b^{v_{1}})(a^{u_{2}}b^{v_{2}})&=a^{u_{1}+r^{v_{1}}u_{2}}b^{v_{1}+v_{2}},\label{eq:identity2} \\
(a^{u}b^{v})^{k}&=a^{u[k]_{r}}b^{vk}, \label{eq:identity3} \\
[a^{u_{1}}b^{v_{1}},a^{u_{2}}b^{v_{2}}]&=\exp_{a}((r^{v_{1}}-1)u_{2}-(r^{v_{2}}-1)u_{1}), \label{eq:identity4}
\end{align}
where the notation $[g,h]=ghg^{-1}h^{-1}$ for commutator is adopted.

In view of (\ref{eq:identity4}), the commutator subgroup $[H,H]$ is generated by $a^{r-1}$.
The abelianization $H^{{\rm ab}}:=H/[H,H]$ has a presentation
\begin{align}
\langle \overline{a},\overline{b}\mid d\overline{a}=0, m\overline{b}=t\overline{a}\rangle, \ \ \ \ \text{with\ \  } d=(r-1,n),
\end{align}
where additive notation is used and $\overline{a}+\overline{b}=\overline{b}+\overline{a}$ is implicitly assumed.


\begin{lem}  \label{lem:well-defined}
%Suppose $0\leq x_{1},x_{2}<n$ and $0\leq y_{1},y_{2}<m$.
There exists a homomorphism $\sigma:H\to H$ with
$\sigma(a)=a^{x_{1}}b^{y_{1}}$, $\sigma(b)=a^{x_{2}}b^{y_{2}}$ if and only if
\begin{align}
(d,t)y_{1}&\equiv 0\pmod{m}, \label{eq:condition1}  \\
x_{2}[m]_{r^{y_{2}}}+ty_{2}-x_{1}[t]_{r^{y_{1}}}-\frac{ty_{1}}{m}t&\equiv 0\pmod{n}, \label{eq:condition2}\\
(r^{y_{1}}-1)x_{2}+([r]_{r^{y_{1}}}-r^{y_{2}})x_{1}+\frac{(r-1)y_{1}}{m}t&\equiv 0\pmod{n}.    \label{eq:condition3}
\end{align}
\end{lem}

\begin{proof}
Sufficient and necessary conditions for $\sigma$ to be well-defined are:
\begin{align*}
e=\sigma(a)^{n}&=a^{x_{1}[n]_{r^{y_{1}}}}b^{y_{1}n}, \\
a^{x_{2}[m]_{r^{y_{2}}}}b^{y_{2}m}=\sigma(b)^{m}&=\sigma(a)^{t}=a^{x_{1}[t]_{r^{y_{1}}}}b^{y_{1}t}, \\
a^{x_{2}}b^{y_{2}}a^{x_{1}}b^{y_{1}}b^{-y_{2}}a^{-x_{2}}=\sigma(b)\sigma(a)\sigma(b)^{-1}&=\sigma(a)^{r}=a^{x_{1}[r]_{r^{y_{1}}}}b^{y_{1}r};
\end{align*}
equivalently,
\begin{align}
m\mid ny_{1},  \hspace{5mm} x_{1}[n]_{r^{y_{1}}}+\frac{ny_{1}}{m}t&\equiv 0\pmod{n},  \label{eq:1st} \\
m\mid ty_{1},  \hspace{5mm} x_{2}[m]_{r^{y_{2}}}+y_{2}t&\equiv x_{1}[t]_{r^{y_{1}}}+\frac{ty_{1}}{m}t\pmod{n}, \label{eq:2nd} \\
m\mid (r-1)y_{1}, \hspace{5mm} x_{2}(1-r^{y_{1}})+x_{1}r^{y_{2}}&\equiv x_{1}[r]_{r^{y_{1}}}+\frac{(r-1)y_{1}}{m}t\pmod{n}.  \label{eq:3rd}
\end{align}
The first halves of (\ref{eq:1st}), (\ref{eq:2nd}), (\ref{eq:3rd}) are equivalent to the single one $m\mid (d,t)y_{1}$.
Then the second half of (\ref{eq:1st}) can be omitted: for each prime divisor $p$ of $n$,
if $p\mid r^{y_{1}}-1$, then by Lemma \ref{lem:simplify} (I), $\deg_{p}([n]_{r^{y_{1}}})\geq\alpha_{p}$; otherwise, since $r^{ny_{1}}-1$ can be divided by $r^{m}-1$ which is a multiple of $n$, we also have $\deg_{p}([n]_{r^{y_{1}}})=\deg_{p}(r^{ny_{1}}-1)\geq\alpha_{p}.$
\end{proof}


Let $\Lambda$ denote the set of prime divisors of $nm$, and for each $p\in\Lambda$, denote
\begin{align}
\alpha_{p}=\deg_{p}(n), \hspace{5mm}
\beta_{p}=\deg_{p}(m), \hspace{5mm}
\gamma_{p}=\deg_{p}(t),  \hspace{5mm}
\delta_{p}=\deg_{p}(r-1).  \label{eq:fac}
\end{align}
Subdivide $\Lambda=\Lambda_{1}\sqcup \Lambda_{2}\sqcup\Lambda'$, with
\begin{align}
\Lambda_{1}=\{p\colon \alpha_{p},\delta_{p}>0\}, \ \ \ \ \Lambda_{2}=\{p\colon \alpha_{p}>\delta_{p}=0\}, \ \ \ \  \Lambda'=\{p\colon \alpha_{p}=0\}.
\end{align}
Denote
\begin{align}
\epsilon=\deg_{2}(r+1).
\end{align}


It follows from $t(r-1)\equiv 0\pmod{n}$ that
\begin{align}
\left\{\begin{array}{ll}
\alpha_{p}-\delta_{p}\le\gamma_{p}\le\alpha_{p}, &p\in\Lambda_{1}, \\
\gamma_{p}=\alpha_{p}, &p\in\Lambda_{2},
\end{array}\right.   \label{ineq:alpha1}
\end{align}
and it follows from $r^{m}-1\equiv 0\pmod{n}$ and Lemma \ref{lem:simplify} (II) that
\begin{align}
\text{for\ each\ } p\in\Lambda_{1}, \ \ \ \ \
\left\{\begin{array}{ll}
\delta_{p}+\beta_{p}\geq\alpha_{p},  & (p,\delta_{p})\neq (2,1) \ \text{or\ } (p,\delta_{p},\beta_{p})=(2,1,0), \\
\epsilon+\beta_{2}\geq\alpha_{2},  & \delta_{2}=1\ \text{and\ } \beta_{2}>0.    \end{array}\right.     \label{ineq:beta}                                    \end{align}
The condition (\ref{eq:condition1}) is equivalent to
\begin{align}
\min\{\delta_{p},\gamma_{p}\}+\deg_{p}(y_{1})\geq\beta_{p} \ \ \ \ \text{for\ all\ } p\in\Lambda.  \label{ineq:m}
\end{align}

Since $H$ is finite, $\sigma$ is invertible if and only if it is injective; equivalently,
{\rm(i)} the induced homomorphism
$\overline{\sigma}:H^{\rm ab}\to H^{\rm ab}$ determined by $\overline{a}\mapsto x_{1}\overline{a}+y_{1}\overline{b}$,
$\overline{b}\mapsto x_{2}\overline{a}+y_{2}\overline{b}$ is injective, and  {\rm(ii)} the restriction $\sigma|_{[H,H]}$ is injective.

%Note that (\ref{eq:condition2}) implies, in particular,
%\begin{align}
%mx_{2}+ty_{2}\equiv tx_{1}+\frac{ty_{1}}{m}t\pmod{d}.   \label{eq:condition0}
%\end{align}


\begin{lem} \label{lem:auto-of-ab}
Let $\delta^{0}_{p}=\min\{\alpha_{p},\delta_{p}\}=\deg_{p}(d)$.
%With {\rm(\ref{eq:condition1})} and {\rm(\ref{eq:condition0})},
The homomorphism $\overline{\sigma}$ is injective if and only if
\begin{align}
\left\{\begin{array}{lll}
p \nmid y_{2}, &p\in\Lambda', \\
p \nmid x_{1},  &p\in\Lambda_{1}, \ \beta_{p}=0,  \\
p \nmid x_{1}y_{2}-x_{2}y_{1},   &p\in\Lambda_{1}, \ \beta_{p}>0.
\end{array}\right.             \label{eq:condition00}
\end{align}
\end{lem}

\begin{proof}
For each $p\in\Lambda'\sqcup\Lambda_{1}$, let
$$H^{{\rm ab}}_{p}=\langle\overline{a}_{p},\overline{b}_{p}\rangle, \ \ \ \ \ \text{with\ \ \ }
\overline{a}_{p}=\frac{d}{p^{\delta_{p}^{0}}}\overline{a}, \ \ \ \overline{b}_{p}=\frac{m}{p^{\beta_{p}}}\overline{b},$$
then %$H^{{\rm ab}}_{p}$ is the Sylow $p$-subgroup of $H^{{\rm ab}}$, and
$\overline{\sigma}$ is injective if and only if $\overline{\sigma}_{p}=\overline{\sigma}|_{H^{{\rm ab}}_{p}}$
is injective for all $p$.
Note that
\begin{align*}
\overline{\sigma}_{p}(\overline{a}_{p})&=\overline{\sigma}\left(\frac{d}{p^{\delta_{p}^{0}}}\overline{a}\right)
=\frac{d}{p^{\delta_{p}^{0}}}(x_{1}\overline{a}+y_{1}\overline{b})=x_{1}\overline{a}_{p}+\frac{dy_{1}p^{\beta}}{mp^{\delta^{0}_{p}}}\overline{b}_{p}, \\
\overline{\sigma}_{p}(\overline{b}_{p})&=\overline{\sigma}\left(\frac{m}{p^{\beta_{p}}}\overline{b}\right)
=\frac{m}{p^{\beta_{p}}}(x_{2}\overline{a}+y_{2}\overline{b})=\frac{m}{p^{\beta_{p}}}wx_{2}\cdot\overline{a}_{p}+y_{2}\overline{b}_{p},
\end{align*}
where $w$ is some integer with $wd/p^{\delta^{0}_{p}}\equiv 1\pmod{p^{\delta^{0}_{p}}}$.

If $p\in\Lambda'$, i.e., $\delta^{0}_{p}=0$, then $H^{{\rm ab}}_{p}=\langle \overline{b}_{p}\rangle$, and it is easy to see that $\sigma_{p}$ is injective if and only if $p\nmid y_{2}$. If $p\in\Lambda_{1}$ with $\beta_{p}=0$, then $H^{{\rm ab}}_{p}=\langle \overline{a}_{p}\rangle$, and it is easy to see that $\sigma_{p}$ is injective if and only if $p\nmid x_{1}$. If $p\in\Lambda_{1}$ with $\beta_{p}>0$, then by Theorem 3.6 of \cite{HR07}, $\overline{\sigma}_{p}$ is injective if and only if
$$p\nmid x_{1}y_{2}-\frac{dy_{1}p^{\beta}}{mp^{\delta^{0}_{p}}}\frac{m}{p^{\beta_{p}}}wx_{2}\equiv x_{1}y_{2}-x_{2}y_{1}\pmod{p}.$$
\end{proof}

Now consider the injectivity of $\sigma|_{[H,H]}$.
Note that for each $p\in\Lambda_{1}$ with $\delta_{p}<\alpha_{p}$,
$r^{y_{1}}-1\equiv (r-1)y_{1}$ and $r^{y_{2}}-1\equiv (r-1)y_{2}\pmod{p^{2\delta_{p}}}$, hence
\begin{align*}
(r^{y_{1}}-1)x_{2}-(r^{y_{2}}-1)x_{1}\equiv (r-1)(x_{2}y_{1}-x_{1}y_{2})\pmod{p^{2\delta_{p}}},
\end{align*}
which together with (\ref{eq:condition00}) implies $\deg_{p}((r^{y_{1}}-1)x_{2}-(r^{y_{2}}-1)x_{1})=\delta_{p}$.
Thus $\sigma|_{[H,H]}$ is injective if and only if
\begin{align}
\deg_{p}((r^{y_{1}}-1)x_{2}-(r^{y_{2}}-1)x_{1})=\delta_{p} \ \ \ \text{for\ all\ } p\in\Lambda_{2}. \label{eq:injective}
\end{align}

%\begin{rmk}
%\rm For $p\in\Lambda_{1}$ with $\delta_{p}<\alpha_{p}$ and $(p,\delta_{p})\ne (2,1)$, by Lemma \ref{lem:simplify} (II), $[r]_{r^{y_{1}}}=1+[r-1]_{r^{y_{1}}}\equiv r-1\pmod{p^{2\delta_{p}}}$, hence (10) implies $x_{1}y_{2}-x_{2}y_{1}\equiv x_{1}+ty_{1}/m\pmod{p}$, so we may write (\ref{eq:condition00}) alternatively as
%\begin{align}
%\left\{\begin{array}{lll}
%p \nmid y_{2},  &p\in\Lambda', \\
%p \nmid x_{1}y_{2}-x_{2}y_{1},   &p\in\Lambda_{1} \ \text{with\ } (p,\delta_{p})=(2,1)\ \text{or\ } \delta_{p},\beta_{p}\ge\alpha_{p}=\gamma_{p}, \\
%p \nmid x_{1}+ty_{1}/m,  &p\in\Lambda_{1}\ \text{with\ } (p,\delta_{p})\ne (2,1) \ \text{and\ } \alpha_{p}>\min\{\beta_{p},\gamma_{p},\delta_{p}\}.  \\
%\end{array}\right.            \label{eq:condition000}
%\end{align}
%\end{rmk}


Summarizing, sufficient and necessary conditions for there to be an automorphism $\sigma$ with $\sigma(a)=a^{x_{1}}b^{y_{1}}$,
$\sigma(b)=a^{x_{2}}b^{y_{2}}$ are (\ref{eq:condition1}), (\ref{eq:condition2}), (\ref{eq:condition3}), (\ref{eq:condition00}) and (\ref{eq:injective}). The main task in the remainder of this section is, assuming (\ref{eq:condition1}) and (\ref{eq:condition00}), to simplify (\ref{eq:condition2}), (\ref{eq:condition3}) and (\ref{eq:injective}).



Let (\ref{eq:condition2})$_{p}$ (resp. (\ref{eq:condition3})$_{p}$) denote the condition (\ref{eq:condition2}) (resp. (\ref{eq:condition3})) with ${\rm mod}\ n$ replaced by ${\rm mod}\ p^{\alpha_{p}}$, then (\ref{eq:condition2}) (resp. (\ref{eq:condition3})) is equivalent to (\ref{eq:condition2})$_{p}$ (resp. (\ref{eq:condition3})$_{p}$) for all $p\in\Lambda_{1}\sqcup\Lambda_{2}$ simultaneously.

If $p\in\Lambda_{1}$ with $\delta_{p}\geq\alpha_{p}$, then $r\equiv 1\pmod{p^{\alpha_{p}}}$,
hence (\ref{eq:condition3})$_{p}$ is trivial, and (\ref{eq:condition2})$_{p}$ becomes
$$t(x_{1}+\frac{ty_{1}}{m}-y_{2})\equiv mx_{2}\pmod{p^{\alpha_{p}}}.$$

We shall discuss the cases $p\in\Lambda_{2}$ and $p\in\Lambda_{1}$ with $\delta_{p}<\alpha_{p}$, respectively.



\subsection{$p\in\Lambda_{2}$}


\begin{lem}  \label{lem:deducing0}
If $p\in\Lambda_{2}$, then $p\neq 2$, and the conditions {\rm(\ref{eq:condition2})}$_{p}$, {\rm(\ref{eq:condition3})}$_{p}$ hold if and only if
$r^{y_{1}}\equiv 1$ and $r^{y_{2}}\equiv r\pmod{p^{\alpha_{p}}}$.
\end{lem}

\begin{proof}
If $p=2$, then $2\mid n$ and $2\mid r$, contradicting to $n\mid r^{m}-1$. Hence $p\neq 2$.

Just write $x=y$ if $x\equiv y\pmod{p^{\alpha_{p}}}$.
Due to (\ref{ineq:alpha1}), we may convert (\ref{eq:condition2})$_{p}$ and (\ref{eq:condition3})$_{p}$ into
\begin{align}
x_{2}[m]_{r^{y_{2}}}-x_{1}[t]_{r^{y_{1}}}&=0, \label{eq:condition22} \\
(r^{y_{1}}-1)x_{2}+([r]_{r^{y_{1}}}-r^{y_{2}})x_{1}&=0. \label{eq:condition32}
\end{align}
The second equation implies
\begin{align*}
rx_{1}[r-1]_{r^{y_{1}}}+(r^{y_{1}}-1)x_{2}-(r^{y_{2}}-1)x_{1}=0,
\end{align*}
which together with (\ref{eq:injective}) implies $p\nmid x_{1}[r-1]_{r^{y_{1}}}$.
Then since
$$(r^{y_{1}}-1)[r-1]_{r^{y_{1}}}=r^{(r-1)y_{1}}-1=(r^{m})^{(r-1)y_{1}/m}-1=0,$$
we have $r^{y_{1}}\equiv 1\pmod{p^{\alpha_{p}}}$.
Now (\ref{eq:condition32}) becomes $(r^{y_{2}}-r)x_{1}=0$,
so $r^{y_{2}}=r$.
Consequently, $[m]_{r^{y_{2}}}=[m]_{r}=0$ and $[t]_{r^{y_{1}}}=t=0$, hence (\ref{eq:condition22}) automatically holds.
\end{proof}

\begin{rmk}   \label{rmk:0}
\rm It is appropriate to point out that  (\ref{eq:injective}) is equivalent to $p\nmid x_{1}$ for all $p\in\Lambda_{2}$.
\end{rmk}



\subsection{$p\in\Lambda_{1}$, $\delta_{p}<\alpha_{p}$}

In this case, we have $\beta_{p}>0$ by (\ref{ineq:beta}).
In the proofs, we abbreviate $\alpha_{p},\beta_{p},\gamma_{p},\delta_{p},\deg_{p}(x)$ to $\alpha,\beta,\gamma,\delta,\deg(x)$, respectively.


\subsubsection{$p\neq 2$}

\begin{lem} \label{lem:deducing}
If $2\neq p\in\Lambda_{1}$, then {\rm(\ref{eq:condition2})}$_{p}$ and {\rm(\ref{eq:condition3})}$_{p}$ hold if and only if
\begin{align}
mx_{2}&\equiv t(x_{1}+\frac{ty_{1}}{m}-y_{2})\pmod{p^{\alpha_{p}}}, \label{eq:lambda1-1} \\
y_{2}&\equiv 1+\frac{ty_{1}}{m}\pmod{p^{\alpha_{p}-\delta_{p}}}.   \label{eq:lambda1-2}
\end{align}
\end{lem}

\begin{proof}
By Lemma \ref{lem:simplify}, the following identities hold in $\mathbb{Z}_{p^{\alpha}}$:
\begin{align*}
r^{y_{1}}=1+(r-1)y_{1}, \ \ \ \ [r]_{r^{y_{1}}}=r, \ \ \ \ [t]_{r^{y_{1}}}=t, \ \ \ \ [m]_{r^{y_{2}}}=m,
\end{align*}
hence we can simplify (\ref{eq:condition2})$_{p}$ into (\ref{eq:lambda1-1}) and rewrite (\ref{eq:condition3})$_{p}$ as
\begin{align}
(r-1)y_{1}x_{2}-(r^{y_{2}}-r)x_{1}+\frac{(r-1)y_{1}}{m}t\equiv 0\pmod{p^{\alpha}}. \label{eq:condition4}
\end{align}

Since by (\ref{ineq:m}), $\deg(y_{1})\geq \beta-\delta\geq\alpha-2\delta$, and by (\ref{ineq:alpha1}),
$\deg(ty_{1}/m)=\deg(y_{1})+\gamma-\beta\geq\alpha-2\delta$,
it follows from (\ref{eq:condition4}) that
\begin{align*}
\deg(y_{2}-1)=\deg((r^{y_{2}}-r))-\delta-\deg(x_{1})\geq\alpha-2\delta-\deg(x_{1}).
\end{align*}
Then by Lemma \ref{lem:simplify} (II),
$r^{y_{2}-1}-1\equiv(y_{2}-1)(r-1)\pmod{p^{\alpha-\deg(x_{1})}},$
and hence, using $(r-1)^{2}(y_{2}-1)x_{1}\equiv 0\pmod{p^{\alpha}}$, we obtain
$$(r^{y_{2}}-r)x_{1}\equiv(y_{2}-1)(r-1)rx_{1}\equiv(y_{2}-1)(r-1)x_{1}\pmod{p^{\alpha}}.$$
Thus (\ref{eq:condition4}) can be converted into
\begin{align}
(y_{2}-1)x_{1}\equiv x_{2}y_{1}+\frac{ty_{1}}{m}\pmod{p^{\alpha-\delta}}.  \label{eq:condition4'}
\end{align}

Finally, (\ref{eq:condition4'}) and (\ref{eq:lambda1-1}) lead to (\ref{eq:lambda1-2}):
\begin{align*}
&(y_{2}-1-\frac{ty_{1}}{m})(x_{1}+\frac{ty_{1}}{m}) \\
=&\ (y_{2}-1)x_{1}+(y_{2}-1-x_{1}-\frac{ty_{1}}{m})\frac{ty_{1}}{m} \\
\equiv &\ x_{2}y_{1}+(y_{2}-x_{1}-\frac{ty_{1}}{m})\frac{ty_{1}}{m}\pmod{p^{\alpha-\delta}} \hspace{20mm}
(\text{by\ }  (\ref{eq:condition4'})) \\
=&\ \frac{y_{1}}{m}(mx_{2}+ty_{2}-tx_{1}-\frac{ty_{1}}{m}t) \equiv 0\pmod{p^{\alpha-\delta}} \hspace{10mm}
(\text{by\ }  (\ref{eq:lambda1-1})),
\end{align*}
where in the last line we use $\alpha+\deg(y_{1})-\beta\geq\alpha-\delta$.
\end{proof}


\subsubsection{$p=2$}


Set
\begin{align}
\mu(y_{1})=\left\{\begin{array}{lll}
2^{\alpha_{2}-\delta_{2}-1}, & \gamma_{2}\neq\beta_{2}, \  \min\{\beta_{2},\gamma_{2}\}=\alpha_{2}-\delta_{2} \ \ \text{and\ } \deg_{2}(y_{1})=\beta_{2}-\delta_{2}, \\
0, & \text{otherwise}.
\end{array}\right.
\end{align}


\begin{lem} \label{lem:deducing2}
Suppose $2\in\Lambda_{1}$. The conditions {\rm(\ref{eq:condition2})}$_{2}$ and {\rm(\ref{eq:condition3})}$_{2}$ hold if and only if
\begin{enumerate}
  \item [\rm(i)] If $\delta_{2}=\gamma_{2}=1$ (so that $\alpha_{2}=2$) and $\beta_{2}>1$, then $\deg_{2}(y_{1})\ge\beta_{2}$;
  \item [\rm(ii)] if $\delta_{2}=1$ and $\gamma_{2}>1$, then
                 $2\mid y_{1}$, $\deg_{2}(x_{2})\ge\alpha_{2}-\beta_{2}-\epsilon+1$ and
                 \begin{align}
                 2^{\epsilon}(y_{1}-y_{2}+1)\equiv\frac{2y_{1}}{m}t \pmod{2^{\alpha_{2}}};   \label{eq:lambda2-0}
                 \end{align}
  \item [\rm(iii)] if $\delta_{2}>1$, then
      \begin{align}
      mx_{2}&\equiv t(x_{1}+\frac{ty_{1}}{m}-y_{2})\pmod{2^{\alpha_{2}}}, \label{eq:lambda2-1} \\
      y_{2}&\equiv 1+\frac{ty_{1}}{m}+\mu(y_{1}) \pmod{2^{\alpha_{2}-\delta_{2}}}.   \label{eq:lambda2-2}
      \end{align}
\end{enumerate}
\end{lem}


\begin{proof}
(i) If $\beta>1$, then $2\mid y_{1}$, hence $2\nmid x_{1}y_{2}$ and $r^{y_{1}}\equiv -r^{y_{2}}\equiv 1\pmod{4}$; it is easy to check that (\ref{eq:condition2})$_{2}$ and (\ref{eq:condition3})$_{2}$ hold if and only if $\deg(y_{1})\ge\beta$.

If $\beta=1$, then since in $\mathbb{Z}_{4}$,
$$r^{y_{1}}=1+2y_{1}, \ \ \ r^{y_{2}}=1+2y_{2}, \ \ \  [m]_{r^{y_{2}}}=2+2y_{2}, \ \ \ [t]_{r^{y_{1}}}=[r-1]_{r^{y_{1}}}=2+2y_{1},$$
we may rewrite
(\ref{eq:condition2})$_{2}$ and (\ref{eq:condition3})$_{2}$ as, respectively,
\begin{align*}
x_{2}(2+2y_{2})+2y_{2}-x_{1}(2+2y_{1})-2y_{1}=0, \\
2y_{1}x_{2}+(2y_{1}-2-2y_{2})x_{1}+2y_{1}=0;
\end{align*}
they can be verified to be true under the assumption $2\nmid x_{1}y_{2}-x_{2}y_{1}$.


(ii) We first show $2\mid y_{1}$.
Assume on the contrary that $2\nmid y_{1}$. Then $\beta=1$ by (\ref{ineq:m}), $2\mid x_{2}$ by (\ref{eq:condition2})$_{2}$, and  $2\nmid x_{1}y_{2}$ by (\ref{eq:condition00}).
By (\ref{ineq:beta}), $\deg(r^{2}-1)=\epsilon+1\geq\alpha,$
hence in $\mathbb{Z}_{2^{\alpha}}$ we have
$$r^{2}=1, \ \ \ \ \ r^{y_{1}}=r^{y_{2}}=r, \ \ \ \ \ [r]_{r^{y_{1}}}=1+\frac{r^{2}-1}{2}.$$
But (\ref{eq:condition3})$_{2}$ implies $x_{1}(r-1)^{2}/2\equiv 0\pmod{4}$, which is impossible.

So $2\mid y_{1}$, $2\nmid x_{1}y_{2}$, and hence $t(x_{1}-y_{2})\equiv 0\pmod{2^{\alpha}}$.

By (\ref{ineq:beta}), (\ref{ineq:m}), $1+\epsilon+\deg(y_{1})\geq\alpha$,
hence
$$\deg(r^{y_{1}}-1)=\deg((r^{2})^{\frac{y_{1}}{2}}-1)=\epsilon+1+\deg(y_{1})-1\ge\alpha-1.$$
Applying Lemma \ref{lem:simplify}, we obtain the following identities in $\mathbb{Z}_{2^{\alpha}}$:
\begin{align*}
r^{y_{1}}&=(1+(r^{2}-1))^{\frac{y_{1}}{2}}=1+(r^{2}-1)\cdot\frac{y_{1}}{2}=1+(r+1)y_{1}, \\
[t]_{r^{y_{1}}}&=t(1+2^{\epsilon+\deg(y_{1})-1})=t,  \\
[r]_{r^{y_{1}}}&=r^{(r-1)y_{1}}+[r-1]_{r^{y_{1}}}=1+(r-1)(1+2^{\epsilon+\deg(y_{1})-1})=r+(r+1)y_{1}.
\end{align*}
Since $t^{2}y_{1}/m\equiv 0\pmod{2^{\alpha}}$, (\ref{eq:condition2})$_{2}$ becomes
$x_{2}[m]_{r^{y_{2}}}\equiv0\pmod{2^{\alpha}}$, which is equivalent to
\begin{align*}
\deg(x_{2})+\beta+\epsilon-1\geq\alpha,
\end{align*}
where we have used $\deg(r^{y_{2}}+1)=\epsilon$.
Consequently
\begin{align*}
\deg((r^{y_{1}}-1)x_{2})=\deg(y_{1})+\epsilon+\deg(x_{2})\geq\beta-1+\epsilon+\deg(x_{2})\geq\alpha,
\end{align*}
hence (\ref{eq:condition3})$_{2}$ becomes
\begin{align}
r^{y_{2}}-r\equiv (r+1)y_{1}+\frac{(r-1)y_{1}}{m}t\pmod{2^{\alpha}}. \label{eq:condition5}
\end{align}
Since $\epsilon+\deg(y_{1})\ge\alpha-1$ and $\deg((r-1)y_{1}t/m)=1+\deg(y_{1})+\gamma-\beta\ge\alpha-1$, we have
$\deg(r^{y_{2}}-r)\ge\alpha-1$,
hence $r^{y_{2}}-r\equiv(r+1)(y_{2}-1)\pmod{2^{\alpha}}$, and (\ref{eq:condition5}) is equivalent to (\ref{eq:lambda2-0}).

(iii) In this case, $2\delta+\deg(y_{1})\geq\alpha$ and the following holds in $\mathbb{Z}_{2^{\alpha}}$:
\begin{align*}
r^{y_{1}}&=
\left\{\begin{array}{ll}
1+(r-1)y_{1},  &2\nmid y_{1}, \\
1+(r-1+2^{2\delta-1})y_{1}, &2\mid y_{1},\end{array}\right. \hspace{8mm}
[r]_{r^{y_{1}}}=r+2^{2\delta-1}y_{1},  \\
[t]_{r^{y_{1}}}&=t(1+2^{\delta-1}y_{1}),  \hspace{40mm}
[m]_{r^{y_{2}}}=m(1+2^{\delta-1}y_{2}).
\end{align*}
By (10)$_{2}$, $x_{1}y_{2}-x_{2}y_{1}\equiv x_{1}+ty_{1}/m\pmod{2}$, hence
\begin{align}
2\nmid x_{1}+\frac{ty_{1}}{m}.    \label{eq:condition000}
\end{align}
%We discuss two cases separately: (iii-1) when $2\mid y_{1}$ and (iii-2) when $2\nmid y_{1}$.

(iii-1) If $2\mid y_{1}$, then $2\nmid x_{1}y_{2}$ by (\ref{eq:condition00}), and  $2\mid ty_{1}/m$ by (\ref{eq:condition000}).
The condition (\ref{eq:condition2})$_{2}$ becomes
\begin{align}
(1+2^{\delta-1}y_{2})mx_{2}-t(x_{1}+\frac{ty_{1}}{m}-y_{2})&\equiv 0 \pmod{2^{\alpha}}, \label{eq:condition6}
\end{align}
which can be converted into (\ref{eq:lambda2-1}) via multiplying $1-2^{\delta-1}y_{2}$, using (\ref{ineq:alpha1}) and
$2\mid x_{1}+ty_{1}/m-y_{2}$.
Moreover, (\ref{eq:condition6}) implies
$\beta+\deg(x_{2})\geq\min\{\gamma+1,\alpha\},$
hence
$2\delta-1+\deg(x_{2})+\deg(y_{1})\geq\alpha.$
Consequently, (\ref{eq:condition3})$_{2}$ becomes
\begin{align}
(r-1)x_{2}y_{1}+2^{2\delta-1}y_{1}+(r-r^{y_{2}})x_{1}+\frac{(r-1)y_{1}}{m}t\equiv 0\pmod{2^{\alpha}}.  \label{eq:condition93}
\end{align}
Similarly as in the proof of Lemma \ref{lem:deducing}, we obtain
$\deg(y_{2}-1)\geq\alpha-2\delta$, and then by Lemma \ref{lem:simplify} (II),
$r^{y_{2}}-r\equiv (r-1)(y_{2}-1)(1+2^{\delta-1})\pmod{2^{\alpha}}.$
Hence (\ref{eq:condition93}) can be converted into
\begin{align}
(y_{2}-1)x_{1}\equiv x_{2}y_{1}+\frac{ty_{1}}{m}+2^{\delta-1}(y_{1}-y_{2}+1) &\pmod{2^{\alpha-\delta}},    \label{eq:condition13}  \\
\text{i.e.},\ \ \ \ \  (x_{1}+2^{\delta-1})(y_{1}-y_{2}+1)\equiv (x_{1}-x_{2})y_{1}+\frac{ty_{1}}{m} &\pmod{2^{\alpha-\delta}},  \nonumber
\end{align}
so $\deg(y_{1}-y_{2}+1)=\alpha-2\delta$ if and only if one of the following cases occurs:
\begin{itemize}
  \item $\deg(y_{1})>\deg(ty_{1}/m)=\alpha-2\delta$,
  \item $\deg(ty_{1}/m)>\deg(y_{1})=\alpha-2\delta$ and $2\mid x_{2}$,
  \item $\deg(ty_{1}/m)=\deg(y_{1})=\alpha-2\delta$ and $2\nmid x_{2}$.
\end{itemize}
In light of (\ref{ineq:alpha1}), (\ref{ineq:beta}), (\ref{ineq:m}) and (\ref{eq:lambda2-1}),  the first case is equivalent to $\gamma=\alpha-\delta$, $\deg(y_{1})=\beta-\delta$ and $\beta>\gamma$; the second case is equivalent to $\deg(y_{1})+\delta=\beta=\alpha-\delta$ and $\gamma>\beta$; the third case is impossible, since it would imply $\beta=\gamma=\alpha-\delta$, which contradicts to (\ref{eq:lambda2-1}).

Rephrasing, $\deg(y_{1}-y_{2}+1)=\alpha-2\delta$ if and only if
\begin{align}
\deg(y_{1})=\beta-\delta, \ \ \ \gamma\neq\beta \  \ \ \text{and\ \ \ } \min\{\beta,\gamma\}=\alpha-\delta. \label{eq:condition-mu}
\end{align}
Thus (\ref{eq:condition13}) is equivalent to
\begin{align}
(y_{2}-1)x_{1}&\equiv x_{2}y_{1}+\frac{ty_{1}}{m}+\mu(y_{1})\pmod{2^{\alpha-\delta}}.   \label{eq:condition13'}
\end{align}
Similarly as in the proof of Lemma \ref{lem:deducing},
\begin{align*}
&(y_{2}-1-\frac{ty_{1}}{m})(x_{1}+\frac{ty_{1}}{m}) \\
=&\ (y_{2}-1)x_{1}+(y_{2}-1-x_{1}-\frac{ty_{1}}{m})\frac{ty_{1}}{m} \\
\equiv &\ x_{2}y_{1}+(y_{2}-x_{1}-\frac{ty_{1}}{m})\frac{ty_{1}}{m}+\mu(y_{1})\pmod{p^{\alpha-\delta}} \hspace{15mm}
(\text{by\ }  (\ref{eq:condition13'})) \\
=&\ \frac{y_{1}}{m}(mx_{2}+ty_{2}-tx_{1}-\frac{ty_{1}}{m}t) \equiv \mu(y_{1})\pmod{p^{\alpha-\delta}} \hspace{12mm}
(\text{by\ }  (\ref{eq:lambda2-1})),
\end{align*}
which results in (\ref{eq:lambda2-2}).

(iii-2) If $2\nmid y_{1}$, then $\delta, \gamma\geq\beta$, hence $2\delta\geq\alpha$.
By Lemma \ref{lem:simplify} (II),
$r^{u}\equiv 1+(r-1)u\pmod{2^{\alpha}}$ for any $u\geq 0$,
hence (\ref{eq:condition2})$_{2}$ and (\ref{eq:condition3})$_{2}$  become
\begin{align}
ty_{2}+m(1+2^{\delta-1}y_{2})x_{2}&\equiv\frac{ty_{1}}{m}t+t(1+2^{\delta-1})x_{1}\pmod{2^{\alpha}},  \label{eq:condition90}  \\
(y_{2}-1)x_{1}&\equiv x_{2}y_{1}+\frac{ty_{1}}{m}+2^{\delta-1}x_{1}\pmod{2^{\alpha-\delta}}.  \label{eq:condition80}
\end{align}

If $\gamma=\beta$, then $2\mid x_{1}$ by (\ref{eq:condition000}), and $2\nmid x_{2}$ by (\ref{eq:condition00}), hence $2\mid y_{2}$ by (\ref{eq:condition90}). We can reduce (\ref{eq:condition90}) to (\ref{eq:lambda2-1}), and reduce (\ref{eq:condition80}) to (\ref{eq:lambda2-2})
in the same way as in the proof of Lemma \ref{lem:deducing}.

Now assume $\gamma>\beta$. Using $\gamma+\delta-1\geq\beta+\delta\geq\alpha$, we can reduce (\ref{eq:condition90}) to (\ref{eq:lambda2-1}) via multiplying by $1-2^{\delta-1}y_{2}$.
Note that
$\delta-1+\deg(x_{1})<\alpha-\delta$
if and only if $\deg(x_{1})=0$ and $\alpha=2\delta$; these two conditions are equivalent to (\ref{eq:condition-mu}), since $\alpha=2\delta$ if and only if $\beta=\delta=\alpha-\delta$.
When (\ref{eq:condition-mu}) holds, we can, similarly as in (iii-1), reduce (\ref{eq:condition80}) to
$y_{2}-1\equiv ty_{1}/m+2^{\alpha-\delta-1}\pmod{2^{\alpha}}$;
otherwise, we get $y_{2}-1\equiv ty_{1}/m\pmod{2^{\alpha}}$. This also proves (\ref{eq:lambda2-2}).
\end{proof}




\subsection{Main result}


Let $m_{0}$ be the smallest positive integer $k$ such that $r^{k}\equiv 1\pmod{p^{\alpha_{p}}}$ for all $p\in\Lambda_{2}$.
%Let $n_{1}=\prod\limits_{2\ne p\in\Lambda_{1}}p^{\alpha_{p}}$.

Combining Lemma \ref{lem:auto-of-ab}, Lemma \ref{lem:deducing0}, Remark \ref{rmk:0}, Lemma \ref{lem:deducing} and Lemma \ref{lem:deducing2}, we have
\begin{thm} \label{thm:main}
Each automorphism of $H(n,m;t,r)$ is given by
\begin{align*}
\sigma_{x_{1},x_{2};y_{1},y_{2}}:\
a^{u}b^{v}\mapsto \exp_{a}(x_{1}[u]_{r^{y_{1}}}+r^{y_{1}u}x_{2}[v]_{r^{y_{2}}})b^{y_{1}u+y_{2}v}, \ \ \ u,v\geq 0,
\end{align*}
for a unique quadruple $(x_{1},x_{2},y_{1},y_{2})$ with $0\le x_{1},x_{2}<n$, $0\le y_{1},y_{2}<m$ and such that
\begin{enumerate}
  \item [\rm(i)] for all $p\in\Lambda$,
        $$\left\{\begin{array}{lll}
        p \nmid y_{2},  &p\in\Lambda', \\
        p \nmid x_{1},  &p\in\Lambda_{2}\ \text{or\ } p\in\Lambda_{1}\ \text{with\ } \beta_{p}=0,\\
        p \nmid x_{1}y_{2}-x_{2}y_{1},  &p\in\Lambda_{1}\ \text{with\ } \beta_{p}>0;
        \end{array}\right.$$
  \item [\rm (ii)] $m\mid (d,t)y_{1}$, $m_{0}\mid y_{1}, y_{2}-1$;
  \item [\rm (iii)] for all $p\in\Lambda_{1}$ with $p\ne 2$,
        \begin{align*}
        mx_{2}\equiv t(x_{1}+\frac{ty_{1}}{m}-y_{2})\pmod{p^{\alpha_{p}}}, \ \ \ \ \ y_{2}\equiv 1+\frac{ty_{1}}{m}\pmod{p^{\alpha_{p}-\delta_{p}}};
        \end{align*}
  \item [\rm(iv)] if $2\in\Lambda_{1}$ with $\delta_{2}>1$ or $\delta_{2}=\alpha_{2}=1$, then
        \begin{align*}
        mx_{2}\equiv t(x_{1}+\frac{ty_{1}}{m}-y_{2})\pmod{2^{\alpha_{2}}}, \ \ \ \ \ y_{2}\equiv 1+\frac{ty_{1}}{m}+\mu(y_{1}) \pmod{2^{\alpha_{2}-\delta_{2}}};
        \end{align*}
  \item [\rm (v)] if $2\in\Lambda_{1}$ with $\delta_{2}=\gamma_{2}=1$, $\alpha_{2}=2$ and $\beta_{2}>1$, then
        $\deg_{2}(y_{1})\ge\beta_{2}$.
  \item [\rm (vi)] if $2\in\Lambda_{1}$ with $\delta_{2}=1$, $\alpha_{2}>1$ and $\gamma_{2}>1$, then
        $$2\mid y_{1}, \ \ \ \ \ \deg_{2}(x_{2})\ge\alpha_{2}-\beta_{2}-\epsilon+1, \ \ \ \ \ 2^{\epsilon}(y_{1}-y_{2}+1)\equiv 2ty_{1}/m \pmod{2^{\alpha_{2}}}.$$
\end{enumerate}
\end{thm}

It is possible to derive a formula for the order of ${\rm Aut}(H(n,m;t,r))$, but we prefer not to do so, as it is just a boring job leading to nothing essentially new; instead, we give an example to illustrate how to enumerate the automorphisms.


\begin{exmp}
\rm Let $n=228, m=30, t=38, r=7$, then (\ref{eq:condition}) is satisfied. We have
\begin{align*}
d=6, \ \ \ \ \Lambda_{1}&=\{2,3\},  \ \ \ \ \Lambda_{2}=\{19\}, \ \ \ \ \Lambda'=\{5\},\ \ \ \ m_{0}=3, \\
(\alpha_{2},\beta_{2},\gamma_{2},\delta_{2})&=(2,1,1,1), \hspace{8mm}  (\alpha_{3},\beta_{3},\gamma_{3},\delta_{3})=(1,1,0,1), \\
(\alpha_{19},\beta_{19},\gamma_{19},\delta_{19})&=(1,0,1,0), \hspace{27mm} \beta_{5}=1.
\end{align*}
By Theorem \ref{thm:main}, each automorphism of $H(228,30;38,7)$ is determined by a unique quadruple $(x_{1},x_{2},y_{1},y_{2})$ with $0\leq x_{1},x_{2}<228$, $0\leq y_{1},y_{2}<30$ and
\begin{enumerate}
  \item[\rm(i)] $5\nmid y_{2}$,  $19\nmid x_{1}$,  $2\nmid x_{1}y_{2}-x_{2}y_{1}$, $3\nmid x_{1}y_{2}-x_{2}y_{1}$;
  \item[\rm(ii)] $15\mid y_{1}$, $3\mid y_{2}-1$;
  \item[\rm(iii)] $x_{1}+38y_{1}/30\equiv y_{2}\pmod{3}$.
\end{enumerate}
Thus either
\begin{itemize}
  \item $y_{1}=0$, $y_{2}\in\{1,7,13,19\}$, $x_{1}\in\{1,7,\ldots,223\}\backslash\{19,133\}$, or
  \item $y_{1}=15$, $y_{2}\in\{1,4,\ldots,28\}\backslash\{10,25\}$, $x_{1}\in\{0,3,\ldots,225\}\backslash \{0,57,114,171\}$, $x_{2}\equiv x_{1}y_{2}+1\pmod{2}$.
\end{itemize}
There are $4\times 36\times 228+8\times 72\times 114=98496$ in total. We have checked this result with GAP.
\end{exmp}



\begin{thebibliography}{}


\bibitem{BC06}
J.N.S. Bidwell, M.J. Curran:
\textsl{The automorphism group of a split metacyclic $p$-group}.
Arch. Math. 87 (2006), 488--497.

\bibitem{Cur07}
M.J. Curran:
\textsl{The automorphism group of a split metacyclic 2-group}.
Arch. Math. 89 (2007), 10--23.

\bibitem{Cur08}
M.J. Curran:
\textsl{The automorphism group of a nonsplit metacyclic p-group}.
Arch. Math. 90 (2008), 483--489.


\bibitem{Dav70}
R.M. Davitt:
\textsl{The automorphism group of a finite metacyclic $p$-group}.
Proceedings of the American Mathematical Society. 25 (1970), 876--879.

\bibitem{GG09}
M. Golasi${\rm\acute{n}}$ski, D.L. Gon\c{c}alves:
\textsl{On automorphisms of split metacyclic groups}.
Manuscripta Math. 128 (2009), 251--273.

\bibitem{Hem00}
C.E. Hempel:
\textsl{Metacyclic groups}.
Communications in Algebra, 28 (2000), no. 8, 3865--3897.

\bibitem{HR07}
C.J. Hillar, D.L. Rhea:
\textsl{Automorphisms of finite abelian groups}.
The American Mathematical Monthly 114 (2007), 917--923.

%\bibitem{Kin73}
%B.W. King,
%\textsl{Presentations of metacyclic groups}.
%Bull. Austral. Math. Soc. 8 (1973), 103--131.

\bibitem{Zas56}
H.J. Zassenhaus:
\textsl{The theory of groups}.
Second edition, Chelsea, New York, 1956.



\end{thebibliography}


\end{document}

1506.02234.extracted_bib
{BC06 J.N.S. Bidwell, M.J. Curran: \textsl{The automorphism group of a split metacyclic $p$-group. Arch. Math. 87 (2006), 488--497.
1506.02234.extracted_bib
{Cur07 M.J. Curran: \textsl{The automorphism group of a split metacyclic 2-group. Arch. Math. 89 (2007), 10--23.
1506.02234.extracted_bib
{Cur08 M.J. Curran: \textsl{The automorphism group of a nonsplit metacyclic p-group. Arch. Math. 90 (2008), 483--489.
1506.02234.extracted_bib
{Dav70 R.M. Davitt: \textsl{The automorphism group of a finite metacyclic $p$-group. Proceedings of the American Mathematical Society. 25 (1970), 876--879.
1506.02234.extracted_bib
{GG09 M. Golasi${\rm\acute{n$ski, D.L. Gon\c{calves: \textsl{On automorphisms of split metacyclic groups. Manuscripta Math. 128 (2009), 251--273.
1506.02234.extracted_bib
{Hem00 C.E. Hempel: \textsl{Metacyclic groups. Communications in Algebra, 28 (2000), no. 8, 3865--3897.
1506.02234.extracted_bib
{HR07 C.J. Hillar, D.L. Rhea: \textsl{Automorphisms of finite abelian groups. The American Mathematical Monthly 114 (2007), 917--923. %
1506.02234.extracted_bib
{Kin73 %B.W. King, %\textsl{Presentations of metacyclic groups. %Bull. Austral. Math. Soc. 8 (1973), 103--131.
1506.02234.extracted_bib
{Zas56 H.J. Zassenhaus: \textsl{The theory of groups. Second edition, Chelsea, New York, 1956.