  Bent functions, which are maximally nonlinear Boolean functions with even
numbers of variables and whose Hamming distance to the set of all affine
functions equals $2^{n-1}\pm 2^{\frac{n}{2}-1}$, were introduced by Rothaus in
1976 when he considered problems in combinatorics. Bent functions have been
extensively studied due to their applications in cryptography, such as S-box,
block cipher and stream cipher. Further, they have been applied to coding
theory, spread spectrum and combinatorial design. Hyper-bent functions, as a
special class of bent functions, were introduced by Youssef and Gong in 2001,
which have stronger properties and rarer elements. Many research focus on the
construction of bent and hyper-bent functions. In this paper, we consider
functions defined over $\mathbb{F}_{2^n}$ by
$f_{a,b}:=\mathrm{Tr}_{1}^{n}(ax^{(2^m-1)})+\mathrm{Tr}_{1}^{4}(bx^{\frac{2^n-1}{5}})$,
where $n=2m$, $m\equiv 2\pmod 4$, $a\in \mathbb{F}_{2^m}$ and
$b\in\mathbb{F}_{16}$. When $a\in \mathbb{F}_{2^m}$ and $(b+1)(b^4+b+1)=0$,
with the help of Kloosterman sums and the factorization of $x^5+x+a^{-1}$, we
present a characterization of hyper-bentness of $f_{a,b}$. Further, we use
generalized Ramanujan-Nagell equations to characterize hyper-bent functions of
$f_{a,b}$ in the case $a\in\mathbb{F}_{2^{\frac{m}{2}}}$.
