  The existing upper and lower bounds between entropy and error are mostly
derived through an inequality means without linking to joint distributions. In
fact, from either theoretical or application viewpoint, there exists a need to
achieve a complete set of interpretations to the bounds in relation to joint
distributions. For this reason, in this work we propose a new approach of
deriving the bounds between entropy and error from a joint distribution. The
specific case study is given on binary classifications, which can justify the
need of the proposed approach. Two basic types of classification errors are
investigated, namely, the Bayesian and non-Bayesian errors. For both errors, we
derive the closed-form expressions of upper bound and lower bound in relation
to joint distributions. The solutions show that Fano's lower bound is an exact
bound for any type of errors in a relation diagram of "Error Probability vs.
Conditional Entropy". A new upper bound for the Bayesian error is derived with
respect to the minimum prior probability, which is generally tighter than
Kovalevskij's upper bound.
