  The main objective of this article is to study the three-dimensional
Rayleigh-Benard convection in a rectangular domain from a pattern formation
perspective. It is well known that as the Rayleigh number crosses a critical
threshold, the system undergoes a Type-I transition, characterized by an
attractor bifurcation. The bifurcated attractor is an (m-1)-dimensional
homological sphere where m is the multiplicity of the first critical
eigenvalue. When m=1, the structure of this attractor is trivial. When m=2, it
is known that the bifurcated attractor consists of steady states and their
connecting heteroclinic orbits. The main focus of this article is then on the
pattern selection mechanism and stability of rolls, rectangles and mixed modes
(including hexagons) for the case where m=2. We derive in particular a complete
classification of all transition scenarios, determining the patterns of the
bifurcated steady states, their stabilities and the basin of attraction of the
stable ones. The theoretical results lead to interesting physical conclusions,
which are in agreement with known experimental results. For example, it is
shown in this article that only the pure modes are stable whereas the mixed
modes are unstable.
