  New HPC machines are getting close to the exascale. Power consumption for
those machines has been increasing, and researchers are studying ways to reduce
it. A second trend is HPC machines' growing complexity, with increasing
heterogeneous hardware components and different clusters architectures
cooperating in the same machine. We refer to these environments with the term
heterogeneous multi-cluster environments. With the aim of optimizing
performance and energy consumption in these environments, this paper proposes
an Energy-Aware-Multi-Cluster (EAMC) job scheduling policy. EAMC-policy is able
to optimize the scheduling and placement of jobs by predicting performance and
energy consumption of arriving jobs for different hardware architectures and
processor frequencies, reducing workload's energy consumption, makespan, and
response time. The policy assigns a different priority to each job-resource
combination so that the most efficient ones are favored, while less efficient
ones are still considered on a variable degree, reducing response time and
increasing cluster utilization. We implemented EAMC-policy in Slurm, and we
evaluated a scenario in which two CPU clusters collaborate in the same machine.
Simulations of workloads running applications modeled from real-world show a
reduction of response time and makespan by up to 25% and 6% while saving up to
20% of total energy consumed when compared to policies minimizing runtime, and
by 49%, 26%, and 6% compared to policies minimizing energy.
