  Motion estimation is one of the important procedures in the all video
encoders. Most of the complexity of the video coder depends on the complexity
of the motion estimation step. The original motion estimation algorithm has a
remarkable complexity and therefore many improvements were proposed to enhance
the crude version of the motion estimation. The basic idea of many of these
works were to optimize some distortion function for mean squared error (MSE) or
sum of absolute difference (SAD) in block matching But it is shown that these
metrics do not conclude the quality as it is, on the other hand, they are not
compatible with the human visual system (HVS). In this paper we explored the
usage of the image quality metrics in the video coding and more specific in the
motion estimation. We have utilized the perceptual image quality metrics
instead of MSE or SAD in the block based motion estimation. Three different
metrics have used: structural similarity or SSIM, complex wavelet structural
similarity or CW-SSIM, visual information fidelity or VIF. Experimental results
showed that usage of the quality criterions can improve the compression rate
while the quality remains fix and thus better quality in coded video at the
same bit budget.
