This paper seems quite interesting, and comes up with an attractiveness rating by comparing a face's relative distance to two averages: of attractive and of unattractive faces.
For some traits, the attractive and unattractive faces may have the same, or similar average. For example, very broad or very narrow mouths may be both considered unattractive. By averaging them out, the result may be an attractive mouth of intermediate proportions.
But, on other traits, the unattractiveness is asymmetrical, and hence will increase in a definite direction. So while very thin and very fat people are probably not very attractive, the weight of unattractiveness falls on the fatness side, and unattractive people tend to be systematically fatter-looking than attractive ones.
For traits of the first kind, the discriminating power will be low: at the limit, attractive and unattractive faces will have the same average, and all faces will be equidistant from the two prototypes. As attractive and unattractive faces become more differentiated -for traits of the second kind- the ability to distinguish between them will increase.
Neural Comput. 2008 Oct 17. [Epub ahead of print]
A Bi-Prototype Theory of Facial Attractiveness.
Chang F, Chou CH.
The attractiveness of human faces can be predicted with a high degree of accuracy if we represent the faces as feature vectors and compute their relative distances from two prototypes: the average of attractive faces and the average of unattractive faces. Moreover, the degree of attractiveness, defined in terms of the relative distance, exhibits a high degree of correlation with the average rating scores given by human assessors. These findings motivate a bi-prototype theory that relates facial attractiveness to the averages of attractive and unattractive faces rather than the average of all faces, as previously hypothesized by some researchers.