<p>Sorry for the brevity, maybe I can come back and expand later. There is work in voting and modeling voters that should be relevant. For example, take a model, i.e. in Mallows model, if A is truly cuter than B, then with probability $p &gt; 0.5$ I rank $A$ ahead of $B$ and otherwise I rank B ahead of A. All rankings are made independently.</p>

<p>Then the corresponding voting rule is to compute the MLE underlying "true" ranking that generates the votes we've seen. For this particular model it is the Kemeny rule and it's NP-hard, but there are other models too.</p>

<p>...</p>

<p>(expansion) Condorcet in the 1700s first used the MLE idea. Say that every voter produces a ranking, and in that ranking any given pair of candidates are in the "correct" order with probability $p &gt; 0.5$. Then, given a bunch of people's votes, we want to compute the single linear order that had the highest probability of generating those votes, the MLE. Young showed that the solution of this is the <a href="http://en.wikipedia.org/wiki/Kemeny%E2%80%93Young_method" rel="nofollow">Kemeny voting rule</a>, which minimizes the average bubblesort distance to all the rankings we are given.</p>

<p><a href="http://www.cs.duke.edu/~conitzer/MLEvotingUAI05.pdf" rel="nofollow">Conitzer and Sandholm ('05)</a> extended this idea to other voting rules, with other noise models (e.g. I put the best candidate in the $n$th place with probability $2^{-n}$).</p>

<p>Since then there's been work, e.g. by <a href="http://ijcai.org/papers11/Papers/IJCAI11-082.pdf" rel="nofollow">Xia and Conitzer (09)</a>, on extending these ideas to when people don't see the whole list of alternatives, but just get to see some of the alternatives (submit partial orders). This should be relevant to your setting, I think. There is also at least some empirical-type work in this vein. I am thinking specifically of <a href="http://www.people.fas.harvard.edu/~xagao/papers/adaptivepolling_AIW2012.pdf" rel="nofollow">Pfeiffer et al (12)</a>. The setting is basically the same as yours. They have some sort of adaptive strategy of which kitten pair (or whatever they used) to show to the next voter. The goal was to get a good ranking with fewer pairwise comparisons (i.e. not asking every voter to rank all the kittens).</p>

<p>That's what I know of the theoretical and AI work. There's also a lot of statistics that I know less about. Probably the stats will be most helpful if you want to actually implement something that aggregates partial rankings/observations.</p>

<p>I know that some of the most popular statistical models for these things are the Mallows model, Bradley-Terry, and Plackett-Luce. All of these are noise models for how voters randomly generate a ranking given the true underlying "correct" ranking. In fact, there is so much work about this going on in statistics that I might suggest you post on stats.se, because I am woefully ignorant but I think they (or perhaps the machine learning community as well) have a lot of work on algorithms for (approximately) finding solutions (e.g. the MLE) given a bunch of rankings and assuming that they were generated using one of these models. At least, I hope these keywords help you in your search!</p>

CC BY-SA 3.0
2013-10-07T20:10:06.393Z
Yes, please expand on that. Even if it's just pointers. Thx!
Peter
CC BY-SA 3.0
2013-10-07T18:01:30.660Z
2013-10-07T22:03:00.633Z
2013-10-07T22:03:00.633Z
usul
<p>I want to build a website to find the cutest kitten(TM) there is. People can upload photos of their kittens, but also can vote on which kitten is the cutest. However, I don't want them to rate on a 1 to 10 scale, but kitten cuteness is relative to other kittens.</p>

<p>Every voter is presented with 3 kittens and needs to order them by cuteness using drag and drop.</p>

<p>From these votes I can compute a weighted cyclic graph with the kittens being nodes and the edges being number of votes</p>

<p>Is there an algorithm to approximate a (partial) linear ordering that takes transitive votes into account ? So one can put the kittens on a 1 to 10 scale after all ?</p>

CC BY-SA 3.0
2013-10-07T15:17:20.810Z
You may want to check the Maximum Acyclic Subgraph problem (which has an easy 2-approximation) and the Minimum Feedback Arc Set problem, which is 3-approximated by a simple algorithm when the input is a tournament (http://dimacs.rutgers.edu/~alantha/papers2/aggregating_journal.pdf)
Sasho Nikolov
CC BY-SA 3.0
2013-10-07T17:34:50.593Z
BTW it was only a question of time before kittens made it to CSTheory :)
Sasho Nikolov
CC BY-SA 3.0
2013-10-07T18:54:45.287Z
this is similar to a scene from Social Network movie and the early history/birth of facebook, except images of college students were used.... so called "facemash" aka [hot or not](http://stackoverflow.com/questions/6394466/hot-or-not-facemash-algorithm-why-elos-rating-algo)? there seems to be some possibility of using [elo ratings](http://en.wikipedia.org/wiki/Elo_rating_system#Mathematical_details). there it was done pairwise instead of with triplets.
vzn
CC BY-SA 3.0
2013-10-07T19:01:35.137Z
see also [how to rank 1M images with crowdsourced sort](http://stackoverflow.com/questions/164831/how-to-rank-a-million-images-with-a-crowdsourced-sort)
vzn
CC BY-SA 3.0
2013-10-07T20:59:33.110Z
vzn, thanks for the pointer to elo rankings, that (or glicko) fits my data structure better (so I can even compute it online).
Peter
<graph-theory><graph-algorithms><partial-order>
Linear ordering from weighted directed graph (kittens)
Peter
# Linear ordering from weighted directed graph (kittens)

(Asked by: username_0 on 2013-10-07)

I want to build a website to find the cutest kitten(TM) there is. People can upload photos of their kittens, but also can vote on which kitten is the cutest. However, I don't want them to rate on a 1 to 10 scale, but kitten cuteness is relative to other kittens.

Every voter is presented with 3 kittens and needs to order them by cuteness using drag and drop.

From these votes I can compute a weighted cyclic graph with the kittens being nodes and the edges being number of votes

Is there an algorithm to approximate a (partial) linear ordering that takes transitive votes into account ? So one can put the kittens on a 1 to 10 scale after all ?

**Tags:** graph-theory, graph-algorithms, partial-order

> **Comments:**
> **username_1**: You may want to check the Maximum Acyclic Subgraph problem (which has an easy 2-approximation) and the Minimum Feedback Arc Set problem, which is 3-approximated by a simple algorithm when the input is a tournament (http://dimacs.rutgers.edu/~alantha/papers2/aggregating_journal.pdf)  
> **username_1**: BTW it was only a question of time before kittens made it to CSTheory :)  
> **username_2**: this is similar to a scene from Social Network movie and the early history/birth of facebook, except images of college students were used.... so called "facemash" aka [hot or not](http://stackoverflow.com/questions/6394466/hot-or-not-facemash-algorithm-why-elos-rating-algo)? there seems to be some possibility of using [elo ratings](http://en.wikipedia.org/wiki/Elo_rating_system#Mathematical_details). there it was done pairwise instead of with triplets.  
> **username_2**: see also [how to rank 1M images with crowdsourced sort](http://stackoverflow.com/questions/164831/how-to-rank-a-million-images-with-a-crowdsourced-sort)  
> **username_0**: username_2, thanks for the pointer to elo rankings, that (or glicko) fits my data structure better (so I can even compute it online).  

---

## 1 answers

### Answer by username_3 on 2013-10-07. Score: 2

Sorry for the brevity, maybe I can come back and expand later. There is work in voting and modeling voters that should be relevant. For example, take a model, i.e. in Mallows model, if A is truly cuter than B, then with probability $p > 0.5$ I rank $A$ ahead of $B$ and otherwise I rank B ahead of A. All rankings are made independently.

Then the corresponding voting rule is to compute the MLE underlying "true" ranking that generates the votes we've seen. For this particular model it is the Kemeny rule and it's NP-hard, but there are other models too.

...

(expansion) Condorcet in the 1700s first used the MLE idea. Say that every voter produces a ranking, and in that ranking any given pair of candidates are in the "correct" order with probability $p > 0.5$. Then, given a bunch of people's votes, we want to compute the single linear order that had the highest probability of generating those votes, the MLE. Young showed that the solution of this is the [Kemeny voting rule](http://en.wikipedia.org/wiki/Kemeny%E2%80%93Young_method), which minimizes the average bubblesort distance to all the rankings we are given.

[Conitzer and Sandholm ('05)](http://www.cs.duke.edu/~conitzer/MLEvotingUAI05.pdf) extended this idea to other voting rules, with other noise models (e.g. I put the best candidate in the $n$th place with probability $2^{-n}$).

Since then there's been work, e.g. by [Xia and Conitzer (09)](http://ijcai.org/papers11/Papers/IJCAI11-082.pdf), on extending these ideas to when people don't see the whole list of alternatives, but just get to see some of the alternatives (submit partial orders). This should be relevant to your setting, I think. There is also at least some empirical-type work in this vein. I am thinking specifically of [Pfeiffer et al (12)](http://www.people.fas.harvard.edu/~xagao/papers/adaptivepolling_AIW2012.pdf). The setting is basically the same as yours. They have some sort of adaptive strategy of which kitten pair (or whatever they used) to show to the next voter. The goal was to get a good ranking with fewer pairwise comparisons (i.e. not asking every voter to rank all the kittens).

That's what I know of the theoretical and AI work. There's also a lot of statistics that I know less about. Probably the stats will be most helpful if you want to actually implement something that aggregates partial rankings/observations.

I know that some of the most popular statistical models for these things are the Mallows model, Bradley-Terry, and Plackett-Luce. All of these are noise models for how voters randomly generate a ranking given the true underlying "correct" ranking. In fact, there is so much work about this going on in statistics that I might suggest you post on stats.se, because I am woefully ignorant but I think they (or perhaps the machine learning community as well) have a lot of work on algorithms for (approximately) finding solutions (e.g. the MLE) given a bunch of rankings and assuming that they were generated using one of these models. At least, I hope these keywords help you in your search!

> **Comments:**
> **username_0**: Yes, please expand on that. Even if it's just pointers. Thx!  

---

