I'm interested in looking at several different metrics for ranking algorithms - there are a few listed on the Learning to Rank wikipedia page, including: 

• Mean average precision (MAP);

• DCG and NDCG;

• Precision@n, NDCG@n, where "@n" denotes that the metrics are evaluated only on top n documents;

• Mean reciprocal rank;

• Kendall's tau

• Spearman's Rho

• Expected reciprocal rank 

• Yandex's pfound

but it isn't clear to me what are the advantages/disadvantages of each or when you may choose one over another (or what it would mean if one algorithm outperformed another on NDGC but was worse when evaluated with MAP).

Is there anywhere I can go to learn more about these questions?

I am actually looking for the same answer, however I should be able to at least partially answer your question.

All of the metrics that you have mentioned have different traits and, unfortunately, the one you should pick depends on what you actually would like to measure. Here are some things that it would be worth to have in mind:


Spearman's rho metric penalises errors at the top of the list with the same weight as mismatches on the bottom, so in most cases this is not the metric to use for evaluating rankings
DCG & NDCG are one of the few metrics that take into account the non-binary utility function, so you can describe how useful is a record and not whether it's useful.
DCG & NDCG have fixed weighs for positions, so a document in a given position has always the same gain and discount independently of the documents shown above it
You usually would prefer NDCG over DCG, because it normalises the value by the number of relevant documents
MAP is supposed to be a classic and a 'go-to' metric for this problem and it seems to be a standard in the field.
(N)DCG should be always computed for a fixed amount of records (@k), because it has a long tail (lots of irrelevant records at the end of the ranking highly bias the metric). This doesn't apply to MAP.
Mean Reciprocal Rank only marks the position of the first relevant document, so if you care about as many relevant docs as possible to be high on the list, then this should not be your choice
Kendall's tau only handles binary utility function, it also should be computed @k (similar to NDCG)


Valuable resources:


Victor Lavrenko lecture on YouTube - it's only a link to the MAP vs NDCG episode, but the whole lecture includes much more (including Kendall's Tau). You should definitely check it out, great lecture!
ERR paper


Can't post more links, because of the fresh account :) If anybody has some more remarks or ideas, I would be happy to hear them as well!
