I am using Cohen's Kappa to calculate the inter-agreement between two judges.

It is calculated as:

$ \frac{P(A) - P(E)}{1 - P(E)}  $

where $P(A)$ is the proportion of agreement and $P(E)$ the probability of agreement by chance.

Now for the following dataset, I get the expected results:

User A judgements: 
  - 1, true
  - 2, false
User B judgements: 
  - 1, false
  - 2, false
Proportion agreed: 0.5
Agreement by chance: 0.625
Kappa for User A and B: -0.3333333333333333


We can see that both judges have not agreed very well. However in the following case where both judges evaluate one criteria, kappa evaluates to zero:

User A judgements: 
  - 1, false
User B judgements: 
  - 1, false
Proportion agreed: 1.0
Agreement by chance: 1.0
Kappa for User A and B: 0


Now I can see that the agreement by chance is obviously 1, which leads to kappa being zero, but does this count as a reliable result? The problem is that I normally don't have more than two judgements per criteria, so these will all never evaluate to any kappa greater than 0, which I think is not very representative.

Am I right with my calculations? Can I use a different method to calculate inter-agreement?

Here we can see that kappa works fine for multiple judgements:

User A judgements: 
  - 1, false
  - 2, true
  - 3, false
  - 4, false
  - 5, true
User A judgements: 
  - 1, true
  - 2, true
  - 3, false
  - 4, true
  - 5, false
Proportion agreed: 0.4
Agreement by chance: 0.5
Kappa for User A and B: -0.19999999999999996


The key difference is that ROC curves will be the same no matter what the baseline probability is, but PR curves may be more useful in practice for needle-in-haystack type problems or problems where the "positive" class is more interesting than the negative class.

To show this, first let's start with a very nice way to define precision, recall and specificity.  Assume you have a "positive" class called 1 and a "negative" class called 0.  $\hat{Y}$ is your estimate of the true class label $Y$.  Then:
$$
\begin{aligned}
&\text{Precision} &= P(Y = 1 | \hat{Y} = 1)  \\
&\text{Recall} = \text{Sensitivity} &= P(\hat{Y} = 1 | Y = 1)  \\
&\text{Specificity} &= P(\hat{Y} = 0 | Y = 0)
\end{aligned}
$$
The key thing to note is that sensitivity/recall and specificity, which make up the ROC curve, are probabilities conditioned on the true class label.  Therefore, they will be the same regardless of what $P(Y = 1)$ is.  Precision is a probability conditioned on your estimate of the class label and will thus vary if you try your classifier in different populations with different baseline $P(Y = 1)$.  However, it may be more useful in practice if you only care about one population with known background probability and the "positive" class is much more interesting than the "negative" class.  (IIRC precision is popular in the document retrieval field, where this is the case.)  This is because it directly answers the question, "What is the probability that this is a real hit given my classifier says it is?". 

Interestingly, by Bayes' theorem you can work out cases where specificity can be very high and precision very low simultaneously.  All you have to do is assume $P(Y = 1)$ is very close to zero.  In practice I've developed several classifiers with this performance characteristic when searching for needles in DNA sequence haystacks.

IMHO when writing a paper you should provide whichever curve answers the question you want answered (or whichever one is more favorable to your method, if you're cynical).  If your question is: "How meaningful is a positive result from my classifier given the baseline probabilities of my problem?", use a PR curve.  If your question is, "How well can this classifier be expected to perform in general, at a variety of different baseline probabilities?", go with a ROC curve.
