  Traditional algorithmic fairness notions rely on label feedback, which can
only be elicited from expert critics. However, in most practical applications,
several non-expert stakeholders also play a major role in the system and can
have distinctive opinions about the decision making philosophy. For example, in
kidney placement programs, transplant surgeons are very wary about accepting
kidney offers for black patients due to genetic reasons. However, non-expert
stakeholders in kidney placement programs (e.g. patients, donors and their
family members) may misinterpret such decisions from the perspective of social
discrimination. This paper evaluates group fairness notions from the viewpoint
of non-expert stakeholders, who can only provide binary
\emph{agreement/disagreement feedback} regarding the decision in context.
Specifically, two types of group fairness notions have been identified: (i)
\emph{definite notions} (e.g. calibration), which can be evaluated exactly
using disagreement feedback, and (ii) \emph{indefinite notions} (e.g. equal
opportunity) which suffer from uncertainty due to lack of label feedback. In
the case of indefinite notions, bounds are presented based on disagreement
rates, and an estimate is constructed based on established bounds. The efficacy
of all our findings are validated empirically on real human feedback dataset.
