  Deep neural networks (DNN) have made impressive progress in the
interpretation of image data, so that it is conceivable and to some degree
realistic to use them in safety critical applications like automated driving.
From an ethical standpoint, the AI algorithm should take into account the
vulnerability of objects or subjects on the street that ranges from "not at
all", e.g. the road itself, to "high vulnerability" of pedestrians. One way to
take this into account is to define the cost of confusion of one semantic
category with another and use cost-based decision rules for the interpretation
of probabilities, which are the output of DNNs. However, it is an open problem
how to define the cost structure, who should be in charge to do that, and
thereby define what AI-algorithms will actually "see". As one possible answer,
we follow a participatory approach and set up an online survey to ask the
public to define the cost structure. We present the survey design and the data
acquired along with an evaluation that also distinguishes between perspective
(car passenger vs. external traffic participant) and gender. Using simulation
based $F$-tests, we find highly significant differences between the groups.
These differences have consequences on the reliable detection of pedestrians in
a safety critical distance to the self-driving car. We discuss the ethical
problems that are related to this approach and also discuss the problems
emerging from human-machine interaction through the survey from a psychological
point of view. Finally, we include comments from industry leaders in the field
of AI safety on the applicability of survey based elements in the design of AI
functionalities in automated driving.
