  Classification of transient and variable light curves is an essential step in
using astronomical observations to develop an understanding of their underlying
physical processes. However, upcoming deep photometric surveys, including the
Large Synoptic Survey Telescope (LSST), will produce a deluge of low
signal-to-noise data for which traditional labeling procedures are
inappropriate. Probabilistic classification is more appropriate for the data
but are incompatible with the traditional metrics used on deterministic
classifications. Furthermore, large survey collaborations intend to use these
classification probabilities for diverse science objectives, indicating a need
for a metric that balances a variety of goals. We describe the process used to
develop an optimal performance metric for an open classification challenge that
seeks probabilistic classifications and must serve many scientific interests.
The Photometric LSST Astronomical Time-series Classification Challenge
(PLAsTiCC) is an open competition aiming to identify promising techniques for
obtaining classification probabilities of transient and variable objects by
engaging a broader community both within and outside astronomy. Using mock
classification probability submissions emulating archetypes of those
anticipated of PLAsTiCC, we compare the sensitivity of metrics of
classification probabilities under various weighting schemes, finding that they
yield qualitatively consistent results. We choose as a metric for PLAsTiCC a
weighted modification of the cross-entropy because it can be meaningfully
interpreted. Finally, we propose extensions of our methodology to ever more
complex challenge goals and suggest some guiding principles for approaching the
choice of a metric of probabilistic classifications.
