  Coalition formation is a key problem in automated negotiation among
self-interested agents, and other multiagent applications. A coalition of
agents can sometimes accomplish things that the individual agents cannot, or
can do things more efficiently. However, motivating the agents to abide to a
solution requires careful analysis: only some of the solutions are stable in
the sense that no group of agents is motivated to break off and form a new
coalition. This constraint has been studied extensively in cooperative game
theory. However, the computational questions around this constraint have
received less attention. When it comes to coalition formation among software
agents (that represent real-world parties), these questions become increasingly
explicit.
  In this paper we define a concise general representation for games in
characteristic form that relies on superadditivity, and show that it allows for
efficient checking of whether a given outcome is in the core. We then show that
determining whether the core is nonempty is $\mathcal{NP}$-complete both with
and without transferable utility. We demonstrate that what makes the problem
hard in both cases is determining the collaborative possibilities (the set of
outcomes possible for the grand coalition), by showing that if these are given,
the problem becomes tractable in both cases. However, we then demonstrate that
for a hybrid version of the problem, where utility transfer is possible only
within the grand coalition, the problem remains $\mathcal{NP}$-complete even
when the collaborative possibilities are given.
