  Secure aggregation is widely used in horizontal Federated Learning (FL), to
prevent leakage of training data when model updates from data owners are
aggregated. Secure aggregation protocols based on Homomorphic Encryption (HE)
have been utilized in industrial cross-silo FL systems, one of the settings
involved with privacy-sensitive organizations such as financial or medical,
presenting more stringent requirements on privacy security. However, existing
HE-based solutions have limitations in efficiency and security guarantees
against colluding adversaries without a Trust Third Party.
  This paper proposes an efficient Doubly Homomorphic Secure Aggregation (DHSA)
scheme for cross-silo FL, which utilizes multi-key Homomorphic Encryption
(MKHE) and seed homomorphic pseudorandom generator (SHPRG) as cryptographic
primitives. The application of MKHE provides strong security guarantees against
up to $N-2$ participates colluding with the aggregator, with no TTP required.
To mitigate the large computation and communication cost of MKHE, we leverage
the homomorphic property of SHPRG to replace the majority of MKHE computation
by computationally-friendly mask generation from SHPRG, while preserving the
security. Overall, the resulting scheme satisfies the stringent security
requirements of typical cross-silo FL scenarios, at the same time providing
high computation and communication efficiency for practical usage. We
experimentally demonstrate our scheme brings a speedup to 20$\times$ over the
state-of-the-art HE-based secure aggregation, and reduces the traffic volume to
approximately 1.5$\times$ inflation over the plain learning setting.
