  Combining Federated Learning (FL) with a Trusted Execution Environment (TEE)
is a promising approach for realizing privacy-preserving FL, which has garnered
significant academic attention in recent years. Implementing the TEE on the
server side enables each round of FL to proceed without exposing the client's
gradient information to untrusted servers. This addresses usability gaps in
existing secure aggregation schemes as well as utility gaps in differentially
private FL. However, to address the issue using a TEE, the vulnerabilities of
server-side TEEs need to be considered -- this has not been sufficiently
investigated in the context of FL. The main technical contribution of this
study is the analysis of the vulnerabilities of TEE in FL and the defense.
First, we theoretically analyze the leakage of memory access patterns,
revealing the risk of sparsified gradients, which are commonly used in FL to
enhance communication efficiency and model accuracy. Second, we devise an
inference attack to link memory access patterns to sensitive information in the
training dataset. Finally, we propose an oblivious yet efficient aggregation
algorithm to prevent memory access pattern leakage. Our experiments on
real-world data demonstrate that the proposed method functions efficiently in
practical scales.
