  Hardware isolation primitives such as secure enclaves aim to protect
sensitive programs, but remain vulnerable to transient execution attacks.
Complete microarchitectural isolation is not a satisfactory defense mechanism
as it leaves out public shared memory, critical for usability and application
performance. Conversely, hardware-software co-designs for secure speculation
can counter these attacks but are not yet practical, since they make
assumptions on the speculation modes, the exposed microarchitectural state, and
the software, which are all hard to support for the entire software stack. This
paper advocates for processors to incorporate microarchitectural isolation
primitives and mechanisms for controlled speculation, enabling different
execution modes. These modes can restrict what is exposed to an attacker,
effectively balancing performance and program-analysis complexity. We introduce
two mechanisms to securely share memory between an enclave and an untrusted OS
in an out-of-order processor. We show that our two modes are complementary,
achieving speculative non-interference with a reasonable performance impact,
while requiring minimal code annotation and simple program analysis doable by
hand. Our prototype, Citadel, is a multicore processor running on an FPGA,
booting untrusted Linux, and supporting comprehensive enclave capabilities,
such as shared memory, and remote attestation. To our knowledge, Citadel is the
first end-to-end enclave platform to run secure applications, such as
cryptographic libraries or small private inference workloads, on a speculative
out-of-order multicore processor while protecting against a significant class
of side-channel attacks.
