  Unlike other accelerators, FPGAs are capable of supporting cache coherency,
thereby turning them into a more powerful architectural option than just a
peripheral accelerator. However, most existing deployments of FPGAs are either
non-cache coherent or support only an asymmetric design where cache coherency
is controlled from the CPU. Taking advantage of a recently released two socket
CPU-FPGA architecture, in this paper we describe ECI, a flexible implementation
of cache coherency on the FPGA capable of supporting both symmetric and
asymmetric protocols. ECI is open and customizable, given applications the
opportunity to fully interact with the cache coherency protocol, thereby
opening up many interesting system design and research opportunities not
available in existing designs. Through extensive microbenchmarks we show that
ECI exhibits highly competitive performance and discuss in detail one use-case
illustrating the benefits of having an open cache coherency stack on the FPGA.
