---
abstract: |
    New generation radio interferometers encode signals from thousands of antenna feeds across large bandwidth. Channelizing and correlating this data requires networking capabilities that can handle unprecedented data rates with reasonable cost. The Canadian Hydrogen Intensity Mapping Experiment (CHIME) correlator processes 8-bits from $N=2048$ digitizer inputs across 400 MHz of bandwidth. Measured in $N^2~\times
      $ bandwidth, it is the largest radio correlator that has been built. Its digital back-end must exchange and reorganize the 6.6 terabit/s produced by its 128 digitizing and channelizing nodes, and feed it to the 256-node spatial correlator in a way that each node obtains data from all digitizer inputs but across a small fraction of the bandwidth [(i.e. ‘corner-turn’).]{} In order to maximize performance and reliability of the corner-turn system while minimizing cost, a custom networking solution has been implemented. The system makes use of Field Programmable Gate Array (FPGA) transceivers to implement direct, passive, full-mesh, high speed serial connections between sixteen circuit boards in a crate, to exchange data between crates, and to offload the data to a cluster of 256 graphics processing unit (GPU) nodes using standard 10 Gbit/s Ethernet links. The GPU nodes complete the corner-turn by combining data from all crates and then computing visibilities. Eye diagrams and frame error counters confirm error-free operation of the corner-turn network in both the currently operating CHIME Pathfinder telescope (a prototype for the full CHIME telescope) and a representative fraction of the full CHIME hardware providing an end-to-end system validation. An analysis of an equivalent corner-turn system built with Ethernet switches instead of custom passive data links is provided.
address: |
     $^{\dagger}$Physics Department, McGill University, Montreal, Quebec H3A 2T8, Canada\
    $^{\S}$Canadian Institute for Advanced Research, Toronto, Canada\
    $^{\ddagger}$LCSEE, West Virginia University, Morgantown, WV 26505\
    $^{\star}$ Three-Speed Logic, Inc., Vancouver, Canada\
author:
- |
    K. Bandura$^{{\ddagger},{\dagger}}$, J.F. Cliche$^{\dagger}$, M.A. Dobbs$^{{\dagger},{\S}}$,\
    A.J. Gilbert$^{\dagger}$, D. Ittah$^{{\dagger}}$, J. Mena Parra$^{\dagger}$, G. Smecher$^{{\dagger}, {\star}}$
bibliography:
- 'CornerTurnReferences.bib'
title: 'ICE-based Custom Full-Mesh Network for the CHIME High Bandwidth Radio Astronomy Correlator'
---

; ; ;

Conclusions {#s_conclusions}
===========

A custom full mesh corner-turn network for radio astronomy applications has been developed and applied to the CHIME instrument. The implementation is comprised of direct FPGA to FPGA connections within a crate of sixteen ICE motherboards, similar links between crates of boards, and direct connections from each ICE motherboard to GPU X-engine nodes. This allows for simple communication protocols, error handling, and data alignment all within the FPGA, and offers the potential to scale to much larger applications. The system scales from sixteen antenna inputs to the 2048 inputs required by CHIME, and far beyond. Large correlators may be implemented by increasing the number of crates in each quadrant from two to three or even five with full hardware support present.

The corner-turn solution has been implemented on the CHIME pathfinder and is found to be very reliable, as expected from a passive network, with error free operation between F-engine and X-engine. Combined with lab tests to ensure compatibility with the $N$=2048 input, 400 MHz configuration, the system has been validated from end-to-end for the 6.6 Tbit/s implementation for full CHIME.

The full CHIME FX correlator is scheduled for commissioning in late 2016.

Acknowledgments {#s_acknowledgments}
===============

We are very grateful for the warm reception and skillful help we have received from the staff of the Dominion Radio Astrophysical Observatory, operated by the National Research Council Canada.

We acknowledge funding from the Natural Sciences and Engineering Research Council of Canada, Canadian Institute for Advanced Research, Canadian Foundation for Innovation and le Cofinancement gouvernement du Québec-FCI. We acknowledge generous support from the Xilinx University Program.
