  In the wake of dwindling Moore's Law, to address the rapidly increasing
complexity and cost of fabricating large-scale, monolithic systems-on-chip
(SoCs), the industry has adopted dis-aggregation as a solution, wherein a large
monolithic SoC is partitioned into multiple smaller chiplets that are then
assembled into a large system-in-package (SiP) using advanced packaging
substrates such as silicon interposer. For such interposer-based SiPs, there is
a push to realize on-interposer inter-chiplet communication bandwidth of
multi-Tb/s and end-to-end communication latency of no more than 10ns. This push
comes as the natural progression from some recent prior works on SiP design,
and is driven by the proliferating bandwidth demand of modern data-intensive
workloads. To meet this bandwidth and latency goal, prior works have focused on
a potential solution of using the silicon photonic interposer (SiPhI) for
integrating and interconnecting a large number of chiplets into an SiP. Despite
the early promise, the existing designs of on-SiPhI interconnects still have to
evolve by leaps and bounds to meet the goal of multi-Tb/s bandwidth. However,
the possible design pathways, upon which such an evolution can be achieved,
have not been explored in any prior works yet. In this paper, we have
identified several design pathways that can help evolve on-SiPhI interconnects
to achieve multi-Tb/s aggregate bandwidth. We perform an extensive link-level
and system-level analysis in which we explore these design pathways in
isolation and in different combinations of each other. From our link-level
analysis, we have observed that the design pathways that simultaneously enhance
the spectral range and optical power budget available for wavelength
multiplexing can render aggregate bandwidth of up to 4Tb/s per on-SiPhI link.
