  Data pre-processing is a fundamental component in any data-driven
application. With the increasing complexity of data processing operations and
volume of data, Cylon, a distributed dataframe system, is developed to
facilitate data processing both as a standalone application and as a library,
especially for Python applications. While Cylon shows promising performance
results, we experienced difficulties trying to integrate with frameworks
incompatible with the traditional Message Passing Interface (MPI). While MPI
implementations encompass scalable and efficient communication routines, their
process launching mechanisms work well with mainstream HPC systems but are
incompatible with some environments that adopt their own resource management
systems. In this work, we alleviated this issue by directly integrating the
Unified Communication X (UCX) framework, which supports a variety of classic
HPC and non-HPC process-bootstrapping mechanisms as our communication
framework. While we experimented with our methodology on Cylon, the same
technique can be used to bring MPI communication to other applications that do
not employ MPI's built-in process management approach.
