  Collective Perception has attracted significant attention in recent years due
to its advantage for mitigating occlusion and expanding the field-of-view,
thereby enhancing reliability, efficiency, and, most crucially, decision-making
safety. However, developing collective perception models is highly resource
demanding due to extensive requirements of processing input data for many
agents, usually dozens of images and point clouds for a single frame. This not
only slows down the model development process for collective perception but
also impedes the utilization of larger models. In this paper, we propose an
agent-based training framework that handles the deep learning modules and agent
data separately to have a cleaner data flow structure. This framework not only
provides an API for flexibly prototyping the data processing pipeline and
defining the gradient calculation for each agent, but also provides the user
interface for interactive training, testing and data visualization. Training
experiment results of four collective object detection models on the prominent
collective perception benchmark OPV2V show that the agent-based training can
significantly reduce the GPU memory consumption and training time while
retaining inference performance. The framework and model implementations are
available at \url{https://github.com/YuanYunshuang/CoSense3D}
