  Surgical scene simulation plays a crucial role in surgical education and
simulator-based robot learning. Traditional approaches for creating these
environments with surgical scene involve a labor-intensive process where
designers hand-craft tissues models with textures and geometries for soft body
simulations. This manual approach is not only time-consuming but also limited
in the scalability and realism. In contrast, data-driven simulation offers a
compelling alternative. It has the potential to automatically reconstruct 3D
surgical scenes from real-world surgical video data, followed by the
application of soft body physics. This area, however, is relatively uncharted.
In our research, we introduce 3D Gaussian as a learnable representation for
surgical scene, which is learned from stereo endoscopic video. To prevent
over-fitting and ensure the geometrical correctness of these scenes, we
incorporate depth supervision and anisotropy regularization into the Gaussian
learning process. Furthermore, we apply the Material Point Method, which is
integrated with physical properties, to the 3D Gaussians to achieve realistic
scene deformations. Our method was evaluated on our collected in-house and
public surgical videos datasets. Results show that it can reconstruct and
simulate surgical scenes from endoscopic videos efficiently-taking only a few
minutes to reconstruct the surgical scene-and produce both visually and
physically plausible deformations at a speed approaching real-time. The results
demonstrate great potential of our proposed method to enhance the efficiency
and variety of simulations available for surgical education and robot learning.
