  Real-time 3D reconstruction of surgical scenes plays a vital role in
computer-assisted surgery, holding a promise to enhance surgeons' visibility.
Recent advancements in 3D Gaussian Splatting (3DGS) have shown great potential
for real-time novel view synthesis of general scenes, which relies on accurate
poses and point clouds generated by Structure-from-Motion (SfM) for
initialization. However, 3DGS with SfM fails to recover accurate camera poses
and geometry in surgical scenes due to the challenges of minimal textures and
photometric inconsistencies. To tackle this problem, in this paper, we propose
the first SfM-free 3DGS-based method for surgical scene reconstruction by
jointly optimizing the camera poses and scene representation. Based on the
video continuity, the key of our method is to exploit the immediate optical
flow priors to guide the projection flow derived from 3D Gaussians. Unlike most
previous methods relying on photometric loss only, we formulate the pose
estimation problem as minimizing the flow loss between the projection flow and
optical flow. A consistency check is further introduced to filter the flow
outliers by detecting the rigid and reliable points that satisfy the epipolar
geometry. During 3D Gaussian optimization, we randomly sample frames to
optimize the scene representations to grow the 3D Gaussian progressively.
Experiments on the SCARED dataset demonstrate our superior performance over
existing methods in novel view synthesis and pose estimation with high
efficiency. Code is available at https://github.com/wrld/Free-SurGS.
