  Modern operating room is becoming increasingly complex, requiring innovative
intra-operative support systems. While the focus of surgical data science has
largely been on video analysis, integrating surgical computer vision with
language capabilities is emerging as a necessity. Our work aims to advance
Visual Question Answering (VQA) in the surgical context with scene graph
knowledge, addressing two main challenges in the current surgical VQA systems:
removing question-condition bias in the surgical VQA dataset and incorporating
scene-aware reasoning in the surgical VQA model design. First, we propose a
Surgical Scene Graph-based dataset, SSG-QA, generated by employing segmentation
and detection models on publicly available datasets. We build surgical scene
graphs using spatial and action information of instruments and anatomies. These
graphs are fed into a question engine, generating diverse QA pairs. Our SSG-QA
dataset provides a more complex, diverse, geometrically grounded, unbiased, and
surgical action-oriented dataset compared to existing surgical VQA datasets. We
then propose SSG-QA-Net, a novel surgical VQA model incorporating a lightweight
Scene-embedded Interaction Module (SIM), which integrates geometric scene
knowledge in the VQA model design by employing cross-attention between the
textual and the scene features. Our comprehensive analysis of the SSG-QA
dataset shows that SSG-QA-Net outperforms existing methods across different
question types and complexities. We highlight that the primary limitation in
the current surgical VQA systems is the lack of scene knowledge to answer
complex queries. We present a novel surgical VQA dataset and model and show
that results can be significantly improved by incorporating geometric scene
features in the VQA model design. The source code and the dataset will be made
publicly available at: https://github.com/CAMMA-public/SSG-QA
