  Language models can hallucinate when performing complex and detailed
mathematical reasoning. Physics provides a rich domain for assessing
mathematical reasoning capabilities where physical context imbues the use of
symbols which needs to satisfy complex semantics (\textit{e.g.,} units,
tensorial order), leading to instances where inference may be algebraically
coherent, yet unphysical. In this work, we assess the ability of Language
Models (LMs) to perform fine-grained mathematical and physical reasoning using
a curated dataset encompassing multiple notations and Physics subdomains. We
improve zero-shot scores using synthetic in-context examples, and demonstrate
non-linear degradation of derivation quality with perturbation strength via the
progressive omission of supporting premises. We find that the models'
mathematical reasoning is not physics-informed in this setting, where physical
context is predominantly ignored in favour of reverse-engineering solutions.
