  Visual Programming has recently emerged as an alternative to end-to-end
black-box visual reasoning models. This type of method leverages Large Language
Models (LLMs) to generate the source code for an executable computer program
that solves a given problem. This strategy has the advantage of offering an
interpretable reasoning path and does not require finetuning a model with
task-specific data. We propose PropTest, a general strategy that improves
visual programming by further using an LLM to generate code that tests for
visual properties in an initial round of proposed solutions. Our method
generates tests for data-type consistency, output syntax, and semantic
properties. PropTest achieves comparable results to state-of-the-art methods
while using publicly available LLMs. This is demonstrated across different
benchmarks on visual question answering and referring expression comprehension.
Particularly, PropTest improves ViperGPT by obtaining 46.1\% accuracy (+6.0\%)
on GQA using Llama3-8B and 59.5\% (+8.1\%) on RefCOCO+ using CodeLlama-34B.
