  Recent research in Needle-in-a-Haystack (NIAH) benchmarks has explored the
capabilities of Large Language Models (LLMs) in retrieving contextual
information from large text documents. However, as LLMs become increasingly
integrated into software development processes, it is crucial to evaluate their
performance in code-based environments. As LLMs are further developed for
program synthesis, we need to ensure that LLMs can understand syntax and write
syntactically correct code. As a step in ensuring LLMs understand syntax, LLMs
can be evaluated in their ability to find and detect syntax bugs. Our
benchmark, Bug In The Code Stack (BICS), is designed to assess the ability of
LLMs to identify simple syntax bugs within large source code. Our findings
reveal three key insights: (1) code-based environments pose significantly more
challenge compared to text-based environments for retrieval tasks, (2) there is
a substantial performance disparity among different models, and (3) there is a
notable correlation between longer context lengths and performance degradation,
though the extent of this degradation varies between models.
