  AI-supported programming has taken giant leaps with tools such as Meta's
Llama and openAI's chatGPT. These are examples of stochastic sources of
programs and have already greatly influenced how we produce code and teach
programming. If we consider input to such models as a stochastic source, a
natural question is, what is the relation between the input and the output
distributions, between the chatGPT prompt and the resulting program?
  In this paper, we will show how the relation between random Python programs
generated from chatGPT can be described geometrically and topologically using
Tree-edit distances between the program's syntax trees and without explicit
modeling of the underlying space. A popular approach to studying
high-dimensional samples in a metric space is to use low-dimensional embedding
using, e.g., multidimensional scaling. Such methods imply errors depending on
the data and dimension of the embedding space. In this article, we propose to
restrict such projection methods to purely visualization purposes and instead
use geometric summary statistics, methods from spatial point statistics, and
topological data analysis to characterize the configurations of random programs
that do not rely on embedding approximations. To demonstrate their usefulness,
we compare two publicly available models: ChatGPT-4 and TinyLlama, on a simple
problem related to image processing.
  Application areas include understanding how questions should be asked to
obtain useful programs; measuring how consistently a given large language model
answers; and comparing the different large language models as a programming
assistant. Finally, we speculate that our approach may in the future give new
insights into the structure of programming languages.
