  Large Language Models (LLMs) have gained significant popularity in the last
few years due to their performance in diverse tasks such as translation,
prediction, or content generation. At the same time, the research community has
shown that LLMs are susceptible to various attacks but can also improve the
security of diverse systems. However, besides enabling more secure systems, how
well do open source LLMs behave as covertext distributions to, e.g., facilitate
censorship resistant communication?
  In this paper, we explore the capabilities of open-source LLM-based covert
channels. We approach this problem from the experimental side by empirically
measuring the security vs. capacity of the open-source LLM model (Llama-7B) to
assess how well it performs as a covert channel. Although our results indicate
that such channels are not likely to achieve high practical bitrates, which
depend on message length and model entropy, we also show that the chance for an
adversary to detect covert communication is low. To ensure that our results can
be used with the least effort as a general reference, we employ a conceptually
simple and concise scheme and only assume public models.
