  Recently, large language models (LLMs) have drawn extensive attention from
academia and the public, due to the advent of the ChatGPT. While LLMs show
their astonishing ability in text generation for various tasks, privacy
concerns limit their usage in real-life businesses. More specifically, either
the user's inputs (the user sends the query to the model-hosting server) or the
model (the user downloads the complete model) itself will be revealed during
the usage. Vertical federated learning (VFL) is a promising solution to this
kind of problem. It protects both the user's input and the knowledge of the
model by splitting the model into a bottom part and a top part, which is
maintained by the user and the model provider, respectively. However, in this
paper, we demonstrate that in LLMs, VFL fails to protect the user input since
it is simple and cheap to reconstruct the input from the intermediate
embeddings. Experiments show that even with a commercial GPU, the input
sentence can be reconstructed in only one second. We also discuss several
possible solutions to enhance the privacy of vertical federated LLMs.
