  In recent years, Approximate Nearest Neighbor Search (ANNS) has played a
pivotal role in modern search and recommendation systems, especially in
emerging LLM applications like Retrieval-Augmented Generation. There is a
growing exploration into harnessing the parallel computing capabilities of GPUs
to meet the substantial demands of ANNS. However, existing systems primarily
focus on offline scenarios, overlooking the distinct requirements of online
applications that necessitate real-time insertion of new vectors. This
limitation renders such systems inefficient for real-world scenarios. Moreover,
previous architectures struggled to effectively support real-time insertion due
to their reliance on serial execution streams. In this paper, we introduce a
novel Real-Time Adaptive Multi-Stream GPU ANNS System (RTAMS-GANNS). Our
architecture achieves its objectives through three key advancements: 1) We
initially examined the real-time insertion mechanisms in existing GPU ANNS
systems and discovered their reliance on repetitive copying and memory
allocation, which significantly hinders real-time effectiveness on GPUs. As a
solution, we introduce a dynamic vector insertion algorithm based on memory
blocks, which includes in-place rearrangement. 2) To enable real-time vector
insertion in parallel, we introduce a multi-stream parallel execution mode,
which differs from existing systems that operate serially within a single
stream. Our system utilizes a dynamic resource pool, allowing multiple streams
to execute concurrently without additional execution blocking. 3) Through
extensive experiments and comparisons, our approach effectively handles varying
QPS levels across different datasets, reducing latency by up to 40%-80%. The
proposed system has also been deployed in real-world industrial search and
recommendation systems, serving hundreds of millions of users daily, and has
achieved good results.
