Spotlight: Perplexity AI Serves 400 Million Search Queries a Month Using NVIDIA Inference Stack
The demand for AI-enabled services continues to grow rapidly, placing increasing pressure on IT and infrastructure teams. These teams are tasked with provisioning the necessary hardware and software to meet that demand while simultaneously balancing cost efficiency with optimal user experience. This challenge was faced by the inference team at Perplexity AI, an AI-powered search engine that handles more than 435 million queries each month. Each query represents multiple AI inference requests.
To meet this demand, the Perplexity inference team turned to
NVIDIA H100 Tensor Core GPUs
,
NVIDIA Triton Inference Server
, and
NVIDIA TensorRT-LLM
for cost-effective
large language model (LLM)
deployment. This post details some of the deployment best practices and TCO savings based on their hands-on experience.
Serving multiple AI models simultaneously
To support its extensive user base and serve a wide range of requests—spanning search, summarization, and question answering, among others—the inference team at Perplexity serves over 20 AI models simultaneously. This includes different variations of the popular open source Llama 3.1 models like 8B, 70B, and 405B.
To match each user request with the appropriate model, the company relies on smaller classifier models that help determine user intent. User tasks detected by the classifiers, like text completion, are then routed to specific models deployed on GPU pods. Each pod consists of one or more NVIDIA H100 GPUs and is managed by an NVIDIA Triton Inference Server instance. The pods operate under strict service-level agreements (SLAs) for both cost efficiency and user interactivity.
Figure 1. Perplexity AI deployed LLM production on a massive scale using NVIDIA HGX H100 four-GPU and eight-GPU systems featuring dynamic scalability, model parallelism, and load balancing capabilities
To accommodate the large Perplexity user base and fluctuating traffic throughout the day, the pods are hosted within a Kubernetes cluster. They feature a front-end scheduler built in-house that routes traffic to the appropriate pod based on their load and usage, ensuring that the SLAs are consistently met.
The scheduling algorithm used by the front-end scheduler can affect inter-token latency, particularly in improving the worst percentile of performance (Figure 2). The team at Perplexity constantly looks for new scheduler optimizations, including how to better account for sequence length variations across requests.
Figure 2. Inter-token latency distribution for round-robin, least requests, and power of two random choices load balancing strategies during an internal Perplexity benchmark at a moderate request rate of 16 QPS
Triton Inference Server is a critical component of Perplexity’s deployment architecture. It serves optimized models across various backends, batches incoming user requests, and provides GPU utilization metrics to the scheduler. This supports scaling up or down the number of deployments and GPUs based on the amount of the inference requests.
For a detailed guide on how to deploy NVIDIA Triton with Kubernetes, see
Scaling LLMs with NVIDIA Triton and NVIDIA TensorRT-LLM Using Kubernetes
.
Meeting strict service-level agreements
To define the right SLAs for the company’s diverse use cases, Perplexity’s inference team conducts comprehensive A/B testing, evaluating different configurations and their impact on user experience. Their goal is to maximize GPU utilization while consistently meeting the target SLA for each specific use case. By improving batching while meeting target SLAs, inference serving cost is optimized.
For smaller models, such as embedding models under 1 billion parameters used in real-time retrieval, the focus is on achieving the lowest possible latency. These are typically hidden from the user and are part of a broader workflow. As a result, configurations for these queries typically have low batch sizes. Given the smaller memory footprints of these models, the team runs multiple models concurrently on the NVIDIA H100 GPU to maintain high resource utilization.
For user-facing models such as Llama 8B, 70B, and 405B, which have a greater impact on user experience and deployment costs, the team conducts a deeper performance analysis and evaluates key metrics such as time to first token, tokens per second per user, and cost per million queries.
To optimize performance while controlling costs, Perplexity parallelizes their deployment of these models across multiple GPUs. Due to the strict SLAs, the team opted to increase tensor parallelism to four and eight GPUs, which they found yields lower serving costs for very latency-sensitive requests within a fixed GPU budget. Data or pipeline parallelism was useful for maximizing throughput in less latency-sensitive settings.
The Perplexity team now uses TensorRT-LLM in combination with proprietary LLM runtimes built with optimized CUDA kernels to successfully serve the Llama-based models within their strict SLAs at minimal costs.
Figure 3. Sharding the Llama 8B model using tensor parallelism across four NVIDIA Hopper GPUs reduces Perplexity’s relative cost per million token by up to 3x for latency-sensitive requests
Ultimately, the Perplexity inference team’s decision to host models depends on their ability to serve these models at a lower cost while still meeting their strict SLAs, compared to using third-party LLM provider APIs. For example, the team estimated that they were able to save approximately $1 million annually by serving models that power their Related-Questions feature on cloud-hosted NVIDIA GPUs. The Related-Questions feature offers Perplexity users suggested follow-up questions to facilitate deeper dives after a search query.
Delivering new levels of performance
The inference team at Perplexity adopts a comprehensive, full-stack approach to their road map, consistently optimizing and enhancing every layer of the stack—from applications and use cases to inference serving middleware and hardware accelerators.
In terms of inference serving middleware, the team is actively collaborating with the NVIDIA Triton engineering team to deploy
disaggregating serving
, a groundbreaking technique that separates the prefill and decode inference phases of an LLM workflow onto separate NVIDIA GPUs. This technique significantly boosts overall system throughput while meeting SLAs, translating to lower cost per token. Additionally, this technique gives Perplexity the flexibility to use different NVIDIA GPU products for each inference phase given its specific hardware resource requirements.
The Perplexity team understands that optimizing the software stack can only drive performance improvements to a certain extent. To deliver new levels of performance, hardware innovations are crucial. This is why they are eager to assess the
NVIDIA Blackwell platform
.
NVIDIA Blackwell delivers
significant performance leaps
enabled by numerous technology innovations, including the second-generation Transformer Engine with support for the FP4 data format, fifth-generation NVLink and NVSwitch enabling significantly larger NVLink domains, and more. Collectively these innovations deliver 30x improvement in inference performance for
trillion parameter LLMs
.
Get started
NVIDIA Triton Inference Server
and
NVIDIA TensorRT
are open-source projects available on GitHub. They are also available as Docker containers that can be pulled from
NVIDIA NGC
. They are part of
NVIDIA AI Enterprise
, which offers enterprise-grade security, stability, and support. Enterprises seeking the fastest time to value can use
NVIDIA NIM
, a set of easy-to-use microservices for accelerated inference on a wide range of AI models, including open-source community and NVIDIA AI Foundation models.
To learn more, check out these resources:
From Zero to Millions: Scaling Large Language Model Inference with TensorRT-LLM
Perplexity Enhances Model Performance for AI-Powered Search Engines with NVIDIA NeMo
Accelerating Large Language Model Inference with NVIDIA in the Cloud
聚焦：Perplexity AI 利用 NVIDIA 推理栈每月服务 4 亿个搜索请求
对 AI 赋能服务的需求持续快速增长，这给 IT 和基础设施团队带来了越来越大的压力。这些团队的任务是配置必要的硬件和软件来满足这一需求，同时平衡成本效益和最佳用户体验。Perplexity AI 的推理团队面临着这一挑战。Perplexity AI 是一款由 AI 驱动的搜索引擎，每月可处理超过 435 million 个查询。每个查询代表多个 AI 推理请求。
为了满足这一需求，Perplexity 推理团队转而使用
NVIDIA H100 Tensor Core GPUs
、
NVIDIA Triton Inference Server
和
NVIDIA TensorRT-LLM
来进行经济高效的
大语言模型（LLM）
部署。本文根据他们的实操经验，详细介绍了一些部署最佳实践和节省的 TCO。
同时为多个 AI 模型提供服务
为了支持其广泛的用户群并满足搜索、总结和问答等方面的各种请求，Perplexity 的推理团队同时为 20 多个 AI 模型提供服务。其中包括热门开源 Llama 3.1 模型的不同变体，如 8B、70B 和 405B。
为了将每个用户请求与适当的模型相匹配，该公司依靠较小的分类器模型来帮助确定用户意图。然后，分类器检测到的用户任务 (例如文本补全) 会被路由到部署在 GPU 节点上的特定模型。每个节点由一个或多个 NVIDIA H100 GPU 组成，并由 NVIDIA Triton Inference Server 实例管理。这些节点遵循严格的服务级别协议 (SLA) 运行，以实现成本效益和用户交互。
图 1. Perplexity AI 使用 NVIDIA HGX H100 4-GPU 和 8-GPU 系统大规模部署 LLM 生产，具有动态可扩展性、模型并行性和负载均衡功能
为了适应庞大的 Perplexity 用户群和全天波动的流量，Pod 托管在 Kubernetes 集群内。它们具有内置的前端调度程序，可根据负载和使用情况将流量路由到适当的 Pod，从而确保始终满足 SLA。
前端调度程序使用的调度算法会影响令牌间延迟，尤其是在改善最差性能百分位方面 (Figure 2)。Perplexity 团队不断寻找新的调度器优化，包括如何更好地考虑不同请求之间的序列长度变化。
图 2、在 16 QPS 的中等请求速率下，在内部 Perplexity 基准测试中，循环、最小请求的令牌间延迟分配，以及两种随机选择负载均衡策略的能力
Triton Inference Server 是 Perplexity 部署架构的关键组件。它跨各种后端提供优化模型、批量处理传入用户请求，并向调度程序提供 GPU 利用率指标。这支持根据推理请求的数量增加或减少部署和 GPU 的数量。
有关如何使用 Kubernetes 部署 NVIDIA Triton 的详细指南，请参阅
使用 Kubernetes 扩展 LLMs with NVIDIA Triton 和 NVIDIA TensorRT-LLM
。
满足严格的服务级别协议
为了为公司的各种用例定义合适的 SLA，Perplexity 的推理团队进行了全面的 A/B 测试，评估不同的配置及其对用户体验的影响。他们的目标是最大限度地提高 GPU 利用率，同时始终满足每个特定用例的目标 SLA。通过在满足目标 SLA 的同时改进 batching，推理服务成本得以优化。
对于较小的模型，例如嵌入实时检索中使用的 10 亿个参数以下的模型，重点在于尽可能降低延迟。这些通常对用户隐藏，并且是更广泛工作流程的一部分。因此，这些查询的配置通常具有较低的批量大小。鉴于这些模型的内存占用空间较小，该团队在 NVIDIA H100 GPU 上同时运行多个模型，以保持高资源利用率。
对于面向用户的模型 (例如 Llama 8B、70B 和 405B)，这些模型对用户体验和部署成本的影响更大，因此该团队会进行更深入的性能分析，并评估关键指标，例如时间到第一个 token、每位用户每秒 token 数以及每百万次查询的成本。
为了在控制成本的同时优化性能，Perplexity 跨多个 GPU 并行部署这些模型。由于严格的服务水平协议 (SLA)，该团队选择将张量并行性增加到 4 个和 8 个 GPU，他们发现这样可以在固定的 GPU 预算内降低对延迟非常敏感的请求的服务成本。数据或 pipeline 并行有助于在延迟敏感度较低的设置中更大限度地提高吞吐量。
Perplexity 团队现在将 TensorRT-LLM 与使用优化的 CUDA 内核构建的专有 LLM 运行时相结合，以更低的成本在严格的 SLA 中成功为基于 Llama 的模型提供服务。
图 3、在四个 NVIDIA Hopper GPU 上使用张量并行机制对 Llama 8B 模型进行分片，可将 Perplexity 每百万个令牌的相对成本降低高达 3 倍，以处理延迟敏感型请求
最终，Perplexity 推理团队决定托管模型，取决于他们能否以较低的成本为这些模型提供服务，同时仍满足严格的服务水平协议 (SLA)，与使用第三方 LLM 提供商 API 相比。例如，该团队估计，与向第三方 LLM API 服务提供商发送相同的请求量相比，通过在云托管的 NVIDIA GPUs 上提供支持其 Related-Questions 功能的模型，他们每年能够节省约 $1 million。“Related-Questions” 功能为 Perplexity 用户提供后续问题建议，以便在搜索查询后进行更深入的研究。
图 4、与使用第三方 LLM API 服务相比，Perplexity 使用其部署架构在云托管的 NVIDIA GPU 上提供 Related-Questions AI 模型，每年可节省约 100 万美元
提供更高水平的性能
Perplexity 的推理团队对其路线图采用全面的全栈方法，不断优化和增强堆栈的每一层 (从应用和用例到为中间件和硬件加速器提供服务的推理)。
在推理服务中间件方面，该团队正在积极与 NVIDIA Triton 工程团队合作，部署
分解服务
，这是一项突破性技术，可将 LLM 工作流的预填充和解码推理阶段分离到单独的 NVIDIA GPU。此技术可显著提高整体系统吞吐量，同时满足 SLA，从而降低每个令牌的成本。此外，鉴于 Perplexity 的特定硬件资源需求，此技术还可让 Perplexity 在每个推理阶段灵活使用不同的 NVIDIA GPU 产品。
Perplexity 团队明白，优化软件堆栈只能在一定程度上推动性能提升。为了提供更高级别的性能，硬件创新至关重要。因此，他们迫切希望评估
NVIDIA Blackwell 平台
。
NVIDIA Blackwell 通过众多技术创新实现了
显著的性能飞跃
，包括支持 FP4 数据格式的第二代 Transformer Engine、支持更大 NVLink 域的第五代 NVLink 和 NVSwitch 等。总体而言，这些创新可将
万亿级参数 LLM
的推理性能提升 30 倍。
开始使用
NVIDIA Triton Inference Server
和
NVIDIA TensorRT
是 GitHub 上的开源项目。它们还可用作 Docker 容器，可从
NVIDIA NGC
中提取。它们是
NVIDIA AI Enterprise
的一部分，可提供企业级安全性、稳定性和支持。寻求快速实现价值的企业可以使用
NVIDIA NIM
，这是一套易于使用的微服务，可加速各种 AI 模型（包括开源社区和 NVIDIA AI Foundation 模型）的推理。
如需了解详情，请查看以下资源：
从 0 扩展到数百万：使用 TensorRT-LLM 扩展大型语言模型推理
Perplexity
借助 NVIDIA NeMo 增强 AI 驱动的搜索引擎的模型性能
借助 NVIDIA 在云端加速大语言模型推理