Improving Translation Quality with Domain-Specific Fine-Tuning and NVIDIA NIM
Translation plays an essential role in enabling companies to expand across borders, with requirements varying significantly in terms of tone, accuracy, and technical terminology handling. The emergence of
sovereign AI
has highlighted critical challenges in
large language models (LLMs)
, particularly their struggle to capture nuanced cultural and linguistic contexts beyond English-dominant frameworks. As global communication becomes increasingly complex, organizations must carefully evaluate translation solutions that balance technological efficiency with cultural sensitivity and linguistic precision.
In this post, we explore how LLMs can address the following two distinct English to Traditional Chinese translation use cases:
Marketing content for websites
: Translating technical text with precision while maintaining a natural promotional tone.
Online training courses:
Translating slide text and markdown content used in platforms like Jupyter Notebooks, ensuring accurate technical translation and proper markdown formatting such as headings, sections, and hyperlinks.
These use cases require a specialized approach beyond general translation. While
prompt engineering
with instruction-tuned LLMs can handle certain contexts, more refined tasks like these often do not meet expectations. This is where fine-tuning
Low-Rank Adaptation (LoRA)
adapters separately on collected datasets specific to each translation context becomes essential.
Implementing LoRA adapters for domain-specific translation
For this project, we are using
Llama 3.1 8B Instruct
as the pretrained model and implementing two models fine-tuned with LoRA adapters using
NVIDIA NeMo Framework
. These adapters were trained on domain-specific datasets—one for marketing website content and one for online training courses. For easy deployment of LLMs with
simultaneous use of multiple LoRA adapters
on the same pretrained model, we are using
NVIDIA NIM
.
Refer to the
Jupyter Notebook
to guide you through executing LoRA fine-tuning with NeMo.
Optimizing LLM deployment with LoRA and NVIDIA NIM
NVIDIA NIM introduces a new level of performance, reliability, agility, and control for deploying professional LLM services. With prebuilt containers and optimized model engines tailored for different GPU types, you can easily deploy LLMs while boosting service performance. In addition to popular pretrained models including the Meta
Llama 3
family and Mistral AI
Mistral
and
Mixtral
models, you can integrate and fine-tune your own models with NIM, further enhancing its capabilities.
LoRA is a powerful customization technique that enables efficient fine-tuning by adjusting only a subset of the model’s parameters. This significantly reduces required computational resources. LoRA has become popular due to its effectiveness and efficiency. Unlike full-parameter fine-tuning, LoRA adapter weights are smaller and can be stored separately from the pretrained model, providing greater flexibility in deployment.
NVIDIA TensorRT-LLM
has established a mechanism that can simultaneously serve
multiple LoRA adapters
on the same pretrained model. This
multi-adapter mechanism is also supported by NIM
.
The following sections demonstrate the advantages of these features and application in multipurpose translation tasks.
Step-by-step LoRA fine-tuning deployment with NVIDIA LLM NIM
This section describes the three steps involved in LoRA fine-tuning deployment using NVIDIA LLM NIM.
Step 1: Set up the NIM instance and LoRA models
First, launch a computational instance equipped with two NVIDIA L40S GPUs as recommended in the
NIM support matrix
.
Next, upload the two fine-tuned NeMo files to this environment. Detailed examples of LoRA fine-tuning using NeMo Framework are available in the
official documentation
and a
Jupyter Notebook
.
To organize the environment, use the following command to create directories for storing the LoRA adapters:
$ mkdir -p loras/llama-3.1-8b-translate-course
$ mkdir -p loras/llama-3.1-8b-translate-web
$ export LOCAL_PEFT_DIRECTORY=$(pwd)/loras
$ chmod -R 777 $(pwd)/loras
$ tree loras
loras
├── llama-3.1-8b-translate-course
│   └── course.nemo
└── llama-3.1-8b-translate-web
    └── web.nemo

2 directories, 2 files
Step 2: Deploy NIM and LoRA models
Now, you can proceed to deploy the NIM container. Replace <NGC_API_KEY> with your actual NGC API token.
Generate an API key
if needed. Then run the following commands:
$ export NGC_API_KEY=<NGC_API_KEY>
$ export LOCAL_PEFT_DIRECTORY=$(pwd)/loras
$ export NIM_PEFT_SOURCE=/home/nvs/loras
$ export CONTAINER_NAME=nim-llama-3.1-8b-instruct

$ export NIM_CACHE_PATH=$(pwd)/nim-cache
$ mkdir -p "$NIM_CACHE_PATH"
$ chmod -R 777 $NIM_CACHE_PATH

$ echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
$ docker run -it --rm --name=$CONTAINER_NAME \
    --runtime=nvidia \
    --gpus all \
    --shm-size=16GB \
    -e NGC_API_KEY=$NGC_API_KEY \
    -e NIM_PEFT_SOURCE \
    -v $NIM_CACHE_PATH:/opt/nim/.cache \
    -v $LOCAL_PEFT_DIRECTORY:$NIM_PEFT_SOURCE \
    -p 8000:8000 \
    nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2
After executing these steps, NIM will load the model. Once complete, you can check the health status and retrieve the model names for both the pretrained model and LoRA models using the following commands:
# NIM health status
$ curl http://<INSTANCE_URL>:8000/v1/health/ready
# Get model names of the base model and LoRA models
$ curl http://<INSTANCE_URL>:8000/v1/models | jq
{
   "data" : [
      {
         "created" : 1725516389,
         "id" : "meta/llama-3.1-8b-instruct",
         "max_model_len" : 131072,
         "object" : "model",
         "owned_by" : "system",
         "parent" : null,
         "permission" : [
            {
               "allow_create_engine" : false,
               "allow_fine_tuning" : false,
               "allow_logprobs" : true,
               "allow_sampling" : true,
               "allow_search_indices" : false,
               "allow_view" : true,
               "created" : 1725516389,
               "group" : null,
               "id" : "modelperm-2274791587e4456b9ce921621377becb",
               "is_blocking" : false,
               "object" : "model_permission",
               "organization" : "*"
            }
         ],
         "root" : "meta/llama-3.1-8b-instruct"
      },
      {
         "created" : 1725516389,
         "id" : "llama-3.1-8b-translate-course",
         "max_model_len" : null,
         "object" : "model",
         "owned_by" : "system",
         "parent" : null,
         "permission" : [
            {
               "allow_create_engine" : false,
               "allow_fine_tuning" : false,
               "allow_logprobs" : true,
               "allow_sampling" : true,
               "allow_search_indices" : false,
               "allow_view" : true,
               "created" : 1725516389,
               "group" : null,
               "id" : "modelperm-cb8be2bce8db442d8347f259966e2c02",
               "is_blocking" : false,
               "object" : "model_permission",
               "organization" : "*"
            }
         ],
         "root" : "meta/llama-3.1-8b-instruct"
      },
      {
         "created" : 1725516389,
         "id" : "llama-3.1-8b-translate-web",
         "max_model_len" : null,
         "object" : "model",
         "owned_by" : "system",
         "parent" : null,
         "permission" : [
            {
               "allow_create_engine" : false,
               "allow_fine_tuning" : false,
               "allow_logprobs" : true,
               "allow_sampling" : true,
               "allow_search_indices" : false,
               "allow_view" : true,
               "created" : 1725516389,
               "group" : null,
               "id" : "modelperm-8e404c4d9f504e5fae92bf6caf04e93c",
               "is_blocking" : false,
               "object" : "model_permission",
               "organization" : "*"
            }
         ],
         "root" : "meta/llama-3.1-8b-instruct"
      }
   ],
   "object" : "list"
}
The output will display details of the models available for deployment.
Step 3: Evaluate translation quality of fine-tuned LoRA models
With NIM running, you can use it to perform English to Traditional Chinese translation, specifying the appropriate LoRA model name in the request body.
Fine-tuning results and performance metrics
We evaluated the translation quality of two test datasets using the pretrained model and the two fine-tuned LoRA models. The BLEU and COMET scores from these evaluations are shown in Figures 1 and 2.
Figure 1. BLEU scores (higher is better) of different test datasets using the base model and two LoRA fine-tuned models
Figure 2. COMET scores (higher is better) of different test datasets using the base model and two LoRA models
These results demonstrate that fine-tuning specific datasets using the LoRA mechanism significantly enhances translation quality within their respective domains. To achieve the best performance, domain-specific datasets should leverage their corresponding fine-tuned LoRA models.
The LoRA model fine-tuned on the web dataset demonstrated a more noticeable improvement in evaluation scores for web-related translations. Table 1 provides a comparison of translation examples.
Source Text
Pretrained Model Output
LoRA Fine-Tuned Model (on Web Dataset) Output
NVIDIA extends lead on MLPerf inference benchmark for computer vision, conversational AI, and recommender workloads.
NVIDIA 在計算視覺、對話式人工智慧和建議系統工作負載方面，領先於 MLPerf 推理基準。
NVIDIA 在電腦視覺、對話式人工智慧和推薦系統工作負載的 MLPerf 推論基準中擴大領先地位。
Built on NVIDIA Omniverse, NVIDIA RTX Remix enables modders to easily capture game assets, automatically enhance materials with generative AI tools, and quickly create stunning RTX remasters with full ray tracing and DLSS 3.
在 NVIDIA Omniverse 上建立，RTX Remix 允許模組開發者輕鬆捕捉遊戲資源、自動使用生成 AI 工具增強材質，並快速創建具有全實時光線追蹤和 DLSS 3 的 RTX 重製版。
RTX Remix 是以 NVIDIA Omniverse 為基礎，讓模組玩家能輕鬆擷取遊戲素材、使用生成式人工智慧工具自動增強材質，並快速創造出充分採用全光線追蹤技術和 DLSS 3 的令人驚豔的 RTX 重製版。
Note that the prompt template is as follows:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are an AI assistant who translates an English context to traditional Chinese. The context will be given after a prompt “Translate the following input text to traditional Chinese”. Never respond text other than translation results.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nTranslate the following input text to traditional Chinese.\n\n
{English Text}
.\n<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
Table 1. Sentence translation comparison for the web dataset
Explore further with NVIDIA NIM and LoRA fine-tuning
Fine-tuning
LoRA adapters
separately on marketing content from websites and online training course datasets significantly improved translation quality. This demonstrates that domain-specific datasets achieve better results when paired with their own LoRA models, which efficiently adjust the weights of pretrained models to enhance performance. Deploying these fine-tuned models within a single NVIDIA NIM instance provides a GPU-efficient solution for serving multiple specialized tasks simultaneously.
Ready to take it further? Explore how
NVIDIA NIM
microservices can help you deploy and fine-tune LLMs for your specific tasks. With
NVIDIA NeMo
, you can fine-tune popular models such as
Llama 3
,
Mistral
, and
Phi
using
LoRA adapters
, unlocking greater development efficiency and enhancing the performance of your applications.
利用特定领域的微调和 NVIDIA NIM 提高翻译质量
翻译在助力公司实现跨国业务扩展方面发挥着至关重要的作用，在语调、准确性和技术术语处理方面的要求差异很大。
主权 AI
的出现凸显了
大语言模型（LLM）
面临的严峻挑战，尤其是他们难以捕捉英语主导框架之外的细微文化和语言背景。随着全球通信变得日益复杂，组织必须仔细评估翻译解决方案，以平衡技术效率与文化敏感性和语言准确性。
在本文中，我们将探讨 LLMs 如何解决以下两种截然不同的英语到繁体中文翻译用例：
网站营销内容：精确翻译技术文本，同时保持自然的宣传基调。
在
线培训课程
：翻译 Jupyter Notebooks 等平台中使用的幻灯片文本和 markdown 内容，确保准确的技术翻译和正确的 markdown 格式 (如标题、部分和超链接)。
这些用例需要采用常规翻译以外的专门方法。虽然
使用指令调整 LLMs 进行提示工程
可以处理某些情境，但此类更精细的任务通常无法达到预期效果。因此，在针对每个翻译环境收集的特定数据集上单独微调 Low-Rank Adaptation（LoRA）适配器变得至关重要。
实施 LoRA 适配器以实现特定领域的翻译
在此项目中，我们将使用
Llama 3.1 8B Instruct
作为预训练模型，并使用
NVIDIA NeMo Framework
实现两个通过 LoRA adapters 微调的模型。这些 adapters 基于特定领域的数据集进行训练，
一个用于营销网站内容，另一个用于在线培训课程
。为了在同一预训练模型上同时使用多个 LoRA adapters 来轻松部署 LLMs，我们使用
NVIDIA NIM
。
请参阅
Jupyter Notebook
，了解如何使用 NeMo 执行 LoRA 微调。
使用 LoRA 和 NVIDIA NIM 优化 LLM 部署
NVIDIA NIM 为部署专业 LLM 服务引入了更高水平的性能、可靠性、敏捷性和控制力。借助针对不同 GPU 类型定制的预构建容器和优化模型引擎，您可以轻松部署 LLM，同时提高服务性能。除了
Meta Llama 3
系列以及
Mistral AI Mistral
和
Mixtral
模型等热门预训练模型外，您还可以使用 NIM 集成和微调自己的模型，从而进一步增强其功能。
LoRA 是一种强大的定制技术，仅通过调整模型的一部分参数即可实现高效的微调。这显著减少了所需的计算资源。LoRA 因其有效性和效率而广受欢迎。与全参数微调不同，LoRA 适配器权重更小，并且可以与预训练模型分开存储，从而提高部署的灵活性。
NVIDIA TensorRT-LLM
建立了一种机制，可在同一预训练模型上同时为
多个 LoRA 适配器
提供服务。NIM 也支持这种
多适配器机制
。
以下各节将展示这些功能的优势以及在多用途翻译任务中的应用。
借助 NVIDIA LLM NIM 逐步实现 LoRA 微调部署
本节将介绍使用 NVIDIA LLM NIM 进行 LoRA 微调部署所涉及的三个步骤。
第 1 步：设置 NIM 实例和 LoRA 模型
首先，
按照 NIM 支持矩阵中的建议
，启动配备两个 NVIDIA L40S GPUs 的计算实例。
接下来，将两个经过微调的 NeMo 文件上传至此环境。有关使用 NeMo Framework 进行 LoRA 微调的详细示例，请参阅
官方文档
和
Jupyter Notebook
。
要组织环境，请使用以下命令创建用于存储 LoRA 适配器的目录：
$ mkdir -p loras/llama-3.1-8b-translate-course
$ mkdir -p loras/llama-3.1-8b-translate-web
$ export LOCAL_PEFT_DIRECTORY=$(pwd)/loras
$ chmod -R 777 $(pwd)/loras
$ tree loras
loras
├── llama-3.1-8b-translate-course
│   └── course.nemo
└── llama-3.1-8b-translate-web
    └── web.nemo

2 directories, 2 files
第 2 步：部署 NIM 和 LoRA 模型
现在，您可以继续部署 NIM 容器。将 替换为您的实际 NGC API 令牌。如有需要，请生成 API 密钥。然后运行以下命令：
$ export NGC_API_KEY=<NGC_API_KEY>
$ export LOCAL_PEFT_DIRECTORY=$(pwd)/loras
$ export NIM_PEFT_SOURCE=/home/nvs/loras
$ export CONTAINER_NAME=nim-llama-3.1-8b-instruct

$ export NIM_CACHE_PATH=$(pwd)/nim-cache
$ mkdir -p "$NIM_CACHE_PATH"
$ chmod -R 777 $NIM_CACHE_PATH

$ echo "$NGC_API_KEY" | docker login nvcr.io --username '$oauthtoken' --password-stdin
$ docker run -it --rm --name=$CONTAINER_NAME \
    --runtime=nvidia \
    --gpus all \
    --shm-size=16GB \
    -e NGC_API_KEY=$NGC_API_KEY \
    -e NIM_PEFT_SOURCE \
    -v $NIM_CACHE_PATH:/opt/nim/.cache \
    -v $LOCAL_PEFT_DIRECTORY:$NIM_PEFT_SOURCE \
    -p 8000:8000 \
    nvcr.io/nim/meta/llama-3.1-8b-instruct:1.1.2
执行这些步骤后，NIM 将加载模型。完成后，您可以使用以下命令查看运行状况，并检索预训练模型和 LoRA 模型的模型名称：
# NIM health status
$ curl http://<INSTANCE_URL>:8000/v1/health/ready
# Get model names of the base model and LoRA models
$ curl http://<INSTANCE_URL>:8000/v1/models | jq
{
   "data" : [
      {
         "created" : 1725516389,
         "id" : "meta/llama-3.1-8b-instruct",
         "max_model_len" : 131072,
         "object" : "model",
         "owned_by" : "system",
         "parent" : null,
         "permission" : [
            {
               "allow_create_engine" : false,
               "allow_fine_tuning" : false,
               "allow_logprobs" : true,
               "allow_sampling" : true,
               "allow_search_indices" : false,
               "allow_view" : true,
               "created" : 1725516389,
               "group" : null,
               "id" : "modelperm-2274791587e4456b9ce921621377becb",
               "is_blocking" : false,
               "object" : "model_permission",
               "organization" : "*"
            }
         ],
         "root" : "meta/llama-3.1-8b-instruct"
      },
      {
         "created" : 1725516389,
         "id" : "llama-3.1-8b-translate-course",
         "max_model_len" : null,
         "object" : "model",
         "owned_by" : "system",
         "parent" : null,
         "permission" : [
            {
               "allow_create_engine" : false,
               "allow_fine_tuning" : false,
               "allow_logprobs" : true,
               "allow_sampling" : true,
               "allow_search_indices" : false,
               "allow_view" : true,
               "created" : 1725516389,
               "group" : null,
               "id" : "modelperm-cb8be2bce8db442d8347f259966e2c02",
               "is_blocking" : false,
               "object" : "model_permission",
               "organization" : "*"
            }
         ],
         "root" : "meta/llama-3.1-8b-instruct"
      },
      {
         "created" : 1725516389,
         "id" : "llama-3.1-8b-translate-web",
         "max_model_len" : null,
         "object" : "model",
         "owned_by" : "system",
         "parent" : null,
         "permission" : [
            {
               "allow_create_engine" : false,
               "allow_fine_tuning" : false,
               "allow_logprobs" : true,
               "allow_sampling" : true,
               "allow_search_indices" : false,
               "allow_view" : true,
               "created" : 1725516389,
               "group" : null,
               "id" : "modelperm-8e404c4d9f504e5fae92bf6caf04e93c",
               "is_blocking" : false,
               "object" : "model_permission",
               "organization" : "*"
            }
         ],
         "root" : "meta/llama-3.1-8b-instruct"
      }
   ],
   "object" : "list"
}
输出将显示可用于部署的模型的详细信息。
第 3 步：评估微调 LoRA 模型的翻译质量
运行 NIM 后，您可以使用 NIM 执行英语到繁体中文的翻译，并在请求正文中指定适当的 LoRA 模型名称。
微调结果和性能指标
我们使用预训练模型和两个经过微调的 LoRA 模型评估了两个测试数据集的翻译质量。这些评估的 BLEU 和 COMET 分数如图 1 和 2 所示。
图 1、使用基础模型和两个 LoRA 微调模型的不同测试数据集的 BLEU 分数 (越高越好)
图 2、使用基础模型和两个 LoRA 模型的不同测试数据集的 COMET 分数 (越高越好)
这些结果表明，使用 LoRA 机制微调特定数据集可显著提高其各自领域内的翻译质量。为了获得最佳性能，特定领域的数据集应利用其相应的微调 LoRA 模型。
在 Web 数据集上微调的 LoRA 模型显示，与 Web 相关的翻译的评估分数有了更显著的提高。表 1 对比了翻译示例。
源文本
预训练模型输出
LoRA
微调模型 (基于 Web 数据集) 输出
NVIDIA 在计算机视觉、对话式 AI 和推荐系统工作负载的 MLPerf 推理基准测试中占据领先地位。
NVIDIA 在計算視覺、對話式人工智慧和建議系統工作負載方面，領先於 MLPerf 推理基準。
NVIDIA 在電腦視覺、對話式人工智慧和推薦系統工作負載的 MLPerf 推論基準中擴大領先地位。
NVIDIA RTX Remix 基于 NVIDIA Omniverse 构建，使 modders 能够轻松捕获游戏资产，使用生成式 AI 工具自动增强材质，并快速创建令人惊艳的 RTX 重制版游戏，具有全景光线追踪和 DLSS 3。
在 NVIDIA Omniverse 上建立，RTX Remix 允许模块开发者轻松捕捉游戏资源、自动使用生成 AI 工具增强材质，并快速创建具有全实时光线追踪和 DLSS 3 的 RTX 重制版。
RTX Remix 是以 NVIDIA Omniverse 为基础，让模块玩家能轻松擷取游戏素材、使用生成式人工智慧工具自动增强材质，并快速创造出充分采用全光线追踪技术和 DLSS 3 的令人惊豔的 RTX 重製版。
请注意，提示模板如下所示：*|begin_of_text|>*|start_header_id|> 系统*|end_header_id|> \ n \ n
<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are an AI assistant who translates an English context to traditional Chinese. The context will be given after a prompt “Translate the following input text to traditional Chinese”. Never respond text other than translation results.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nTranslate the following input text to traditional Chinese.\n\n
.\n<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
系统将在提示“Translate the following input text to traditional chinese”(将以下
输入文本翻译成繁体中文
) 后提供上下文。切勿回复翻译结果以外的文本。*|eot_id|>*|start_header_id|> 用户*|end_header_id|> \ n \ n 将以下输入文本翻译成繁体中文。\ n \ n
{English Text}
.\ n}|eot_id|>*|start_header_id|> 助手*|end_header_id|> \ n \ n
表 1、Web 数据集的句子翻译比较
通过 NVIDIA NIM 和 LoRA 微调进一步探索
分别在网站和在线培训课程数据集的营销内容上微调
LoRA 适配器
，可显著提高翻译质量。这表明，特定领域的数据集在与自己的 LoRA 模型搭配使用时可获得更好的结果，从而高效调整预训练模型的权重以提高性能。通过在单个 NVIDIA NIM 实例中部署这些经过微调的模型，可提供高效利用 GPU 的解决方案，从而同时为多个专业任务提供服务。
准备好更进一步了吗？探索
NVIDIA NIM 微服务
如何帮助您针对特定任务部署和微调 LLM。借助
NVIDIA NeMo
，您可以使用
LoRA 适配器
微调热门模型（例如
Llama 3
、
Mistral
和
Phi
），从而提高开发效率并增强应用性能。