JSON Lines Reading with pandas 100x Faster Using NVIDIA cuDF
JSON is a widely adopted format for text-based information working interoperably between systems, most commonly in web applications and large language models (LLMs). While the JSON format is human-readable, it is complex to process with data science and data engineering tools.
JSON data often takes the form of newline-delimited
JSON Lines
(also known as NDJSON) to represent multiple records in a dataset. Reading JSON Lines data into a dataframe is a common first step in data processing.
In this post, we compare the performance and functionality of Python APIs for converting JSON Lines data into a dataframe using the following libraries:
pandas
DuckDB
pyarrow
RAPIDS cuDF pandas Accelerator Mode
We demonstrate good scaling performance and high data processing throughput with the JSON reader in cudf.pandas, especially for data with a complex schema. We also review the versatile set of JSON reader options in cuDF that improve compatibility with Apache Spark and empower Python users to handle quote normalization, invalid records, mixed types and other JSON anomalies.
JSON parsing versus JSON reading
When it comes to JSON data processing, it’s important to distinguish between parsing and reading.
JSON parsers
JSON parsers, such as
simdjson
, convert a buffer of character data into a vector of tokens. These tokens represent the logical components of JSON data, including field names, values, array begin/end, and map begin/end. Parsing is a critical first step in extracting information from JSON data, and significant research has been dedicated to reaching high parsing throughput.
To use information from JSON Lines in data processing pipelines, the tokens must often be converted into a Dataframe or columnar format, such as
Apache Arrow
.
JSON readers
JSON readers, such as
pandas.read_json
convert input character data into a Dataframe organized by columns and rows. The reader process begins with a parsing step and then detects record boundaries, manages the top-level columns and nested struct or list child columns, handles missing and null fields, infers data types, and more.
JSON readers convert unstructured character data into a structured Dataframe, making JSON data compatible with downstream applications.
JSON Lines reader benchmarking
JSON Lines is a flexible format for representing data. Here are some important properties of JSON data:
Number of records per file
Number of top level columns
Depth of struct or list nesting for each column
Data types of values
Distribution of string lengths
Fraction of missing keys
For this study, we held the record count fixed at 200K and swept the column count from 2 to 200, exploring a range of complex schemas. The four data types in use are as follows:
list<int>
and
list<str>
with two child elements
struct<int>
and
struct<str>
with a single child element
Table 1 shows the first two columns of the first two records for data types, including
list<int>
,
list<str>
,
struct<int>
, and
struct<str>
.
Data type
Example records
list<int>
{"c0":[848377,848377],"c1":[164802,164802],...\n{"c0":[732888,732888],"c1":[817331,817331],...
list<str>
{"c0":["FJéBCCBJD","FJéBCCBJD"],"c1":["CHJGGGGBé","CHJGGGGBé"],...\n{"c0":["DFéGHFéFD","DFéGHFéFD"],"c1":["FDFJJCJCD","FDFJJCJCD"],...
struct<int>
{"c0":{"c0":361398},"c1":{"c0":772836},...\n{"c0":{"c0":57414},"c1":{"c0":619350},...
struct<str>
{"c0":{"c0":"FBJGGCFGF"},"c1":{"c0":"ïâFFéâJéJ"},...\n{"c0":{"c0":"éJFHDHGGC"},"c1":{"c0":"FDâBBCCBJ"},...
Table 1. Example JSON Lines character data
Table 1 shows the first two columns of the first two records for data types, including
list<int>
,
list<str>
,
struct<int>
, and
struct<str>
.
Performance statistics were collected on the 25.02 branch of
cuDF
and with the following library versions: pandas 2.2.3, duckdb 1.1.3, and pyarrow 17.0.0. The execution hardware used an NVIDIA H100 Tensor Core 80 GB HBM3 GPU and Intel Xeon Platinum 8480CL CPU with 2TiB of RAM. Timing was collected from the third of three repetitions, to avoid initialization overhead and ensure that the input file data was present in the OS page cache.
In addition to the zero code change cudf.pandas, we also collected performance data from
pylibcudf
, a Python API for the
libcudf
CUDA C++ computation core. The runs with pylibcudf used a CUDA async memory resource through RAPIDS Memory Manager (RMM). Throughput values were computed using the JSONL input file size and the reader runtime of the third repetition.
Here are some examples from several Python libraries for invoking the JSON Lines reader:
# pandas and cudf.pandas
import pandas as pd
df = pd.read_json(file_path, lines=True)

# DuckDB
import duckdb
df = duckdb.read_json(file_path, format='newline_delimited')

# pyarrow
import pyarrow.json as paj
table = paj.read_json(file_path)

# pylibcudf
import pylibcudf as plc
s = plc.io.types.SourceInfo([file_path])
opt = plc.io.json.JsonReaderOptions.builder(s).lines(True).build()
df = plc.io.json.read_json(opt)
JSON Lines reader performance
Overall, we found a wide range of performance characteristics for the JSON readers available in Python, with overall runtimes varying from 1.5 seconds to almost 5 minutes.
Table 2 shows the sum of the timing data from seven JSON reader configurations when processing 28 input files with a total file size of 8.2 GB:
Using cudf.pandas for JSON reading shows about 133x speedup over pandas with the default engine and 60x speedup over pandas with the pyarrow engine.
DuckDB and pyarrow show good performance as well, with about 60 seconds total time for DuckDB, and 6.9 seconds for pyarrow with block size tuning.
The fastest time comes from pylibcudf at 1.5 seconds, showing about 4.6x speedup over pyarrow with
block_size
tuning.
Reader label
Benchmark runtime
(sec)
Comment
cudf.pandas
2.1
Using
-m cudf.pandas
from the command line
pylibcudf
1.5
pandas
281
pandas-pa
130
Using the pyarrow engine
DuckDB
62.9
pyarrow
15.2
pyarrow-20MB
6.9
Using a 20 MB
block_size
value
Table 2. Sum of timing data for JSON reading of 28 input files
Table 2 includes the input columns counts 2, 5, 10, 20, 50, 100, and 200, and the data types
list<int>
,
list<str>
,
struct<int>
, and
struct<str>
.
Zooming into the data by data type and column count, we found that JSON reader performance varies over a wide range based on the input data details and the data processing library, from 40 MB/s to 3 GB/s for CPU-based libraries and from 2–6 GB/s for the GPU-based cuDF.
Figure 1 shows the data processing throughput based on input size for 200K rows and 2–200 columns, with input data sizes varying from about 10 MB to 1.5 GB.
Figure 1. JSON Lines reading throughput by number of input columns
In Figure 1, each subplot corresponds to the data type of the input columns. File size annotations align to the x-axis.
For cudf.pandas
read_json
, we observed 2–5 GB/s throughput that increased with larger column count and input data size. We also found that the column data type does not significantly affect throughput. The pylibcudf library shows about 1–2 GB/s higher throughput than cuDF-python, due to lower Python and pandas semantic overhead.
For pandas
read_json
, we measured about 40–50 MB/s throughput for the default UltraJSON engine (labeled as “pandas-uj”). Using the pyarrow engine (
engine="pyarrow"
) provided a boost up to 70–100 MB/s due to faster parsing (pandas-pa). The pandas JSON reader performance appears to be limited by the need to create Python list and dictionary objects for each element in the table.
For DuckDB
read_json
, we found about 0.5–1 GB/s throughput for
list<str>
and
struct<str>
processing with lower values <0.2 GB/s for
list<int>
and
struct<int>
. Data processing throughput remained steady over the range of column counts.
For pyarrow
read_json
, we measured data processing throughputs up to 2–3 GB/s for 5-20 columns, and lower throughput values as column count increased to 50 and above. We found data type to have a smaller impact on reader performance than column count and input data size. For column counts of 200 and a record size of ~5 KB per row, throughput dropped to about 0.6 GB/s.
Raising the pyarrow
block_size
reader option to 20 MB (pyarrow-20MB) led to increased throughput for column counts 100 or more, while also degrading throughput for column counts 50 or fewer.
Overall, DuckDB primarily showed throughput variability due to data types, whereas cuDF and pyarrow primarily showed throughput variability due to column count and input data size. The GPU-based cudf.pandas and pylibcudf showed the highest data processing throughput for complex list and struct schema, especially for input data sizes >50 MB.
JSON Lines reader options
Given the text-based nature of the JSON format, JSON data often includes anomalies that result in invalid JSON records or don’t map well to a dataframe. Some of these JSON anomalies include single-quoted fields, cropped or corrupted records, and mixed struct or list types. When these patterns occur in your data, they can break the JSON reader step in your pipeline.
Here are some examples of these JSON anomalies:
# 'Single quotes'
# field name "a" uses single quotes instead of double quotes
s = '{"a":0}\n{\'a\':0}\n{"a":0}\n'

# ‘Invalid records'
# the second record is invalid
s = '{"a":0}\n{"a"\n{"a":0}\n'

# 'Mixed types'
# column "a" switches between list and map
s = '{"a":[0]}\n{"a":[0]}\n{"a":{"b":0}}\n'
To unlock advanced JSON reader options in cuDF, we recommend incorporating cuDF-Python (
import cudf
) and pylibcudf into your workflow. If single-quoted field names or string values appear in your data, cuDF provides a reader option to normalize single quotes into double quotes. cuDF supports this feature to provide compatibility with the
allowSingleQuotes
option that is enabled by default in Apache Spark.
If invalid records appear in your data, cuDF and DuckDB both provide error recovery options to replace these records with null. When error handling is enabled, if a record generates a parsing error, all of the columns for the corresponding row are marked as null.
If mixed list and struct values are associated with the same field name in your data, cuDF provides a dtype schema override option to coerce the datatype to string. DuckDB uses a similar approach by inferring a
JSON
data type.
For mixed types, the pandas library has perhaps the most faithful approach, using Python list and dictionary objects to represent the input data.
Here is an example in cuDF-Python and pylibcudf that shows the reader options, including a dtype schema override for column name “a”. For more information, see
cudf.read_json
and
pylibcudf.io.json.read_json
.
For pylibcudf, the
JsonReaderOptions
object can be configured either before or after the
build
function.
# cuDF-python
import cudf
df = cudf.read_json(
    file_path, 
    dtype={"a":str},
    on_bad_lines='recover',
    lines=True,
    normalize_single_quotes=True
)

# pylibcudf 
import pylibcudf as plc
s = plc.io.types.SourceInfo([file_path])
opt = (
    plc.io.json.JsonReaderOptions.builder(s)
    .lines(True)
    .dtypes([("a",plc.types.DataType(plc.types.TypeId.STRING), [])])
    .recovery_mode(plc.io.types.JSONRecoveryMode.RECOVER_WITH_NULL)
    .normalize_single_quotes(True)
    .build()
    )
df = plc.io.json.read_json(opt)
Table 3 summarizes the behavior of several JSON readers with Python APIs for a few common JSON anomalies. Crosses denote that the reader function raised an exception, and checkmarks denote that the library successfully returned a Dataframe. These results may change in future versions of the libraries.
Single quotes
Invalid records
Mixed types
cuDF-Python, pylibcudf
✔️ Normalize to double quotes
✔️ Set to null
✔️ Represent as a string
pandas
❌ Exception
❌ Exception
✔️ Represent as a Python object
pandas (
engine="pyarrow
“
)
❌ Exception
❌ Exception
❌ Exception
DuckDB
❌ Exception
✔️ Set to null
✔️ Represent as a JSON string-like type
pyarrow
❌ Exception
❌ Exception
❌ Exception
Table 3. JSON reader outcomes when reading JSONL files with anomalies including: single quotes, mixed types and invalid records
cuDF supports several additional JSON reader options that are critical for compatibility with Apache Spark conventions, and now are available to Python users as well. Some of these options include:
Validation rules for numbers and strings
Custom record delimiters
Column pruning by the schema provided in dtype
Customization of
NaN
values
For more information, see the libcudf C++ API documentation on
json_reader_options
.
For more information about multi-source reading for efficiently processing many smaller JSON Lines files, or byte-range support for breaking up large JSON Lines files, see
GPU-Accelerated JSON Data Processing with RAPIDS
.
Summary
RAPIDS cuDF provides powerful, flexible, and accelerated tools for working with JSON data in Python.
GPU-accelerated JSON data processing is also available in RAPIDS Accelerator For Apache Spark, starting in the 24.12 release. For information, see
Accelerating JSON Processing on Apache Spark with GPUs
.
For more information, see the following resources:
cuDF documentation
/rapidsai/cudf
GitHub repo
RAPIDS Docker containers
(available for releases and nightly builds)
Accelerate Data Science Workflows with Zero Code Changes
DLI course
Mastering the cudf.pandas Profiler for GPU Acceleration
使用 NVIDIA cuDF，pandas 读取 JSON 行文件速度提升100倍
JSON 是一种广泛采用的格式，用于在系统之间 (通常用于 Web 应用和大语言模型 (LLMs)) 以互操作方式运行的基于文本的信息。虽然 JSON 格式是人类可读的，但使用数据科学和数据工程工具进行处理十分复杂。
JSON
数据通常采用换行分隔的 JSON 行
(也称为 NDJSON) 的形式来表示数据集中的多个记录。将 JSON 行数据读入数据帧是数据处理中常见的第一步。
在本文中，我们比较了使用以下库将 JSON 行数据转换为数据帧的 Python API 的性能和功能：
pandas
DuckDB
pyarrow
RAPIDS cuDF pandas
加速器模式
我们使用 cudf.pandas 中的 JSON 读取器展示了良好的扩展性能和高数据处理吞吐量，特别是对于具有复杂模式的数据。我们还审查了 cuDF 中的一组通用 JSON 读取器选项，这些选项可提高与 Apache Spark 的兼容性，并使 Python 用户能够处理引文归一化、无效记录、混合类型和其他 JSON 异常。
JSON 解析与 JSON 读取
当涉及到 JSON 数据处理时，区分解析和读取非常重要。
JSON 解析器
JSON 解析器 (例如
simdjson
) 可将字符数据缓冲区转换为令牌向量。这些令牌代表 JSON 数据的逻辑组件，包括字段名、值、数组开始/结束和映射开始/结束。解析是从 JSON 数据中提取信息的关键第一步，并且我们致力于实现高解析吞吐量。
要在数据处理工作流中使用来自 JSON 行的信息，必须经常将令牌转换为 Dataframe 或列式格式，例如
Apache Arrow
。
JSON 阅读器
JSON 读取器 (例如
pandas.read_json
) 将输入字符数据转换为按列和行排列的 Dataframe。读取器流程从解析步骤开始，然后检测记录边界、管理顶层列和嵌套结构体或列表子列、处理缺失和空字段、推理数据类型等。
JSON 读取器可将非结构化字符数据转换为结构化 Dataframe，从而使 JSON 数据与下游应用兼容。
JSON Lines 读取器基准测试
JSON Lines 是一种灵活的数据表示格式。以下是 JSON 数据的一些重要属性：
每个文件的记录数
顶层列的数量
每列的结构体或列表嵌套深度
值的数据类型
字符串长度分布
缺少密钥的百分比
在这项研究中，我们将记录计数固定为 200K，并将列计数从 2 扫至 200，同时探索了一系列复杂的模式。使用的四种数据类型如下所示：
包含两个子元素的
list<int>
和
list<str>
包含单个子元素的
struct<int>
和
struct<str>
表 1 显示了前两列数据类型记录的前两列，包括
list<int>
、
list<str>
、
struct<int>
和
struct<str>
。
数据类型
记录示例
list<int>
{"c0":[848377,848377],"c1":[164802,164802],...\n{"c0":[732888,732888],"c1":[817331,817331],...
list<str>
{"c0":["FJéBCCBJD","FJéBCCBJD"],"c1":["CHJGGGGBé","CHJGGGGBé"],...\n{"c0":["DFéGHFéFD","DFéGHFéFD"],"c1":["FDFJJCJCD","FDFJJCJCD"],...
struct<int>
{"c0":{"c0":361398},"c1":{"c0":772836},...\n{"c0":{"c0":57414},"c1":{"c0":619350},...
struct<str>
{"c0":{"c0":"FBJGGCFGF"},"c1":{"c0":"ïâFFéâJéJ"},...\n{"c0":{"c0":"éJFHDHGGC"},"c1":{"c0":"FDâBBCCBJ"},...
表 1. JSON 行字符数据示例
表 1 显示了前两列数据类型记录的前两列，包括
list<int>
、
list<str>
、
struct<int>
和
struct<str>
性能统计数据在
cuDF
的 25.02 分支上收集，并包含以下库版本：pandas 2.2.3、duckdb 1.1.3 和 pyarrow 17.0.0。执行硬件使用 NVIDIA H100 Tensor Core 80 GB HBM3 GPU 和 Intel Xeon Platinum 8480CL CPU 以及 2TiB 的 RAM。计时数据从三次重复的第三次中收集，以避免初始化开销，并确保输入文件数据存在于操作系统页面缓存中。
除了零代码更改 cudf.pandas 之外，我们还从
py
libcudf
(用于
libcudf
CUDA C++计算核心的 Python API) 收集了性能数据。
py
libcudf
运行通过 RAPIDS 内存管理器 (RMM) 使用 CUDA 异步内存资源。使用 JSONL 输入文件大小和第三次重复的读取器运行时计算吞吐量值。
以下是来自多个 Python 库的一些调用 JSON 行读取器的示例：
# pandas and cudf.pandas
import pandas as pd
df = pd.read_json(file_path, lines=True)

# DuckDB
import duckdb
df = duckdb.read_json(file_path, format='newline_delimited')

# pyarrow
import pyarrow.json as paj
table = paj.read_json(file_path)

# pylibcudf
import pylibcudf as plc
s = plc.io.types.SourceInfo([file_path])
opt = plc.io.json.JsonReaderOptions.builder(s).lines(True).build()
df = plc.io.json.read_json(opt)
JSON 行读取器性能
总体而言，我们发现 Python 中的 JSON 读取器具有各种性能特征，总体运行时间从 1.5 秒到近 5 分钟不等。
表 2 显示了在处理 28 个输入文件 (总文件大小为 8.2 GB) 时，来自 7 个 JSON 读取器配置的定时数据的总和：
使用 cudf.pandas 进行 JSON 读取显示，与使用默认引擎的 pandas 相比，速度提高了 133 倍，使用 pyarrow 引擎的 pandas 速度提高了 60 倍。
DuckDB 和 pyarrow 也表现出良好的性能，在调整块大小时，DuckDB 的总时间约为 60 秒，而 pyarrow 的总时间为 6.9 秒。
pylibcudf 生成的最快时间为 1.5 秒，与 pyarrow 相比，使用
block_size
调优的速度提高了约 4.6 倍。
阅读器标签
基准运行时
(秒)
评论
cudf.pandas
2.1
在命令行中使用
-m cudf.pandas
pylibcudf
1.5
pandas
271
pandas-pa
130
使用 pyarrow 引擎
DuckDB
62.9
pyarrow
15.2
pyarrow-20MB
6.9
使用 20 MB 的
block_size
值
表 2、用于 JSON 读取 28 个输入文件的定时数据总和
表 2 包括输入列计数 2、5、10、20、50、100 和 200，以及数据类型
list<int>
、
list<str>
、
struct<int>
和
struct<str>
通过按数据类型和列数量放大数据，我们发现 JSON 读取器的性能因输入数据详细信息和数据处理库的不同而差异很大，基于 CPU 的库的性能介于 40 MB/s 到 3 GB/s 之间，而基于 GPU 的 cuDF 的性能介于 2–6 GB/s 之间。
图 1 显示了基于 200K 行、2–200 列输入大小的数据处理吞吐量，输入数据大小在约 10 MB 到 1.5 GB 之间变化。
图 1. JSON Lines 读取吞吐量按输入列数量
在图 1 中，每个子图均对应输入列的数据类型。文件大小标注与 x 轴对齐。
对于
cudf.pandas read_json
，我们观察到，随着列数量和输入数据大小的增加，吞吐量增加了 2–5 GB/秒。我们还发现，列数据类型不会对吞吐量产生重大影响。由于 Python 和 pandas 语义用度较低，pylibcudf 库的吞吐量比 cuDF-python 高约 1–2 GB/秒。
对于
pandas read_json
，我们测量了默认 UltraJSON 引擎 (标记为“pandas-uj”) 的吞吐量约为 40–50 MB/s。由于解析速度更快 (pandas-pa)，使用 pyarrow 引擎 (
engine="pyarrow"
) 可将速度提升高达 70–100 MB/s。由于需要为表中的每个元素创建 Python 列表和字典对象，因此 pandas JSON 读取器的性能似乎受到限制。
对于 DuckDB
read_json
，我们发现
list<str>
和
struct<str>
处理的吞吐量约为 0.5–1 GB/s，而
list<int>
和
struct<int>
的较低值 < 0.2 GB/s。数据处理吞吐量在列数量范围内保持稳定。
对于 pyarrow
read_json
，我们测量了 5-20 列的高达 2–3 GB/s 的数据处理吞吐量，以及随着列数量增加到 50 及以上而降低的吞吐量值。我们发现，与列数量和输入数据大小相比，数据类型对读取器性能的影响较小。如果列数量为 200，且每行的记录大小约为 5 KB，吞吐量将下降到约 0.6 GB/s。
将 pyarrow
block_size
reader 选项提升至 20 MB (pyarrow-20MB) 会导致列数量增加 100 或以上的吞吐量增加，同时还会降低 50 或以下列数量的吞吐量。
总体而言，DuckDB 主要因数据类型而显示吞吐量可变性，而 cuDF 和 pyarrow 主要因列数量和输入数据大小而显示吞吐量可变性。基于 GPU 的 cudf.pandas 和 pylibcudf 为复杂列表和结构模式（尤其是输入数据大小 > 50 MB）提供了超高的数据处理吞吐量。
JSON 行读取器选项
鉴于 JSON 格式基于文本的特性，JSON 数据通常包含异常，导致 JSON 记录无效或无法很好地映射到数据帧。其中一些 JSON 异常包括单引号字段、已裁剪或损坏的记录，以及混合结构或列表类型。当数据中出现这些模式时，它们可能会中断工作流中的 JSON 读取器步骤。
以下是这些 JSON 异常的一些示例：
# 'Single quotes'
# field name "a" uses single quotes instead of double quotes
s = '{"a":0}\n{\'a\':0}\n{"a":0}\n'

# ‘Invalid records'
# the second record is invalid
s = '{"a":0}\n{"a"\n{"a":0}\n'

# 'Mixed types'
# column "a" switches between list and map
s = '{"a":[0]}\n{"a":[0]}\n{"a":{"b":0}}\n'
要在 cuDF 中解锁高级 JSON 读取器选项，我们建议您将 cuDF-Python (
import cudf
) 和 pylibcudf 集成到您的工作流中。如果数据中出现单引号字段名称或字符串值，cuDF 会提供读取器选项，用于将单引号归一化为双引号。cuDF 支持此功能，可与 Apache Spark 中默认启用的
allowSingleQuotes
选项兼容。
如果您的数据中出现无效记录，cuDF 和 DuckDB 都会提供错误恢复选项，将这些记录替换为 null。启用错误处理后，如果记录生成解析错误，则相应行的所有列均标记为 null。
如果混合 list 和 struct 值与数据中的相同字段名相关联，cuDF 提供一个 dtype 模式覆盖选项，以将数据类型强制转换为字符串。DuckDB 使用类似的方法来推理
JSON
数据类型。
对于混合类型，pandas 库可能是最可靠的方法，使用 Python 列表和字典对象来表示输入数据。
以下是 cuDF-Python 和 pylibcudf 中的示例，其中显示了读取器选项，包括列名称“a”的 dtype 模式覆盖。如需了解更多信息，请参阅
cudf.read_json
和
pylibcudf.io.json.read_json
。
对于 pylibcudf，可以在
build
函数之前或之后配置
JsonReaderOptions
对象。
# cuDF-python
import cudf
df = cudf.read_json(
    file_path, 
    dtype={"a":str},
    on_bad_lines='recover',
    lines=True,
    normalize_single_quotes=True
)

# pylibcudf 
import pylibcudf as plc
s = plc.io.types.SourceInfo([file_path])
opt = (
    plc.io.json.JsonReaderOptions.builder(s)
    .lines(True)
    .dtypes([("a",plc.types.DataType(plc.types.TypeId.STRING), [])])
    .recovery_mode(plc.io.types.JSONRecoveryMode.RECOVER_WITH_NULL)
    .normalize_single_quotes(True)
    .build()
    )
df = plc.io.json.read_json(opt)
表 3 总结了使用 Python API 的多个 JSON 读取器针对一些常见 JSON 异常的行为。交叉表示读取器函数引发异常，勾号表示库已成功返回 Dataframe。在未来版本的库中，这些结果可能会发生变化。
单引号
无效记录
混合类型
cuDF-Python、pylibcudf
归一化为双引号
设置为 null
表示为字符串
pandas
*例外
*例外
表示为 Python 对象
pandas (
engine="pyarrow
“
)
*例外
*例外
*例外
DuckDB
*例外
设置为 null
表示为类似 JSON 字符串的类型
pyarrow
*例外
*例外
*例外
表 3、读取异常情况 (包括单引号、混合类型和无效记录) 的 JSONL 文件时的 JSON 读取器结果
cuDF 支持多个额外的 JSON 读取器选项，这些选项对于与 Apache Spark 惯例的兼容性至关重要，现在也可供 Python 用户使用。其中一些选项包括：
数字和字符串的验证规则
自定义记录分隔符
根据 dtype 中提供的模式进行列剪枝
自定义
NaN
值
有关更多信息，请参阅有关
json_reader_options
的 libcudf C++ API 文档。
有关多源读取以高效处理许多较小的 JSON 行文件的更多信息，或有关分解大型 JSON 行文件的字节范围支持的更多信息，请参阅使用
RAPIDS 进行 GPU 加速的 JSON 数据处理
。
总结
RAPIDS cuDF 为在 Python 中处理 JSON 数据提供了功能强大、灵活且加速的工具。
从 24.12 版本开始，您还可以在适用于 Apache Spark 的 RAPIDS Accelerator 中使用 GPU 加速的 JSON 数据处理功能。有关信息，请参阅
使用 GPU 在 Apache Spark 上加速 JSON 处理
。
有关更多信息，请参阅以下资源：
cuDF 文档
/rapidsai/cudf
GitHub 存储库
RAPIDS Docker 容器 (可用于版本和夜间构建)
零代码更改加速数据科学工作流程
DLI 课程
掌握用于 GPU 加速的 cudf.pandas Profiler