Using NetworkX, Jaccard Similarity, and cuGraph to Predict Your Next Favorite Movie
As the amount of data available to everyone in the world increases, the ability for a consumer to make informed decisions becomes increasingly difficult. Fortunately, large datasets are a beneficial component for recommendation systems, which can make a sometimes overwhelming decision much easier.
Graphs are excellent choices for modeling the relationships inherent in the data that fuel recommendation systems, and NetworkX is a very popular option that many data scientists turn to for graph analytics in Python. NetworkX is easy to learn and use, stocked with a wide breadth of graph algorithms, backed by a large and friendly community, and has copious examples available in notebooks, documents, Stack Overflow, and your favorite LLM.
However, to the disappointment of countless developers that broke into graph analytics with or even because of NetworkX, it famously falls short in performance at the scales used by typical recommendation systems.
This begs the question: Can an effective graph-based recommendation system be written in a few simple lines of Python? More generally, can developers and data scientists have both easy-to-use and high-performance graph analytics?
The answer to both questions is, “Yes.”
Read on to discover how you can create a simple and effective recommendation system in Python using NetworkX, a dataset of 33M movie reviews, the Jaccard Similarity algorithm, and the NVIDIA cuGraph back-end, which provides the >250x speedup necessary for modern large-scale graph data.
The MovieLens dataset
Here’s the most important part of the system: the data. The
MovieLens dataset
is generously made available for public download and is described in more detail in the
README file
. The full set includes about 331K anonymized users reviewing 87K movies, resulting in 34M ratings.
Figure 1. MovieLens data represented as a graph, where the individual ratings easily map to edges between user and movie nodes
Extracting recommendations from the data: bipartite graphs and Jaccard Similarity
The type of graph created from the MovieLens data is a
bipartite graph
because there are only two types of nodes: movies and users. The reviews (edges) can only occur between a user and a movie. This makes it particularly easy to apply the
Jaccard Similarity
algorithm to find similarities between movies.
Jaccard Similarity compares pairs of nodes and computes a similarity coefficient using their relationships in the graph. In this case, movies are related to each other based on how users have chosen to watch and review them.
Figure 2. Jaccard Similarity computes a similarity coefficient using the sizes of the sets of neighbors for the two nodes being compared
Based on the viewing preferences of users, you can see m3 is more similar to m2 than it is to m1, and movies m4 and m1 aren’t similar at all. This system would recommend m2 to someone who likes m3 and wouldn’t recommend m1 to someone who likes m4.
NetworkX makes it easy… for smaller graphs
Not surprisingly, NetworkX supports the type of analysis described earlier, and it’s quite easy to start seeing results in just a few lines of Python. But as you’ll see, performance becomes a limitation for larger-sized graphs—such as those needed for your movie recommendation system—when using NetworkX without the GPU-accelerated cuGraph backend.
I discuss the key pieces of the recommendation system later in this post, but the full source code is available in the
/rapidsai/nx-cugraph
GitHub repo.
Because the Jaccard Similarity algorithm you’re using doesn’t take edge weights into account, it considers all reviews equal. You don’t want movies with low reviews to be recommended, so filter out all reviews under a certain threshold, which has the side effect of making the graph smaller too.
# Create a separate DataFrame containing only "good" reviews (rating >= 3).
good_ratings_df = ratings_df[ratings_df["rating"] >= 3]
good_user_ids = good_ratings_df["userId"].unique()
good_movie_ids = good_ratings_df["movieId"].unique()
If you print the sizes of the data you’re working with, you see that your graph of good reviews is approximately 330K nodes and 28M edges, with an average degree (number of neighbors per node) of 84:
total number of users: 330975
total number of reviews: 33832162
average number of total reviews/user: 102.22
total number of users with good ratings: 329127
total number of good reviews: 27782577
average number of good reviews/user: 84.41
As mentioned earlier, graphs of this size often present a challenge to NetworkX, but GPU acceleration using the cuGraph backend removes the performance limitations often associated with this much data. However, I’ll continue with a CPU-only environment to demonstrate the default performance.
All the following examples were run on a workstation using NetworkX 3.4.2 and a Intel Xeon Platinum 8480CL at 2.0 GHz with 2 TB RAM.
Using a NetworkX graph created from users and good movie reviews, pick a user, find one of their highest rated movies, and use Jaccard Similarity to find other movies like it:
# Pick a user and one of their highly-rated movies
user = good_user_ids[321]
user_reviews = good_user_movie_G[user]
highest_rated_movie = max(
	user_reviews,
	key=lambda n: user_reviews[n].get("rating", 0)
)
When you look up the node ID in the movie name map, you see that one of this user’s highest rated movies is the animated film,
Mulan
:
highest rated movie for user=289308 is Mulan (1998), id: 1907, rated: {'rating': 5.0}
You can now use Jaccard Similarity to recommend a movie based on the user’s preferences and viewing history:
%%time
# Run Jaccard Similarity
jacc_coeffs = list(nx.jaccard_coefficient(good_user_movie_G, ebunch))
CPU times: user 2min 5s, sys: 15.4 ms, total: 2min 5s
Wall time: 2min 14s
The Jaccard Similarity computation using the default NetworkX implementation ran for over two minutes. Using these results, you can now provide a recommendation.
# Sort by coefficient value, which is the 3rd item in the tuples
jacc_coeffs.sort(key=lambda t: t[2], reverse=True)
 
# Create a list of recommendations ordered by "best" to "worst" based on the
# Jaccard Similarity coefficients and the movies already seen
movies_seen = list(good_user_movie_G.neighbors(user))
recommendations = [mid for (_, mid, _) in jacc_coeffs
               	  if mid not in movies_seen]
Now you can print the first movie in the sorted list of recommendations:
User ID 289308 might like Tarzan (1999) (movie ID: 2687)
The code is easy and the results look good, but performance holds us back
As you can see, the recommendation seems reasonable; someone who likes
Mulan
seems likely to also enjoy the 1999 Disney animated film
Tarzan
.
However, if the goal was to provide a service or to analyze hundreds or thousands of movies, the two-minute runtime would have you start looking for an alternative to NetworkX. You can see that finding similarities between other movies using this system isn’t any faster:
%%time
# 1196: "Star Wars: Episode V - The Empire Strikes Back (1980)"
print_similar_movies(1196)
movies similar to Star Wars: Episode V - The Empire Strikes Back (1980):
movieId=260, Star Wars: Episode IV - A New Hope (1977)
movieId=1210, Star Wars: Episode VI - Return of the Jedi (1983)
movieId=1198, Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)
CPU times: user 13min 47s, sys: 71.8 ms, total: 13min 47s
Wall time: 11min 30s
Here’s another example:
%%time
# 318: "Shawshank Redemption, The (1994)"
print_similar_movies(318)
movies similar to "Shawshank Redemption, The (1994)":
movieId=296, Pulp Fiction (1994)
movieId=593, "Silence of the Lambs, The (1991)"
movieId=356, Forrest Gump (1994)
CPU times: user 28min 28s, sys: 172 ms, total: 28min 28s
Wall time: 16min 49s
The quality of the recommendations returned is impressive given that this system is composed of only a few lines of code. However, the runtime performance makes it virtually unusable. ‌As described earlier, finding recommendations based on
Shawshank Redemption, The (1994)
takes nearly 17 minutes.
NVIDIA cuGraph makes it transformatively faster
The graph algorithm in this workflow is prohibitively expensive, but by using the NVIDIA cuGraph backend and a compatible GPU, you can dramatically improve performance without changing the code.
Jaccard Similarity is supported in nx-cugraph version 25.02 or later. Version 25.02 is available from nightly builds and will be part of future stable releases later this month. Instructions on installing nx-cugraph, as well as other RAPIDS packages, from both nightly and stable channels using conda or pip, are available in the
RAPIDS Installation Guide
.
After being installed, enable nx-cugraph by setting an environment variable:
NX_CUGRAPH_AUTOCONFIG=True
cuGraph uses the GPU to dramatically accelerate the neighbor lookups and set comparisons needed for the Jaccard Similarity computation. As the graph scales and the number of movies and reviews per movie increases, performance remains almost constant.
The best part of the system—the simplicity of the code—does not change, and the results are identical, but performance increases by over 250x for the run that previously took nearly 17 minutes, reducing it to under 4 seconds.
Figure 3. Speedup of cuGraph over NetworkX for Jaccard Similarity computation for various movies
Software: NetworkX 3.4.2, cuGraph/nx-cugraph 25.02
;
CPU: Intel(R) Xeon(R) Platinum 8480CL @ 2.0GHz 2TB RAM
;
GPU: NVIDIA Quadro RTX 8000 48GB RAM
Conclusion
This post covered a simple and effective recommendation system that’s easy to write in Python using NetworkX. Although there are many other approaches you could take—as covered in
What Is a Recommendation System?
—few would match the low effort required to start exploring data that graph analysis with NetworkX offers.
However, productive and meaningful data exploration requires quick turnaround, and NetworkX has traditionally struggled to scale to larger, real-world problem sizes.
The NVIDIA cuGraph backend for NetworkX accelerates the familiar and flexible NetworkX API to also make it performant at scale, generating results in seconds instead of tens of minutes, keeping you focused and productive. You can now continue using NetworkX, the most popular graph analytics library, without concern for scaling issues simply by adding a GPU and the cuGraph backend to your environment.
For more information about accelerated graph analysis using NetworkX and NVIDIA cuGraph, see
RAPIDS: GPU-Accelerated NetworkX Backend
.
使用 NetworkX、Jaccard Similarity 和 cuGraph 预测您下一部最喜欢的电影
随着全球每个人可用的数据量不断增加，消费者做出明智决策的能力也越来越难。幸运的是，对于推荐系统而言，大型数据集是一个非常有用的组件，有时这会让某些情况下的决策变得更加容易。
对于为推荐系统提供支持的数据中固有的关系建模而言，图形是一个很好的选择，而 NetworkX 是许多数据科学家在 Python 中进行图形分析时非常喜欢的选择。NetworkX 易于学习和使用，拥有各种图形算法，并由庞大而友好的社区提供支持，并且在 Notebook、文档、Stack Overflow 和您喜欢的 LLM 中提供了大量示例。然而，令无数开发人员失望的是，他们使用 NetworkX 或甚至因为 NetworkX 而涉足图形分析，但众所周知，它在典型推荐系统使用的规模上的性能表现不佳。
这就引出了一个问题：能否用 Python 的几行简单代码编写有效的基于图形的推荐系统？更一般地说，开发者和数据科学家能否同时进行易于使用的高性能图形分析？
这两个问题的答案都是“Yes”
请继续阅读，了解如何使用 NetworkX、Jaccard Similarity 算法和 NVIDIA cuGraph 后端 (可将现代大规模图形数据所需的速度提高 250 倍以上)，在 Python 中创建简单有效的推荐系统，使用 3300 万条电影评论的数据集。
MovieLens 数据集
我们先从系统中最重要的部分开始：数据。MovieLens 数据集
1
可供公开
下载
，
README 文件
中有更详细的说明。该系列包括大约 331k 匿名用户，他们在观看 87k 部电影，获得了 34M 的评分。
图 1、MovieLens 数据可以表示为一个图形，其中各个评分可轻松映射到用户和电影节点之间的边缘。
从数据中提取建议：二分图和 Jaccard Similarity
我们根据 MovieLens 数据创建的图形类型是二部图，因为只有两种类型的节点：电影节点和用户节点，并且评论（边缘）只能在用户和电影之间发生。这使得应用
Jaccard Similarity
算法来查找电影之间的相似性变得特别容易。Jaccard Similarity 比较节点对，并使用它们在图形中的关系计算相似性系数。在这种情况下，电影根据用户选择的观看和审查方式相互关联。
图 3、Jaccard Similarity 使用所比较的两个节点的近邻集的大小来计算相似性系数。根据用户的观看偏好，我们可以看到 m3 与 m2 更相似，电影 m4 和 m1 完全不相似。此系统会向喜欢 m3 的用户推荐 m2，而不会向喜欢 m4 的用户推荐 m1。
NetworkX 可以轻松处理较小的图形
不足为奇的是，NetworkX 支持我们上述的分析类型，而且只需使用几行 Python 代码即可轻松查看结果。但正如我们将看到的，当使用无 GPU 加速的 cuGraph 后端的 NetworkX 时，大型图形 (例如我们的电影推荐系统所需的图形) 的性能会受到限制。
我们将在下方查看推荐系统的关键部分，但可在
此处
获取完整源代码。
由于我们使用的 Jaccard Similarity 算法未考虑边缘权重，因此会将所有评论视为相同。我们不希望推荐好评度低的电影，因此我们会过滤掉某个值下的所有好评度，这会导致图形也变小。
# Create a separate DataFrame containing only "good" reviews (rating >= 3).
good_ratings_df = ratings_df[ratings_df["rating"] >= 3]
good_user_ids = good_ratings_df["userId"].unique()
good_movie_ids = good_ratings_df["movieId"].unique()
如果我们打印正在处理的数据的大小，我们会看到好评图大约有 330k 个节点，28M 个边缘，平均度数 (每个节点的近邻数) 为 84：
total number of users: 330975
total number of reviews: 33832162
average number of total reviews/user: 102.22
total number of users with good ratings: 329127
total number of good reviews: 27782577
average number of good reviews/user: 84.41
如上所述，这种规模的图形通常会给 NetworkX 带来挑战，但使用 cuGraph 后端的 GPU 加速消除了通常与如此多的数据相关的性能限制。不过，我们将继续使用 CPU 环境来演示默认性能。
注意
以下所有示例均在使用 NetworkX 3.4.2 和 Intel(R) Xeon(R) Platinum 8480CL@2.0GHz（2TB RAM）的工作站上运行
使用由用户创建的 NetworkX 图形和优秀的电影评论，我们来选择一个用户，找到他们评分最高的电影之一，并使用 Jaccard Similarity 找到类似的其他电影。
# Pick a user and one of their highly-rated movies
user = good_user_ids[321]
user_reviews = good_user_movie_G[user]
highest_rated_movie = max(
	user_reviews,
	key=lambda n: user_reviews[n].get("rating", 0)
)
当我们在电影名称贴图中查找节点 ID 时，我们会发现该用户评分最高的电影之一是动画电影“Mulan”：
highest rated movie for user=289308 is Mulan (1998), id: 1907, rated: {'rating': 5.0}
我们现在可以使用 Jaccard Similarity 根据用户的偏好和观看历史记录来推荐电影：
%%time
# Run Jaccard Similarity
jacc_coeffs = list(nx.jaccard_coefficient(good_user_movie_G, ebunch))
CPU times: user 2min 5s, sys: 15.4 ms, total: 2min 5s
Wall time: 2min 14s
使用默认 NetworkX 实现的 Jaccard 相似性计算运行了两分钟以上。根据这些结果，我们现在可以提供推荐。
# Sort by coefficient value, which is the 3rd item in the tuples
jacc_coeffs.sort(key=lambda t: t[2], reverse=True)
 
# Create a list of recommendations ordered by "best" to "worst" based on the
# Jaccard Similarity coefficients and the movies already seen
movies_seen = list(good_user_movie_G.neighbors(user))
recommendations = [mid for (_, mid, _) in jacc_coeffs
               	  if mid not in movies_seen]
现在，我们只需在已排序的推荐列表中打印出第一部电影：
User ID 289308 might like Tarzan (1999) (movie ID: 2687)
代码很简单，结果看起来不错，但性能却拖累了我们
如我们所见，这个推荐似乎是合理的；喜欢“Mulan”的人似乎也喜欢 1999 年的迪士尼动画电影“Tarzan”。
但是，如果我们的目标是提供服务，或分析数百乃至数千部电影，那么两分钟的运行时间就能让我们开始寻找 NetworkX 的替代方案。我们可以看到，使用此系统查找其他电影相似点的速度并没有加快：
%%time
# 1196: "Star Wars: Episode V - The Empire Strikes Back (1980)"
print_similar_movies(1196)
movies similar to Star Wars: Episode V - The Empire Strikes Back (1980):
movieId=260, Star Wars: Episode IV - A New Hope (1977)
movieId=1210, Star Wars: Episode VI - Return of the Jedi (1983)
movieId=1198, Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)
CPU times: user 13min 47s, sys: 71.8 ms, total: 13min 47s
Wall time: 11min 30s
%%time
# 318: "Shawshank Redemption, The (1994)"
print_similar_movies(318)
movies similar to "Shawshank Redemption, The (1994)":
movieId=296, Pulp Fiction (1994)
movieId=593, "Silence of the Lambs, The (1991)"
movieId=356, Forrest Gump (1994)
CPU times: user 28min 28s, sys: 172 ms, total: 28min 28s
Wall time: 16min 49s
鉴于此系统仅由几行代码组成，因此所返回的推荐内容的质量令人印象深刻。但是，运行时性能使其几乎无法使用。如上所述，根据“Shawshank Redemption, The (1994)”查找推荐内容大约需要 17 分钟。
NVIDIA cuGraph 使其变革性地加快
上述工作流程中的图形算法成本高昂，但通过使用 NVIDIA cuGraph 后端和兼容的 GPU，我们可以在不更改代码的情况下显著提高性能。
nx-cugraph 版本 25.02 或更高版本支持 Jaccard Similarity。版本 25.02 可在 nightly builds 中使用，并将于本月晚些时候纳入未来的稳定版本中。有关如何使用 conda 或 pip 从 nightly 和 stable 通道安装 nx-cugraph 以及其他 RAPIDS 包的说明，请参阅
RAPIDS Installation Guide
。
安装后，只需设置环境变量即可启用 nx-cugraph：
NX_CUGRAPH_AUTOCONFIG=True
cuGraph 利用 GPU 显著加速近邻查找，并设置 Jaccard 相似性计算所需的比较结果。此外，随着图形规模以及每部电影的电影和评论数量的增加，性能几乎保持不变。
该系统最优秀的部分，即代码的简单性，并没有改变，结果也是一样的，但在过去近 17 分钟的运行中，性能提高了 250 倍以上，缩短到 4 秒以内。
图 4、图表显示了各种电影的 cuGraph over NetworkX for Jaccard Similarity 计算速度
。
软件：NetworkX 3.4.2，cuGraph/nx-cugraph 25.02
CPU：Intel(R) Xeon(R) Platinum 8480CL@2.0GHz 2TB RAM
GPU：NVIDIA Quadro RTX 8000 48GB RAM
结束语
这篇博文介绍了一个简单而有效的推荐系统，它可以使用 NetworkX 轻松地用 Python 编写。虽然我们可以采用许多其他方法(如此处所述)，但很少有方法能够做到与开始探索 NetworkX 图形分析提供的数据所需的工作量不相上下。然而，高效和有意义的数据探索需要快速的周转，而 NetworkX 传统上一直难以扩展到更大的实际问题规模。
适用于 NetworkX 的 NVIDIA cuGraph 后端可对熟悉且灵活的 NetworkX API 进行加速，还可大规模提升其性能，在几秒钟 (而非数十分钟) 内生成结果，从而让您专注工作并高效工作。现在，用户只需向环境中添加 GPU 和 cuGraph 后端，即可继续使用热门的图形分析库 NetworkX，而无需担心扩展问题。
如需了解有关使用 NetworkX 和 NVIDIA cuGraph 进行加速图形分析的更多信息，请访问
https://rapids.ai/nx-cugraph
。