  The rapid development of Large Language Models (LLMs) in vertical domains,
including intellectual property (IP), lacks a specific evaluation benchmark for
assessing their understanding, application, and reasoning abilities. To fill
this gap, we introduce IPEval, the first evaluation benchmark tailored for IP
agency and consulting tasks. IPEval comprises 2657 multiple-choice questions
across four major dimensions: creation, application, protection, and management
of IP. These questions span patent rights (inventions, utility models,
designs), trademarks, copyrights, trade secrets, and other related laws.
Evaluation methods include zero-shot, 5-few-shot, and Chain of Thought (CoT)
for seven LLM types, predominantly in English or Chinese. Results show superior
English performance by models like GPT series and Qwen series, while
Chinese-centric LLMs excel in Chinese tests, albeit specialized IP LLMs lag
behind general-purpose ones. Regional and temporal aspects of IP underscore the
need for LLMs to grasp legal nuances and evolving laws. IPEval aims to
accurately gauge LLM capabilities in IP and spur development of specialized
models. Website: \url{https://ipeval.github.io/}
