<p>I understand that automated Chinese word segmentation (breaking up text into constituent words) is a hard problem, as it's sometimes possible to segment the same sentence in different ways. As a result, most software packages that accomplish this task with high accuracy seem to use machine learning (e.g. <a href="https://pypi.org/project/pywordseg/" rel="nofollow noreferrer">https://pypi.org/project/pywordseg/</a> or <a href="https://nlp.stanford.edu/software/segmenter.shtml" rel="nofollow noreferrer">https://nlp.stanford.edu/software/segmenter.shtml</a>).</p>
<p>Naively, it seems like it should be possible to achieve modest accuracy with a much simpler method, especially if you're allowed to use a dictionary (like CEDict). For example, you could first match all four-character words, then three, then two.*</p>
<p>Would this be a sensible procedure, and would it produce reasonably accurate results? If not, how can one improve it without resorting to machine learning or something that takes a ton of manual effort?</p>
<p>(*ambiguities will occur, so you might choose at random. If you had a word frequency list, you could prioritize more commonly used words).</p>

CC BY-SA 4.0
2022-01-27T03:06:02.510Z
Some computer languages are interpreted languages. The interpreter analyses the code, line by line.  What you want is an interpreter for human language. This is what AI is doing. Ever called a number and got an AI on the line? At least in China, keep saying 人工服务，eventually you will get to speak to a person, if you are lucky! If you are not deliberately obtuse, AI can deal with a lot of queries. The computer code to do that will however, be a lot more complex than say, a Python interpreter. Ask google for a glance at their AI code!
Pedroski
CC BY-SA 4.0
2022-01-28T12:11:56.873Z
Hei there.
Here is a batter package https://pypi.org/project/jieba/
He seems to be doing better.
Chinohana
<grammar><sentence-structure>
What is the simplest viable method to segment Chinese text?
# What is the simplest viable method to segment Chinese text?

(Asked by: username_0 on 2022-01-27)

I understand that automated Chinese word segmentation (breaking up text into constituent words) is a hard problem, as it's sometimes possible to segment the same sentence in different ways. As a result, most software packages that accomplish this task with high accuracy seem to use machine learning (e.g. <https://pypi.org/project/pywordseg/> or <https://nlp.stanford.edu/software/segmenter.shtml>).

Naively, it seems like it should be possible to achieve modest accuracy with a much simpler method, especially if you're allowed to use a dictionary (like CEDict). For example, you could first match all four-character words, then three, then two.\*

Would this be a sensible procedure, and would it produce reasonably accurate results? If not, how can one improve it without resorting to machine learning or something that takes a ton of manual effort?

(\*ambiguities will occur, so you might choose at random. If you had a word frequency list, you could prioritize more commonly used words).

**Tags:** grammar, sentence-structure

> **Comments:**
> **username_1**: Some computer languages are interpreted languages. The interpreter analyses the code, line by line.  What you want is an interpreter for human language. This is what AI is doing. Ever called a number and got an AI on the line? At least in China, keep saying 人工服务，eventually you will get to speak to a person, if you are lucky! If you are not deliberately obtuse, AI can deal with a lot of queries. The computer code to do that will however, be a lot more complex than say, a Python interpreter. Ask google for a glance at their AI code!  
> **username_2**: Hei there. Here is a batter package https://pypi.org/project/jieba/ He seems to be doing better.  

---

## 0 answers

