chinese words segmentation algorithm

cws.md

Chinese Word Segmentation

Ref:

• https://arxiv.org/ftp/arxiv/papers/1901/1901.06079.pdf

Tagging

• B: begin
• M: middle
• E: end
• S: single

Machine Learning:

• Maximum Entropy Markov
• Conditional Random Field (semi-CRF and linear CRF)

Deep Learning:

• Collobert with char embedding
• Max-Margin Tensor Neural Network(MMTNN)
• Gated recursive neural network(GRNN)
• Long Short Term Memory Neural Networks(LSTM)
• GRNN & LSTM

MMSEG

Ref:

• http://technology.chtsai.org/mmseg/
• http://yongsun.me/2013/06/simple-implementation-of-mmseg-with-python/

maximum matching algorithm

Simple: find the longest match

complex: find 3-word chunks, choose the 1st chunk with the maximum length

ambiguity resolution rules

1. maximum matching (59.5%)
2. largest average word length (30.6%)
3. smallest variance of word lengths (1%)
4. largest sum of degree of morphemic freedom of one-character words (9%)

HMM

Ref:

• https://towardsdatascience.com/introduction-to-hidden-markov-models-cd2c93e6b781

CRF

Ref: *