Module tokenizer

Structs

Analyzer

Analyzer analyzes a text into a list of tokens.

ChineseTokenizer

ChineseTokenizer tokenizes a Chinese text.

EnglishTokenizer

EnglishTokenizer tokenizes an English text.

JIEBA 🔒

Constants

VALID_ASCII_TOKEN 🔒

A-Z, a-z, 0-9, and ‘_’ are true

Traits

Tokenizer

Tokenizer tokenizes a text into a list of tokens.