Module tokenizer

Structs

Analyzer

Analyzer analyzes a text into a list of tokens.

ChineseTokenizer

ChineseTokenizer tokenizes a Chinese text.

EnglishTokenizer

EnglishTokenizer tokenizes an English text.

JIEBA 🔒

Traits

Tokenizer

Tokenizer tokenizes a text into a list of tokens.