Module tokenizer

Structs

• Analyzer
  Analyzer analyzes a text into a list of tokens.
• ChineseTokenizer
  ChineseTokenizer tokenizes a Chinese text.
• EnglishTokenizer
  EnglishTokenizer tokenizes an English text.

Traits

• Tokenizer
  Tokenizer tokenizes a text into a list of tokens.