Skip to main content
Module tokenizer
index
1.4.0-alpha.0
Module tokenizer
Module Items
Structs
Traits
In index::
fulltext_
index
index
::
fulltext_index
Module
tokenizer
Copy item path
Source
Structs
§
Analyzer
Analyzer
analyzes a text into a list of tokens.
Script
Tokenizer
ScriptTokenizer
produces the tokens described in [
term_token::tokenize
].
Traits
§
Tokenizer
Tokenizer
tokenizes a text into a list of tokens.