Skip to main content

Tokenizer

index::fulltext_index::tokenizer

Trait Tokenizer 

Source 
pub trait Tokenizer: Send {
    // Required method
    fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>;
}
Expand description

Tokenizer tokenizes a text into a list of tokens.

Required Methods§

Source

fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§