Trait Tokenizer

pub trait Tokenizer: Send {
    // Required method
    fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>;
}

Tokenizer tokenizes a text into a list of tokens.

Required Methods

fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>

Implementors

impl Tokenizer for ChineseTokenizer

impl Tokenizer for EnglishTokenizer