Cadmium::SentenceTokenizer
Inherits Cadmium::Tokenizer < Reference < Object
Constants
REGEX_PATTERN = /([\"\'\‘\“\'\"\[\(\{\⟨][^\.\?\!]+[\.\?\!][\"\'\’\”\'\"\]\)\}\⟩]|[^\.\?\!]+[\.\?\!])\s?/
Instance methods
tokenize(string : String) : Array(String)
Source