Cadmium::PragmaticTokenizer

This tokenizer is based off of the pragmatic_tokenizer ruby gem. It is much more robust than any of the other tokenizers, but has more features than you'll need for most use cases.

Constructor Options

• filter_languages : Array - user-supplied array of languages from which that language's stop words, abbreviations and contractions should be used when calculating the resulting tokens
• language : Symbol | String - two character ISO 639-1 code; can be a String or symbol (default :en)
• expand_contractions : Bool - (default: false)
• remove_stop_words : Bool - (default: false)
• abbreviations : Set(String) - user-supplied array of abbreviations (each element should be downcased with final period removed)
• stop_words : Set(String) - user-supplied array of stop words - array elements should be of the String class
• contractions : Hash(String, String) - user-supplied hash of contractions (key is the contracted form; value is the expanded form - both the key and value should be downcased)
• punctuation : PunctuationOptions - see description below
  • all: Does not remove any punctuation from the result
  • semi: Removes common punctuation (such as full stops) and does not remove less common punctuation (such as questions marks). This is useful for text alignment as less common punctuation can help identify a sentence (like a fingerprint) while common punctuation (like stop words) should be removed.
  • none: Removes all punctuation from the result
  • only: Removes everything except punctuation. The returned result is an array of only the punctuation.
• numbers : NumberOptions - see description below
  • all: Does not remove any numbers from the result
  • semi: Removes tokens that include only digits
  • none: Removes all tokens that include a number from the result (including Roman numerals)
  • only: Removes everything except tokens that include a number
• minimum_length : Int32 - minimum length of the token in characters
• long_word_split : Int32 - the specified length to split long words at any hyphen or underscore. 0 = no split (default).
• mentions : MentionOptions - what to do with mentions (such as '@watzon')
  • remove: will completely remove it
  • keep_and_clean: will prefix
  • keep_original: don't alter the token at all (default)
• hashtags : HashtagOptions - what to do with hashtags (such as '#crystal')
  • remove: will completely remove it,
  • keep_and_clean: will prefix
  • keep_original: don't alter the token at all (default)
• downcase : Bool - downcase all tokens (default: true)
• clean : Bool - removes some symbols (default: false)
• classic_filter : Bool - removes dots from acronyms and 's from the end of tokens. [link](# https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html#FilterDescriptions-ClassicFilter) (default: false)
• remove_emoji : Bool - strip emojis (default: false)
• remove_emails : Bool - strip emails (default: false)
• remove_urls : Bool - strip urls (default: false)
• remove_domains : Bool - strip domains (default: false)

Examples

tokenizer = Cadmium::Tokenizers::Pragmatic.new
tokenizer.tokenize("Hello world.")
# => ["hello", "world", "."]

tokenizer.tokenize("Jan. 2015 was 20% colder than now. But not in inter- and outer-space.")
# => ["jan.", "2015", "was", "20%", "colder", "than", "now", ".", "but", "not", "in", "inter", "-", "and", "outer-space", "."]

tokenizer.contractions = {"supa'soo" => "super smooth"}
tokenizer.expand_contractions = true
tokenizer.tokenize("Hello supa'soo guy.")
# => ["hello", "super", "smooth", "guy", "."]

tokenizer.clean = true
tokenizer.tokenize("This sentence has a long string of dots .......................")
# => ["this", "sentence", "has", "a", "long", "string", "of", "dots"]