class

Text::Splitter

Inherits Reference < Object

Text::Splitter intelligently splits text into chunks for LLM/RAG applications.

It supports two splitting modes:

  • Characters: Splits based on character count (default)
  • Words: Splits based on word count (more semantic)

Both modes respect sentence boundaries and support configurable overlap to preserve context between chunks.

Example

# Character-based splitting
splitter = Text::Splitter.new(
  chunk_size: 1000,
  chunk_overlap: 200
)

chunks = splitter.split_text("Your long document...")
# => ["First chunk...", "Second chunk with overlap..."]

# Word-based splitting
word_splitter = Text::Splitter.new(
  chunk_size: 280,
  chunk_overlap: 50,
  mode: :words
)

Constructors

new(chunk_size : Int32, chunk_overlap : Int32, mode : ChunkMode = ChunkMode::Characters)

Creates a new text splitter

  • chunk_size: Maximum size of each chunk (in characters or words)
  • chunk_overlap: Overlap between chunks for context preservation
  • mode: Splitting mode (:characters or :words)
Source

Instance methods

chunk_overlap

Amount of overlap between chunks (characters or words depending on mode)

Source
chunk_size

Size of each chunk (characters or words depending on mode)

Source
each_chunk(text : String, & : String -> )

Yields each chunk lazily without allocating all chunks upfront

This is more memory efficient for large texts

splitter = Text::Splitter.new(chunk_size: 100, chunk_overlap: 20)
splitter.each_chunk(text) do |chunk|
  puts chunk
end
Source
each_chunk(text : String)

Returns an iterator over chunks

splitter = Text::Splitter.new(chunk_size: 100, chunk_overlap: 20)
splitter.each_chunk(text).first(10) # Get first 10 chunks only
Source
mode

Splitting mode (characters or words)

Source
split_text(text : String) : Array(String)

Splits text into chunks based on the configured mode

Returns an empty array if text is empty or only whitespace

splitter = Text::Splitter.new(chunk_size: 100, chunk_overlap: 20)
chunks = splitter.split_text("Long document text here...")
Source

Nested types