class

Logarithm::TfidfVectorizer

Inherits Logarithm::AbstractVectorizer < Reference < Object

TF-IDF vectorizer for text feature extraction.

This class converts log text into numerical vectors using Term Frequency-Inverse Document Frequency (TF-IDF) weighting. It tokenizes logs, builds a vocabulary, and transforms text into sparse vectors suitable for machine learning.

Features:

  • Text sanitization (removes special characters, normalizes whitespace)
  • Tokenization with configurable vocabulary size limits
  • IDF calculation for term weighting
  • Sparse vector representation

The vectorizer can expand its vocabulary with new terms encountered during incremental training, allowing adaptation to evolving log patterns.

Constructors

new(max_features : Int32 = 10000, memory_limit : Int32 = (100 * 1024) * 1024)
Source

Instance methods

fit(logs : Array(String))

Learns vocabulary and transformation parameters from training logs.

This method analyzes the training data to build internal data structures needed for vectorization. It should set the vocab_size property.

Parameters:

  • logs: Array of training log messages

This method is called once during training.

Source
load(path : String)

Loads the vectorizer state from a file.

Parameters:

  • path: File path to load from
Source
load_from_string(data : String)

Deserializes the vectorizer from a string.

Parameters:

  • data: String representation of vectorizer state
Source
save(path : String)

Saves the vectorizer state to a file.

Parameters:

  • path: File path to save to
Source
save_to_string

Serializes the vectorizer to a string for storage/encryption.

Returns: String representation of the vectorizer state

Source
transform(log : String) : Tensor

Converts a single log message to its vector representation.

The output vector should have vocab_size dimensions and be compatible with the ML model's input requirements.

Parameters:

  • log: Single log message to vectorize

Returns: Numerical vector representation as Tensor

Source