Logarithm::AbstractVectorizer

Abstract base class for text vectorization algorithms.

Vectorizers convert raw text log messages into numerical vectors that can be processed by machine learning models. They learn patterns in the text during training and apply the same transformation during inference.

Role in Pipeline

Vectorizers are the second stage in the anomaly detection pipeline:

1. Log sources provide raw text
2. Vectorizers convert text to vectors
3. ML models learn patterns in vectors
4. Anomalies are detected based on reconstruction error

Interface

All vectorizers must implement:

• fit(logs): Learn vocabulary and patterns from training data
• transform(log): Convert a single log to vector representation
• vocab_size: Property indicating learned vocabulary size
• Save/load methods for model persistence

Example Implementation

class SimpleVectorizer < AbstractVectorizer
  @word_to_index = {} of String => Int32

  def fit(logs : Array(String))
    # Build vocabulary from training logs
    words = logs.flat_map(&.split).uniq
    @word_to_index = words.each_with_index.to_h
    @vocab_size = words.size
  end

  def transform(log : String) : Tensor
    # Convert log to bag-of-words vector
    vector = Array(Float32).new(@vocab_size, 0.0_f32)
    log.split.each do |word|
      if index = @word_to_index[word]?
        vector[index] = 1.0_f32
      end
    end
    Tensor.new(vector)
  end

  # Implement save/load methods...
end

Built-in Implementations

• TfidfVectorizer: TF-IDF weighted term vectors with configurable vocabulary