class

Logarithm::Pipeline

Inherits Reference < Object

The main pipeline for log anomaly detection.

This class orchestrates the entire anomaly detection workflow:

  1. Log ingestion from various sources (journald, files)
  2. Text vectorization using TF-IDF
  3. Machine learning model training/prediction
  4. Anomaly detection and alerting

The pipeline supports flexible retraining modes to adapt to evolving log patterns:

  • incremental: Load existing models and train on new logs (default)
  • full: Ignore existing models and start fresh training
  • hybrid: Load models but force vocabulary expansion

Example usage:

pipeline = Pipeline.new(log_source, vectorizer, model, config)
pipeline.train(24.hours) # Train for 24 hours
pipeline.monitor         # Start monitoring for anomalies

Constructors

new(log_source : AbstractLogSource | Nil, vectorizer : Logarithm::AbstractVectorizer, model : Logarithm::AbstractModel, config : Logarithm::Config, verbose : Bool = false, expand_vocab : Bool = false, retrain_mode : String = "incremental", use_cross_validation : Bool = false, cv_folds : Int32 = 5)

Creates a new pipeline instance.

Parameters:

  • log_source: Source for log ingestion (journald, files, etc.)
  • vectorizer: Text vectorization implementation (TF-IDF)
  • model: Machine learning model (autoencoder)
  • config: Configuration settings
  • verbose: Enable detailed logging
  • expand_vocab: Expand vectorizer vocabulary with new terms
  • retrain_mode: Retraining strategy ("incremental", "full", "hybrid")
  • use_cross_validation: Enable cross-validation during training
  • cv_folds: Number of cross-validation folds
Source

Instance methods

config
Source
cv_folds
Source
expand_vocab?
Source
log_source
Source
model
Source
monitor

Starts real-time anomaly monitoring.

This method continuously monitors log sources for anomalies using the trained model. It runs indefinitely until interrupted or an error occurs.

The monitoring process:

  1. Loads the trained vectorizer and model
  2. Starts log ingestion from configured sources
  3. Processes each log entry in real-time:
    • Vectorizes the log text
    • Runs anomaly detection using the model
    • Logs alerts for anomalies above the threshold

Anomalies are detected when the reconstruction error exceeds the configured threshold. The system uses a keep-alive mechanism to prevent idle timeouts.

Source
mse_error(original : Tensor, reconstructed : Tensor) : Float64

Calculates Mean Squared Error between original and reconstructed vectors.

This is the core anomaly detection metric. The autoencoder tries to reconstruct the input vector, and the reconstruction error indicates how "normal" the input is. Higher errors suggest anomalies.

Formula: MSE = Σ((original_i - reconstructed_i)²) / n

Parameters:

  • original: The original vectorized log
  • reconstructed: The autoencoder's reconstruction

Returns: Mean squared error as a Float64

Source
retrain_mode
Source
rollback

Rolls back to the previous model version.

This method restores the backup models created during the last training session. Useful if new training resulted in poor model performance or unexpected behavior.

The rollback process:

  1. Checks for backup files (.bak extension)
  2. Renames backup files to active model files
  3. Logs the rollback operation

Note: Only one level of rollback is supported. Multiple rollbacks will require retraining from scratch.

Source
train(duration : Time::Span)

Trains the anomaly detection model on log data.

This method collects logs for the specified duration and trains or retrains the machine learning model based on the configured retrain_mode.

The training process:

  1. Determines whether to load existing models based on retrain_mode
  2. Collects logs from the configured sources for the given duration
  3. Performs vectorization and model training
  4. Saves the trained models with automatic backup

Parameters:

  • duration: How long to collect training logs

Raises:

  • IngestionError: If log collection fails
  • ModelError: If model training fails
Source
train_with_cv(duration : Time::Span, cv_folds : Int32 | Nil = nil) : Hash(String, Float64 | Array(Float64))

Trains the anomaly detection model with cross-validation.

This method performs k-fold cross-validation during training to evaluate model performance and prevent overfitting. It provides more robust model evaluation compared to single train/validation split.

Parameters:

  • duration: How long to collect training logs
  • cv_folds: Number of cross-validation folds (overrides instance setting)

Returns: Cross-validation results including mean loss and standard deviation

Source
use_cross_validation?
Source
vectorizer
Source