Logarithm

Logarithm: Self-Learning Anomaly Detection Agent

Logarithm is a machine learning-powered diagnostics agent for GNU/Linux systems that automatically learns normal system behavior and detects anomalies in real-time. Built with Crystal for performance and reliability, it uses unsupervised learning to identify potential security threats, system issues, or unusual behavior.

Architecture Overview

Logarithm follows a modular pipeline architecture:

Log Sources → Vectorization → ML Model → Anomaly Detection → Alerts
     ↓            ↓            ↓            ↓            ↓
  journald    TF-IDF      Autoencoder   Threshold    syslog/
  syslog      +vocab      (neural)     scoring      stdout
  files       expansion   network

Core Components

• Ingestion: Reads logs from systemd journal, syslog files, or custom sources
• Vectorization: Converts text logs to numerical vectors using TF-IDF
• Model: Autoencoder neural network that learns normal log patterns
• Detection: Real-time anomaly scoring with configurable thresholds
• CLI: Command-line interface for training and monitoring

Quick Start

Training (Learn Normal Behavior)

# Train on systemd journal logs for 24 hours
logarithm train --journald --time 24h

# Train on syslog files
logarithm train /var/log/syslog /var/log/messages

# Incremental retraining (preserve existing model)
logarithm train --journald --retrain-mode incremental

Monitoring (Detect Anomalies)

# Monitor systemd journal in real-time
logarithm monitor --journald

# Monitor multiple sources
logarithm monitor /var/log/syslog --journald

Advanced Features

Flexible Retraining

• Incremental: Load existing models, train on new logs (default)
• Full: Start fresh training, ignore existing models
• Hybrid: Load models but expand vocabulary for new terms

Time-Based Filtering

# Train on logs from last 2 hours
logarithm train --journald --since "2 hours ago"

# Train from specific timestamp
logarithm train --journald --since "2024-01-01 12:00:00"

Model Management

# Expand vocabulary with new terms
logarithm train --expand-vocab

# Rollback to previous model version
logarithm train --rollback

Configuration

Logarithm supports YAML configuration files, environment variables, and CLI flags:

# config.yaml
data_dir: ~/.local/share/logarithm
threshold: 0.85          # Anomaly detection threshold
vocab_size: 1000         # TF-IDF vocabulary size
batch_size: 10000        # Training batch size

Security & Reliability

• AES-256 encryption for model storage
• Audit logging for all operations
• Input validation and sanitization
• Retry logic with exponential backoff
• Circuit breakers for resilient operation
• Permission checking for log access

Integration Examples

Programmatic Usage

require "logarithm"

# Create and configure pipeline
vectorizer = Logarithm::TfidfVectorizer.new
model = Logarithm::Autoencoder.new(1000)
config = Logarithm::Config.new
log_source = Logarithm::JournaldLogSource.new

pipeline = Logarithm::Pipeline.new(
  log_source, vectorizer, model, config,
  verbose: true,
  retrain_mode: "incremental"
)

# Train the model
pipeline.train(1.hour)

# Start monitoring
pipeline.monitor

Systemd Service

[Unit]
Description=Logarithm Anomaly Detection
After=systemd-journald.service

[Service]
ExecStart=/usr/local/bin/logarithm monitor --journald
Restart=always
User=logarithm

[Install]
WantedBy=multi-user.target

API Reference

The main entry points are:

• Logarithm::CLI: Command-line interface
• Logarithm::Pipeline: Core processing pipeline
• Logarithm::Config: Configuration management
• Various log sources, vectorizers, and models in their respective modules

See the generated API documentation for detailed method references.