class

Logarithm::CLI::Train

Inherits Admiral::Command < Reference < Object

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts

Constants

HELP = {"description" => ""}

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts

Class methods

description

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts
Source
run(*args, **params)

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts
Source

Instance methods

__rescue_from___Admiral__Error(e)

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts
arguments

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts
Source
flags

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts
Source
help
run

The run command.

Source
sub(command, *args, **params)

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Arguments

  • paths: Log file or directory paths to monitor (space-separated, use quotes for paths with spaces)
    • Supports glob patterns and recursive directory traversal
    • Can be combined with --journald for hybrid monitoring
    • Example: "/var/log/syslog /var/log/messages" or "/var/log/*.log"

Flags

Data Sources

  • --recursive, -r: Recursively monitor directories for log files

    • When true, traverses subdirectories to find log files
    • Useful for monitoring log directories with date-based subfolders

  • --journald, -j: Include systemd journal in monitoring

    • Reads directly from systemd's binary journal
    • Requires appropriate permissions (typically needs systemd-journal group)
    • Can be combined with file paths for comprehensive monitoring

  • --since, -s: Start journal from specific time

    • Format: "1 hour ago", "2 days ago", "2024-01-01 12:00:00"
    • Supports relative times (hours, minutes, seconds, days ago)
    • Supports absolute ISO8601 timestamps
    • Only affects journald source, not file sources

Model Configuration

  • --data-dir, -D: Directory to store trained models

    • Overrides LOGARITHM_DATA_DIR environment variable
    • Must be writable and have sufficient space for model files
    • Models are stored as vectorizer.json and model.json

  • --vocab-size, -V: Vocabulary size for TF-IDF vectorizer

    • Range: 50-500 (based on log volume and complexity)
    • Larger vocabularies capture more terms but increase memory usage
    • Typical values: 100-200 for most applications
    • 0 uses config file default

  • --batch-size, -B: Batch size for incremental training

    • Range: 500-100000 (based on log volume and available memory)
    • Larger batches improve training stability but use more RAM
    • 0 uses config file default

  • --max-batches, -M: Maximum number of batches to collect

    • Range: 1-50 (limits training time and prevents overfitting)
    • Lower values train faster but may underfit complex patterns
    • Higher values provide better models but take longer
    • 0 uses config file default

Retraining Options

  • --retrain-mode, -m: Retraining mode

    • incremental (default): Load existing models, train on new logs
    • full: Ignore existing models, start fresh training
    • hybrid: Load models but force vocabulary expansion
    • Incremental preserves learning while adapting to new patterns

  • --expand-vocab, -e: Expand vectorizer vocabulary with new terms

    • When true, adds new terms encountered during training
    • Useful for adapting to new log formats or applications
    • Automatically enabled in hybrid retraining mode
    • Increases model size but improves coverage

  • --rollback, -b: Rollback to previous model version

    • Restores backup models created during last training
    • Useful if new training produces poor results
    • Only one level of rollback supported

Runtime Options

  • --time, -t: Training duration

    • Format: "30s", "5m", "2h", "1d" (seconds, minutes, hours, days)
    • Determines how long to collect training data
    • Longer training generally produces better models
    • Can be interrupted with Ctrl+C

  • --verbose, -v: Enable verbose logging

    • Shows detailed progress during training
    • Includes log collection stats, vectorization progress, training metrics
    • Useful for debugging and monitoring training progress

  • --config, -c: Config file path

    • Path to YAML configuration file
    • Overrides default location (~/.config/logarithm/config.yaml)
    • Settings merged with command-line flags (flags take precedence)

Examples

Basic Training

# Train on syslog for 24 hours
logarithm train /var/log/syslog --time 24h

# Train on multiple files
logarithm train "/var/log/syslog /var/log/messages" --time 12h

Journal Training

# Train on systemd journal
logarithm train --journald --time 6h

# Train on recent journal entries
logarithm train --journald --since "2 hours ago" --time 1h

Advanced Retraining

# Incremental retraining (default)
logarithm train --journald --retrain-mode incremental

# Full retraining (start fresh)
logarithm train /var/log/syslog --retrain-mode full

# Hybrid retraining with vocabulary expansion
logarithm train --journald --retrain-mode hybrid

# Expand vocabulary on existing model
logarithm train --expand-vocab --time 2h

# Rollback failed training
logarithm train --rollback

Hybrid Sources

# Combine journal and files
logarithm train "/var/log/syslog /var/log/auth.log" --journald --time 8h

# Recursive directory monitoring
logarithm train /var/log --recursive --time 4h

Training subcommand for model training and retraining.

This command collects log data for a specified duration and trains or retrains the anomaly detection model. Supports various retraining modes to adapt to evolving log patterns while preserving training history.

Generated Structs

Admiral automatically generates the following structs from the flag definitions:

Train::Flags

Contains all command-line flags and their parsed values for the training subcommand.

Flag Categories:

Data Source Configuration

  • recursive: Enable recursive directory traversal for log files
  • journald: Include systemd journal as a log source
  • since: Time-based filtering for journal entries

Model Training Parameters

  • data_dir: Override default model storage directory
  • vocab_size: Maximum vocabulary size for TF-IDF vectorizer
  • batch_size: Training batch size for incremental learning
  • max_batches: Maximum number of training batches to collect

Retraining Behavior

  • retrain_mode: Strategy for model updates ("incremental", "full", "hybrid")
  • expand_vocab: Allow vocabulary expansion during retraining
  • rollback: Restore previous model version

Runtime Configuration

  • time: Training duration specification
  • verbose: Enable detailed logging output
  • config: Path to YAML configuration file override

Train::Arguments

Contains positional arguments for the training subcommand.

  • paths: Space-separated list of log file or directory paths

Usage in Code

# Access parsed flags in command execution
if flags.verbose
  Logarithm::Log.info("Verbose training mode enabled")
end

# Use flag values for configuration
config.vocab_size = flags.vocab_size if flags.vocab_size > 0

# Parse space-separated paths
log_paths = arguments.paths.split.map(&.strip)

Validation

Flag and argument values are validated during parsing:

  • Time formats are checked for validity
  • Numerical ranges are enforced where applicable
  • File paths are validated for accessibility
  • Arguments are checked for directory traversal attempts
Source

Nested types