module

Logarithm::Retry

Resilience utilities for handling transient failures in distributed systems.

This module provides two key resilience patterns essential for reliable anomaly detection in production environments: exponential backoff retry and circuit breaker. These patterns prevent cascading failures and provide graceful degradation.

Exponential Backoff Retry

The with_backoff method implements truncated exponential backoff with jitter. It automatically retries operations that may fail due to transient issues like network timeouts, temporary file locks, or resource contention.

Algorithm:

  • Attempt 1: Immediate execution
  • Attempt 2: Wait base_delay * 2^(1-1) = base_delay
  • Attempt 3: Wait base_delay * 2^(2-1) = base_delay * 2
  • Attempt N: Wait base_delay * 2^(N-1)

Circuit Breaker Pattern

The CircuitBreaker class implements the circuit breaker pattern to prevent cascading failures. When a service repeatedly fails, the circuit "opens" to stop wasting resources on doomed operations, allowing the system to fail fast and recover gracefully.

States:

  • CLOSED: Normal operation, requests pass through
  • OPEN: Failure threshold exceeded, requests fail immediately
  • HALF_OPEN: Testing if service has recovered

Integration with Anomaly Detection

These resilience patterns are used throughout the Logarithm pipeline:

  • Model Loading: Retry failed model file reads due to temporary I/O issues
  • Log Ingestion: Handle temporary network issues with journald/systemd
  • External APIs: Circuit break calls to external services or databases
  • File Operations: Retry file system operations in distributed environments

Configuration Guidelines

Retry Configuration:

  • max_attempts: 3-5 for most operations, higher for critical paths
  • base_delay: Start with 100ms, adjust based on operation latency

Circuit Breaker Configuration:

  • failure_threshold: 3-5 consecutive failures to trigger open state
  • timeout: 30-60 seconds before attempting reset
  • success_threshold: 2-3 successes needed to close circuit

Usage Examples

# Simple retry with defaults
result = Logarithm::Retry.with_backoff { unreliable_operation() }

# Configured retry for slow operations
result = Logarithm::Retry.with_backoff(max_attempts: 5, base_delay: 1.second) {
  external_api_call()
}

# Circuit breaker for external service
breaker = Logarithm::Retry::CircuitBreaker.new(
  failure_threshold: 3,
  timeout: 30.seconds,
  success_threshold: 2
)

result = breaker.call { external_service.request(data) }

Error Handling Strategy

  • Retryable Errors: Network timeouts, temporary I/O errors, resource locks
  • Non-Retryable Errors: Authentication failures, invalid data, configuration errors
  • Circuit Breaking: Persistent service failures, resource exhaustion

The system distinguishes between retryable and non-retryable errors to avoid wasting resources on operations that will never succeed.

Class methods

with_backoff(max_attempts : Int32 = 3, base_delay : Time::Span = 0.1.seconds, &)

Retries a block with exponential backoff for transient failures.

This method implements truncated exponential backoff, doubling the delay with each retry attempt. It preserves the last exception if all attempts fail.

Parameters:

  • max_attempts: Maximum number of retry attempts (default: 3)
  • base_delay: Initial delay between retries (default: 100ms)
  • block: The operation to retry

Returns: The result of the successful operation

Raises: The last exception encountered if all attempts fail

Example:

data = Logarithm::Retry.with_backoff(max_attempts: 5) {
  File.read("model.json") # May fail due to temporary locks
}
Source

Nested types