module

Logarithm::Systemd

Systemd journal integration for comprehensive log ingestion.

This module provides low-level and high-level interfaces for accessing systemd's binary journal format. It enables Logarithm to ingest logs directly from the systemd journal daemon, providing access to structured log data with rich metadata.

Overview

The systemd journal is a centralized logging system that stores log entries in a binary format with structured metadata. This module bridges Crystal applications with the systemd journal C API, providing both raw access and high-level abstractions.

Key Features

  • Direct Journal Access: Read from systemd's binary journal without text parsing
  • Structured Metadata: Access to timestamps, priorities, units, and custom fields
  • Efficient Seeking: Position-based navigation through journal entries
  • Cursor Support: Resume reading from specific positions
  • Time-based Filtering: Seek to specific timestamps
  • Memory Efficient: Streaming access without loading entire journal

Architecture

Application
    ↓
JournaldLogSource (high-level)
    ↓
JournalReader (Crystal wrapper)
    ↓
LibSystemd (C API bindings)
    ↓
systemd-journald (system service)

Requirements

  • System: Linux system with systemd
  • Permissions: Access to systemd journal (typically systemd-journal group)
  • Libraries: libsystemd.so (usually available on systemd systems)
  • Crystal: Standard library (no additional dependencies)

Common Use Cases

Log Aggregation

reader = Systemd::JournalReader.new
reader.open
reader.seek_tail

while entry = reader.next_entry
  puts "#{entry.timestamp} [#{entry.syslog_identifier}] #{entry.message}"
end

Time-based Analysis

reader = Systemd::JournalReader.new
reader.open
reader.seek_realtime(Time.utc - 1.hour)

# Process last hour's entries
while entry = reader.next_entry
  analyze_entry(entry)
end

Resume from Cursor

reader = Systemd::JournalReader.new
reader.open

if saved_cursor = load_cursor()
  reader.seek_cursor(saved_cursor)
end

# Continue processing
while entry = reader.next_entry
  process_entry(entry)
  save_cursor(entry.cursor)
end

Security Considerations

  • Access Control: Journal access requires appropriate permissions
  • Data Validation: Always validate journal field data
  • Resource Limits: Journal reading can consume significant memory
  • Rate Limiting: Implement throttling for high-volume scenarios

Performance Notes

  • Memory Usage: Journal entries are read one at a time
  • Disk I/O: Journal access involves disk reads from journal files
  • Seeking: Time-based seeks are O(log n), cursor seeks are O(1)
  • Filtering: Client-side filtering after reading entries

Error Handling

Journal operations can fail due to:

  • Permission Denied: Insufficient access to journal files
  • Journal Corruption: Damaged journal files
  • Resource Exhaustion: System running out of memory/disk
  • Network Issues: Remote journal access failures

Always check return values and handle errors gracefully.

Troubleshooting

Permission Issues

# Check journal access
sudo -u youruser journalctl --list-boots

# Add to systemd-journal group
sudo usermod -a -G systemd-journal youruser

Journal Not Available

# Check if journald is running
systemctl status systemd-journald

# Check journal files exist
ls -la /var/log/journal/

Performance Issues

  • Use cursors for resumable processing
  • Implement batching for high-volume scenarios
  • Consider time-based filtering to reduce data volume

Nested types