Logarithm::MemoryEfficientBatchProcessor
Memory-efficient batch processor.
This class processes logs in batches while monitoring memory usage to prevent out-of-memory errors.
Example usage:
processor = MemoryEfficientBatchProcessor.new(
batch_size: 1000,
max_memory_mb: 512
)
logs.each do |log|
if processor.add_entry(log)
# Entry added to current batch
else
# Batch full, process current batch first
process_batch(processor.current_batch)
processor.reset_batch
processor.add_entry(log)
end
end
Constructors
Instance methods
batch_size
Sourcecurrent_batch
Sourcemax_memory_mb
Source