class

Logarithm::TrainingBatchProcessor

Inherits Reference < Object

Memory-efficient batch processor for training logs

Constructors

new(batch_size : Int32 = 1000, max_memory_mb : Int32 = 256, use_chunked_storage : Bool = true)
Source

Instance methods

add_entry(log : String) : Bool

Add a log entry to the current batch if memory allows

Source
batch_ready?

Check if batch is ready for processing

Source
batch_size
Source
chunked_storage

Get access to chunked storage for reading

Source
cleanup

Clean up chunked storage

Source
current_batch
Source
current_memory_usage

Get current memory usage in bytes

Source
max_memory_mb
Source
memory_usage_percent

Get current memory usage as percentage of max

Source
reset_batch

Reset the current batch

Source