class

Logarithm::ChunkedTrainingStorage

Inherits Reference < Object

File-based chunked storage for training samples to prevent memory accumulation

Constructors

new(temp_dir : String = "", chunk_size : Int32 = 10000, max_chunks : Int32 = 100)
Source

Instance methods

add_entry(log : String) : Bool

Add a log entry to the current chunk file

Source
chunk_files

Get all chunk files

Source
chunk_size
Source
cleanup

Clean up all temporary files

Source
max_chunks
Source
read_all_logs

Read all logs from chunk files in chunks

Source
rotate_chunk

Close current chunk and start a new one

Source
temp_dir
Source
total_logs

Get total number of logs across all chunks

Source