module

Bakelite::Chunker

Utility module for splitting large files into compressed chunks.

Class methods

chunk_data(data : Bytes, chunk_size : UInt32 = 65536_u32, compression : CompressionType = CompressionType::Deflate, base_offset : Int64 = 0_i64) : Array(Chunk)

Splits input byte slice into an array of compressed chunks of chunk_size.

Source

chunk_stream(io : IO, total_size : Int64, chunk_size : UInt32 = 65536_u32, compression : CompressionType = CompressionType::Deflate, base_offset : Int64 = 0_i64, &block : Chunk -> ) : Tuple(Array(Chunk), UInt32)

Reads an arbitrary IO stream in chunks of chunk_size and compresses each chunk. Yields each chunk to the provided block as soon as it is generated, maintaining strict O(chunk_size) peak RAM usage even for multi-gigabyte streams. Returns a Tuple containing the Array of Chunk descriptors and the total uncompressed CRC32.

Source

chunk_stream(io : IO, total_size : Int64, chunk_size : UInt32 = 65536_u32, compression : CompressionType = CompressionType::Deflate, base_offset : Int64 = 0_i64) : Tuple(Array(Chunk), UInt32)

Overload for chunk_stream when block is not provided

Source

compress_slice(slice : Bytes, compression : CompressionType) : Bytes

Compresses a raw byte slice using the requested algorithm.

Source