class

Mica::Core::Transcoder

Read-only IO that turns UTF-16, UTF-32 or ISO-8859-1 from source into UTF-8, in Crystal (no iconv).

Invalid input is never repaired or dropped. It becomes a marker byte that is never valid UTF-8, so the format's scanner stops at the exact place and reports it:

INVALID_UNIT (0xFF) stands for one whole bad code unit: an unpaired UTF-16 surrogate, or a UTF-32 value that is a surrogate or above U+10FFFF.
TRUNCATED_BYTE (0xFE) stands for one byte of an incomplete final code unit (one per leftover byte).

source_size maps decoded bytes back to source bytes, which is how ByteInput reports error offsets in the original input.

Constants

INVALID_UNIT = 255_u8

TRUNCATED_BYTE = 254_u8

Constructors

new(source : IO, source_encoding : Encoding, prefix : Bytes = Bytes.empty)

prefix holds source bytes already read from source (for example while sniffing a byte order mark); they are decoded first. The constructor copies prefix unmetered: a caller with a meter charges a long prefix before constructing (TextInput#switch_to_transcoder does); the XML reader's sniffed prefix is at most four bytes.

Source

Class methods

source_size(encoding : Encoding, decoded : Bytes) : Int64

Returns the number of source bytes the decoded UTF-8 decoded came from. decoded must start on a sequence boundary of this transcoder's output.

Source

Instance methods

after_source_read

Called after every read from the source returns, so the owner can check its deadline between the several reads one output fill may need. TextInput sets it.

Source

after_source_read=(after_source_read : Proc(Nil) | Nil)

Called after every read from the source returns, so the owner can check its deadline between the several reads one output fill may need. TextInput sets it.

Source

read(slice : Bytes) : Int32

Fills slice with UTF-8. Returns 0 at the end of the source. Needs room for one encoded code point (4 bytes); smaller slices get the bytes one read at a time through an internal carry. Each call reads from the source at most once for every code unit it needs to complete (at most four reads), then returns what it has decoded.

Source

source_encoding

Returns the encoding of the source bytes.

Source

write(slice : Bytes) : Nil

Writes the contents of slice into this IO.

io = IO::Memory.new
slice = Bytes.new(4) { |i| ('a'.ord + i).to_u8 }
io.write(slice)
io.to_s # => "abcd"

Source