Mica::Core::UTF8
UTF-8 decoding and encoding (RFC 3629), shared by every text format.
Decoding is strict: overlong forms, encoded surrogates (U+D800 to U+DFFF), code points above U+10FFFF, the retired 5- and 6-byte forms (lead bytes F8 to FD), FE and FF, stray continuation bytes and truncated sequences are all rejected. Nothing is ever replaced with U+FFFD.
Class methods
Decodes the sequence at offset in bytes, reading no byte at or
past limit (at most bytes.size). Scanners call this form on their
hot path with their buffer, position and fill level; every read is
bounds-checked.
Decodes the sequence at the start of bytes. Returns
{code_point, byte_length}; on failure the length is 0 and the code
point is the Problem value.
Mica::Core::UTF8.decode("é".to_slice) # => {233, 2}
Mica::Core::UTF8.decode(Bytes[0xC0, 0x80]) # => {2, 0} (Problem::Overlong)Writes the UTF-8 encoding of code (a scalar value) at the start of
target, which must have room for encoded_size(code) bytes. Returns
the number of bytes written.
Returns the offset of the first invalid sequence in bytes, or nil when all of it is valid UTF-8. Runs in one pass.