module

Marcel::Magic::Xml

Procedural refinement of XML documents by their root element.

Many XML vocabularies — feeds, KML, property lists, XSLT, Office 2003 XML — share the same leading bytes, and a prolog of arbitrary length (declaration, comments, DOCTYPE) can precede the element that tells them apart. Apache Tika identifies them by the root element's namespace and local name instead, and those rules are generated into ROOT_XML. This module scans a bounded prefix past the prolog to the root start-tag, resolves the root's namespace from its own xmlns declarations, and looks the pair up to refine a generic application/xml match.

Constants

ATTRIBUTE = Regex.new("\\G#{SPACE}+(#{NAME})#{SPACE}*=#{SPACE}*(?:\"([^\"<]*)\"|'([^'<]*)')", SCAN)
COMMENT_CLOSE = "-->"
COMMENT_OPEN = "<!--"
DOCTYPE_DELIMITERS = Bytes[34, 39, 91, 93, 60, 62]
DOCTYPE_OPEN = "<!DOCTYPE"
DOUBLE_HYPHEN = "--"
ENCODING_NAME = "[A-Za-z][A-Za-z0-9._\\-]*"
MAX_SCAN = 64 * 1024

Tika examines at most this much of a document for its root element.

NAME = "[A-Za-z_:\\x80-\\x{10FFFF}][A-Za-z0-9._:\\-\\x80-\\x{10FFFF}]*"

Scanner pattern for names: XML Name restricted to ASCII plus any non-ASCII character (в UTF-режиме PCRE2 — до U+10FFFF включительно, аналог любого не-ASCII байта в Ruby /n). It only locates tokens; every captured name is then validated against the real Unicode name grammar (valid_name? below) after decoding.

PREDEFINED_ENTITIES = ["amp", "lt", "gt", "apos", "quot"] of ::String
PROCESSING_INSTRUCTION = Regex.new("\\G<\\?(#{NAME})", SCAN)
PROCESSING_INSTRUCTION_CLOSE = "?>"
PROCESSING_INSTRUCTION_OPEN = "<?"
PSEUDO_ENCODING_NAMES = ["locale", "external", "filesystem"] of ::String

Ruby резолвит их в процессные умолчания, а не в charset.

REFERENCE = Regex.new("&(?:(#{NAME})|\\#([0-9]+)|\\#x([0-9A-Fa-f]+));", SCAN)

Entity and character references; anything else after & in an attribute value is a well-formedness error. (группы: 1 — имя, 2 — decimal, 3 — hex)

REFINABLE_TYPE = "application/xml"
REJECTED_ENCODING = /utf-?7|utf-?16|utf-?32|ucs-?2|ucs-?4|iso-2022|hz|ansi_x3\.110|binary|ascii-8bit/i

Не-ASCII-совместимые и «dummy» кодировки Ruby (UTF-16/32, UCS, UTF-7, ISO-2022-*, HZ) и псевдонимы бинарной — iconv их знает, но скан байтов по ним невозможен; find_encoding их отклоняет, как Encoding#find + проверки.

REPLACEMENT_CHARACTER = "�"
RESERVED_TARGET = Regex.new("\\A[Xx][Mm][Ll]\\z", SCAN)
SCAN = Regex::Options::MATCH_INVALID_UTF

Сканер работает по сырым (возможно, невалидным) UTF-8 байтам, как Ruby-регэкспы с флагом /n: MATCH_INVALID_UTF запрещает совпадения сквозь невалидные последовательности, но не мешает совпадениям до них и не бросает ArgumentError.

SPACE = "[ \\t\\r\\n]"
SPACE_BYTES = [32, 9, 13, 10]
START_TAG = Regex.new("\\G<(#{NAME})", SCAN)
START_TAG_END = Regex.new("\\G#{SPACE}*\\/?>", SCAN)
UTF16BE_BOM = "\xFE\xFF"
UTF16LE_BOM = "\xFF\xFE"
UTF8_BOM = "\uFEFF"
WHITESPACE = Regex.new("\\G#{SPACE}*", SCAN)
XML_DECLARATION = Regex.new((("\\A<\\?xml#{SPACE}+version#{SPACE}*=#{SPACE}*(?:\"(1\\.[01])\"|'(1\\.[01])')" + "(?:#{SPACE}+encoding#{SPACE}*=#{SPACE}*(?:\"(#{ENCODING_NAME})\"|'(#{ENCODING_NAME})'))?") + "(?:#{SPACE}+standalone#{SPACE}*=#{SPACE}*(?:\"(?:yes|no)\"|'(?:yes|no)'))?") + "#{SPACE}*\\?>", SCAN)

The complete declaration grammar: a SAX-supported version, then optionally encoding, then optionally standalone, in that order, with nothing else before ?>. Matched in full so that trailing or duplicated tokens cannot hide behind the PI skipper. (группы: 1,2 — version; 3,4 — encoding)

XML_NAMESPACE = "http://www.w3.org/XML/1998/namespace"
XMLNS_NAMESPACE = "http://www.w3.org/2000/xmlns/"

Class methods

byte_at?(data : String, pos : Int32) : UInt8 | Nil
Source
byteslice?(data : String, pos : Int32, size : Int32) : String | Nil
Source
contains_forbidden_char?(text : String, version : String) : Bool

Символы, запрещённые XML-версией как литералы (аналог FORBIDDEN_CHARS).

Source
decode(prefix : String) : Tuple(String, Symbol, String) | Nil

Returns {текст для сканирования, режим (:utf8/:replaced), версия XML}, или nil, когда префикс не декодируется. Сканирование всегда читает UTF-8 байты: magic-гейт гарантирует BOM на UTF-16 входе, который строго транскодируется заранее; документы уже в UTF-8 сканируются как есть; любая другая объявленная кодировка транскодируется с заменой (invalid/undef → U+FFFD), так как её многобайтовые символы могут нести ASCII-подобные хвостовые байты (Shift_JIS ソ кончается на ASCII-бэкслеш), которые саботировали бы побайтовое сканирование.

Source
find_encoding(name : String) : String | Nil
Source
name_char?(cp : Int32) : Bool

NameChar: NameStartChar плюс '-' '.' 0-9 U+00B7 U+0300-U+036F U+203F-U+2040.

Source
name_start_char?(cp : Int32) : Bool

NameStartChar под XML 1.0 (5-е изд.), Unicode-грамматика имён.

Source
namespace_bindings(attributes : Hash(String, String), version : String) : Hash(String | Nil, String | Nil) | Nil

The tag's namespace declarations as {prefix or nil => namespace or nil}, with the xml prefix implicitly bound. The xml and xmlns names and namespaces are reserved. A prefix bound to an empty namespace is an undeclaration, permitted by Namespaces 1.1 (except for xml) and an error under 1.0; violations are namespace errors, returned as nil.

Source
next_delimiter(data : String, pos : Int32) : Int32 | Nil
Source
parse_start_tag(data : String, pos : Int32, mode : Symbol, version : String, doctype_seen : Bool) : Tuple(String | Nil, String | Nil) | Nil

Reads the root start-tag's qualified name and attributes. Attribute values are parsed as quoted literals so a > within one cannot truncate the tag; the tag must close within the scanned prefix, and everything consumed up to that point must be valid in the document's encoding and free of forbidden characters.

Source
read_prefix(io : IO) : String
Source
reencode(bytes : Bytes, encoding : String) : String

Replace-transcodes a legacy-encoded prefix to UTF-8 (аналог Ruby encode(invalid: :replace, undef: :replace)): каждая невалидная или непредставимая последовательность становится U+FFFD.

Source
refine(io : IO | String, base_type : String) : String

Returns the type ROOT_XML assigns to the IO's root element if +base_type+ is the generic XML type and the root element can be read, or +base_type+ unchanged. Partial reads, roots past MAX_SCAN, malformed or mis-encoded prologs, namespace errors and unknown roots refine nothing: the base type stands.

Source
resolve_name(qualified_name : String, bindings : Hash(String | Nil, String | Nil)) : Tuple(String | Nil, String | Nil) | Nil

Resolves the root element's name against the tag's own bindings: the default namespace for an unprefixed name, the prefix's binding otherwise. An unbound prefix or invalid name is a namespace error.

Source
root_element(data : String, mode : Symbol, version : String) : Tuple(String | Nil, String | Nil) | Nil

Skips the prolog — whitespace, processing instructions (including the XML declaration), comments and the document type declaration — and returns the root element's {namespace, local name}, or nil if no root start-tag is found.

Source
skip_comment(data : String, pos : Int32) : Int32 | Nil

A comment ends at its first "--", which must be the one closing it.

Source
skip_doctype(data : String, pos : Int32) : Int32 | Nil

Advances past a document type declaration, honouring quoted literals and the bracketed internal subset so that markup declarations like

<!ENTITY x "<foo>"> and comments within the subset cannot end the scan early.
Source
skip_past(data : String, terminator : String, pos : Int32) : Int32 | Nil
Source
skip_processing_instruction(data : String, pos : Int32) : Int32 | Nil

The XML declaration is the one processing instruction allowed the xml target, and only as the very first bytes of the document (checked by xml_declaration). A PI's data, when present, must be separated from its target by whitespace.

Source
split_qualified_name(name : String) : Tuple(String | Nil, String | Nil) | Nil

Splits a QName into {prefix or nil, local name}; more than one colon or an empty part is a namespace error, returned as nil.

Source
transcode(bytes : Bytes, encoding : String) : String

A code unit or surrogate pair split by the scan limit is the one sequence that may legitimately be incomplete: it is dropped, since it lies beyond the scan. Any other invalid sequence raises and refines nothing.

Source
valid_attributes?(attributes : Hash(String, String), bindings : Hash(String | Nil, String | Nil), version : String, doctype_seen : Bool) : Bool

Ordinary attributes resolved by expanded name, as a namespace-aware parser sees them: names must be valid NCNames, prefixes bound, {namespace, local name} pairs unique (the default namespace does not apply to attributes), and values free of malformed references.

Source
valid_character_reference?(digits : String, base : Int32, version : String) : Bool

A character reference admits unlimited leading zeros, so the scalar's magnitude is bounded lexically — at most seven significant decimal digits or six hex cover every code point through U+10FFFF — before conversion, never by converting an unbounded digit run. All zeros denote #x0, rejected as a scalar.

Source
valid_character_scalar?(value : Int, version : String) : Bool

A character reference must denote a character the XML version admits: never a surrogate, a noncharacter, or a code point beyond U+10FFFF; XML 1.0 admits only tab, newline, carriage return and #x20 up, while XML 1.1 admits everything from #x1 — including as references the control characters it forbids as literals.

Source
valid_name?(name : String, xml_name : Bool = false) : Bool

Names must satisfy the Unicode name grammar once decoded — NCName unless the laxer XML Name (с двоеточием) is requested — with the byte-level scanner accepting a superset purely to locate them.

Source
valid_references?(value : String, version : String, doctype_seen : Bool) : Bool

Every & in an attribute value must begin a well-formed reference: a character reference to an admissible character, a predefined entity, or — only when a DTD that could declare it was seen — any well-named entity. Never resolved.

Source
valid_text?(consumed : String, mode : Symbol, version : String) : Bool

The consumed prefix must be free of the characters its XML version forbids as literals, wherever they fall — comment, PI, DOCTYPE or attribute — since a SAX parser rejects them before the root. Raw and strictly transcoded input must be valid UTF-8, in which a genuine U+FFFD literal is legal XML; replace-transcoded input must carry no U+FFFD, each one marking a sequence invalid in the declared encoding.

Source
xml_declaration(data : String) : Tuple(String, String | Nil) | Nil

Returns {version, declared encoding name or nil} when the document either has no XML declaration (version defaults to 1.0) or opens with one that is well-formed in full, and nil when it opens with a malformed declaration or a processing instruction that misuses the xml target.

Source