HTML5::Tokenizer

returns a new HTML5 Tokenizer for the given IO Reader, for tokenizing an existing element's InnerHTML fragment. context_tag is that element's tag, such as "div" or "iframe".

For example, how the InnerHTML "a<b" is tokenized depends on whether it is for a

or a tag.

The input is assumed to be UTF-8 encoded.

returns a new HTML5 Tokenizer for the given IO Reader. The input is assumed to be UTF-8 encoded

allow_cdata sets whether or not the tokenizer recognizes <![CDATA[foo]]> as the text "foo". The default value is false, which means to recognize it as a bogus comment "<!-- [CDATA[foo]] -->" instead.

Strictly speaking, an HTML5 compliant tokenizer should allow CDATA if and only if tokenizing foreign content, such as MathML and SVG. However, tracking foreign-contentness is difficult to do purely in the tokenizer, as opposed to the parser, due to HTML5 integration points: an element can contain a <foreignObject> that is foreign-to-SVG but not foreign-to- HTML5. For strict compliance with the HTML5 tokenization algorithm, it is the responsibility of the user of a tokenizer to call allow_cdata as appropriate. In practice, if using the tokenizer without caring whether MathML or SVG CDATA is text or comments, such as tokenizing HTML5 to find all the anchor text, it is acceptable to ignore this responsibility.

buffered returns a slice containing data buffered but not yet tokenized

sets a limit on the amount of data buffered during tokenization. A value of 0 means unlimited

scans the next token and returns its type.

next_is_not_raw_text instructs the tokenizer that the next token should not be considered as 'raw text'. Some elements, such as script and title elements, normally require the next token after the opening tag to be 'raw text' that has no child elements. For example, tokenizing "<title>acd</title>" yields a start tag token for "<title>", a text token for "acd", and an end tag token for "</title>". There are no distinct start tag or end tag tokens for the "" and "".

This tokenizer implementation will generally look for raw text at the right times. Strictly speaking, an HTML5 compliant tokenizer should not look for raw text if in foreign content: <title> generally needs raw text, but a

<title> inside an does not. Another example is that a <textarea> generally needs raw text, but a <textarea> is not allowed as an immediate child of a <select>; in normal parsing, a <textarea> implies </select>, but one cannot close the implicit element when parsing a <select>'s InnerHTML. Similarly to allow_cdata, tracking the correct moment to override raw-text- ness is difficult to do purely in the tokenizer, as opposed to the parser. For strict compliance with the HTML5 tokenization algorithm, it is the responsibility of the user of a tokenizer to call NextIsNotRawText as appropriate. In practice, like allow_cdata, it is acceptable to ignore this responsibility for basic usage.

Note that this 'raw text' concept is different from the one offered by the Tokenizer.raw method.

raw returns the unmodified text of the current token. Calling Next, Token, Text, TagName or TagAttr may change the contents of the returned slice.

The token stream's raw bytes partition the byte stream (up until an ErrorToken). There are no overlaps or gaps between two consecutive token's raw bytes. One implication is that the byte offset of the current token is the sum of the lengths of all previous tokens' raw bytes.

tag_attr returns the HTML5.lower-cased key and unescaped value of the next unparsed attribute for the current tag token and whether there are more attributes. The contents of the returned slices may change on the next call to next.

tag_name returns the HTML5.lower-cased name of a tag token (the "img" out of ) and whether the tag has attributes. The contents of the returned slice may change on the next call to next.

text returns the unescaped text of a text, comment or doctype token. The contents of the returned slice may change on the next call to next.

token returns the current Token. The result's data and attr values remain valid after subsequent next calls.